File fields, put, read and the disk driver

A File field, store.put and store.read each refuse what doesn't fit. This topic has the exact rules, and where jadpo dev and tests keep files, for when a write is refused or a read is too large (Files has how to use them).

What a File holds

A File holds the connection its bytes are in, the key the runtime made for them, its name (none for one store.put made), its size and its FileType. It is kept as its JSON in a jsonb column.

Only the runtime builds one: File { ... } is type.not_record.

Nothing from outside carries a File, which would let the sender name any file. It is type.batch_input in an Input or a route's placeholder, in an operation's output: or wire_output:, in a webhook's event, and in an event a queue lists.

A route sends a File as a signed link, which comes with downloads. Until those are built, a File in a route's output: or a failure's client: is toolchain.not_built_yet. A failure's internal: fields stay on the server and may hold one.

Sending a file to a provider isn't built. A File in an operation's input: or a webhook's reply: is toolchain.not_built_yet, since the provider would get the store and key and none of the bytes (Types).

Each of these holds however deep: a list, a map, an Object, an entity's field named as a type, a pick:, or an ExportSummary, which holds the file it wrote. The other ways a value leaves or arrives refuse one too:

A File field on an EmailMessage is the one that leaves: it is sent as an attachment, the bytes rather than the store and key (Attachments).

A program reads none of its parts. file.key, and file.nope, are type.field_on_non_record, in code and in a template.

A File field's settings

• store: must name a connection that provides files.
• max_size: is required: a whole number of b, kb, mb or gb, each a thousand times the last. 5mb is 5,000,000 bytes, which is why this page writes "100 MB" apart from "16 MiB".
• types: is optional.

A mistake in any of them is data.file_field_invalid. The three settings on a field that isn't a File are type.unknown_setting. Every write of the field holds the file it is given to them (What a write checks).

store.put(bytes, type)

You call store.put only in an action, before its first write, as every outside call is made. Its key is made from the invocation's randomness, so a replay keeps it. It goes out as a keyed call, since writing the same key again writes the same object, which never changes.

A store with no room answers storage_full, the call's one condition. Its attempt maps it to a failure of kind unavailable; it isn't a fault of a value, so invalid: doesn't cover it. A store that can't be reached is the runtime's own unavailable.

What a write checks

A File is kept in a field only when it fits the field's settings. Every create, update and patch that sets a File field checks, before its statement runs, that the file is:

1. in the store the field's store: names;
2. of a type its types: lists, when it lists any;
3. no larger than its max_size:.

A file that breaks one stops the write with the fault file_refused, naming the field and what didn't fit, and nothing the action wrote is kept (Defects). It is a fault, not a failure, because only the runtime makes a File, from the bytes, the type and the store the action gave put, so the mismatch is the program's own. Bound the bytes before the put, such as by the max_length: of the text they are made from, or change the field's settings. Clearing a field, or a patch that leaves it out, checks nothing.

Here the bakery's sheets are at most 32 bytes, so a four-line sheet isn't kept:

domains/bakery/entities/price_sheet.jadpo:

use bakery/types { SupplierName }/// A supplier's price sheet, kept as it was sent.entity PriceSheet {    id: Uuid    supplier: SupplierName    sheet: File {        store: sheets        max_size: 32b        types: [FileType.csv]    }    policy: {        create: [Access.public]        read: [Access.public]    }}

domains/bakery/actions/keep_prices.test.jadpo:

use bakery/entities { PriceSheet }use bakery/types { SheetText }test "a short sheet is kept" {    var price_sheet: PriceSheet = attempt keep_prices("Millers", "flour,1.20\nrye,2.40\n")    assert price_sheet.supplier == "Millers"}test "a longer sheet isn't kept" {    var sheet: SheetText = "flour,1.20\nrye,2.40\nspelt,3.10\noats,0.90\n"    var price_sheet: PriceSheet = attempt keep_prices("Millers", sheet)    assert price_sheet.supplier == "Millers"}

FAIL  a longer sheet isn't kept --> domains/bakery/actions/keep_prices.test.jadpo:11  fault: file_refused in action keep_prices  at: domains/bakery/actions/keep_prices.jadpo:11:12   |11 |     return PriceSheet.create {   |            ^^^^^^^^^^^^^^^^^^^ not kept  reason: `PriceSheet.sheet` holds files of at most 32 bytes, and this one is 41 bytes; a field keeps only a file that fits its settings, so the write stopped before it ran and nothing the action wrote was kept.  next: Bound the bytes before they are put, such as by the `max_length:` of the text they are made from, or raise `PriceSheet.sheet`'s `max_size:` (agent)  stack: domains/bakery/actions/keep_prices.jadpo:11       action keep_prices         domains/bakery/actions/keep_prices.test.jadpo:11  test "a longer sheet isn't kept"  rule: test.fault  rerun: jadpo test domains/bakery/actions/keep_prices.test.jadpo:11 --seed 1test failed: 1 failed, 1 passed, 2 action tests

store.read(file)

store.read reads at most 10 MB into memory. Reading a field whose max_size is over that is type.file_read_too_large, where it is read. A File whose field isn't known, such as a parameter of type File, is held to the limit as it runs, and going over it is a fault. A file whose bytes are gone from its store is a fault too.

FileContents is the file's bytes, and a program reads none of its parts. It goes to a provider: an operation's input: holding one sends its Base64 in the JSON, as Bytes is sent.

Two other places can hold one on its way there:

• An event's field, so a subscriber can pass the bytes on. They are kept in the outbox, as Base64 in the event's JSON, until the event is deleted (What the tick deletes): 24 hours after it was emitted once every delivery has succeeded, and for as long as a delivery has failed. An event's JSON is at most 64 KB, so emit faults on a larger file.
• A failure's internal: field, which stays on the server, so the caller's answer holds none of the bytes.

Nothing else takes it:

Each holds however deep, but for an email. A provider's own bytes come as Bytes, which put keeps.

Like put, read is called only in an action, before its first read or write. A test reaches files through the application's actions.

The disk driver in jadpo dev

jadpo dev keeps every store of files under .jadpo/files/<connection>/, whatever its driver, so nothing leaves your machine:

• Each file is kept under its key, sharded into two directory levels by its id. A key of any other shape is refused, and so is a symbolic link anywhere on the way.
• A file is written to a temporary file readable by the application's user only, synced, and renamed into place, and its folder is synced.
• A full disk answers storage_full, and so does a put below the store's free_space_floor:, as it does in production.

Its reads and writes block, a window of at most a megabyte at a time, on your own disk.

Files in a test

In a test, one in-memory store serves every connection (Provider calls and capabilities in tests), and a file helper's run reads two rows a batch. fixture_file, and telling the store to fail from a test's source, come with the testing fakes (What is built).