Files

Export records to a file writes a file your app makes, start to finish.

To keep a file, such as a price sheet a supplier sends, give an entity a field of type File. The record holds the file's key, name, size and type, and the bytes live in a store of files. A store is a connection that provides the files capability, which today is the built-in disk driver, on the native target (The disk driver).

A bakery keeps its price sheets in a folder on its server. app.jadpo names the store:

application {    connections: {        sheets: {            driver: Driver.disk            path: "/var/lib/bakery/sheets"        }    }}

sheets is your name for the connection, and Driver.disk and its path: are built in.

domains/bakery/entities/price_sheet.jadpo keeps a file in that store. File and FileType are built in:

/// A supplier's price sheet, kept as it was sent.entity PriceSheet {    id: Uuid    supplier: Text {        min_length: 1        max_length: 80    }    sheet: File {        store: sheets        max_size: 5mb        types: [FileType.csv]    }    policy: {        create: [Access.public]        read: [Access.public, Access.system]    }}

The field's settings:

• store: names the connection, as it does for entities and caches. Two folders are two connections.
• max_size: is the largest file the field accepts. You must write it.
• types: lists the file types it accepts. Leave it out and any type goes.

A file is kept in the field only when it fits all three, which every write checks (What a write checks).

Writing a file your app makes: put

Your app writes a file with attempt sheets.put(bytes, FileType.csv), which returns a File to store in a field. put is an outside call, so you make it before the action's writes, while no transaction is open (Outside calls hold no transaction). In an action with several changing calls it is a keyed step, like a provider call, and its key is derived as any step's is, so a rerun writes the same object again (Several changing calls in one action). Bytes that are never attached to a record are deleted after a day.

Here sheets.put keeps the bytes before the action reads or writes anything, and gives the File that the record keeps. An action's parameter is always a named type, so SheetText is one of your app's.

domains/bakery/types/sheet_text.jadpo:

/// A price sheet as a supplier sends it.type SheetText = Text {    min_length: 1    max_length: 100000}

A store with no room answers storage_full, which only a failure of kind unavailable can map, and the action has to say which of your failures that is.

domains/bakery/failures/sheet_not_kept.jadpo:

/// The sheets store had no room for a sheet.type SheetNotKept = Failure {    kind: FailureKind.unavailable    message: "There was no room to keep the sheet."}

domains/bakery/actions/keep_prices.jadpo. Its use lines import from your app's other files, use app { sheets } the connection in app.jadpo:

use app { sheets }use bakery/entities { PriceSheet }use bakery/types { SheetText }use bakery/failures { SheetNotKept }/// Keeps a price sheet as it was sent.action keep_prices(    supplier: PriceSheet.supplier,    sheet_text: SheetText) fails SheetNotKept -> PriceSheet {    var sheet: File = attempt sheets.put(encoding.to_utf8(sheet_text), FileType.csv) {        storage_full: SheetNotKept    }    return PriceSheet.create {        set: {            supplier: supplier            sheet: sheet        }    }}

A test runs it against an in-memory store, so nothing touches the disk.

domains/bakery/actions/keep_prices.test.jadpo:

use bakery/entities { PriceSheet }test "a supplier's sheet is kept on its record" {    var price_sheet: PriceSheet = attempt keep_prices("Millers", "flour,1.20\nrye,2.40\n")    assert price_sheet.supplier == "Millers"}

Reading a file: read, and formats

A store moves bytes in and out and knows nothing of what is inside. Formats (CSV, TSV, JSON, JSON Lines) turn bytes into your app's types and back, whatever store the bytes came from. Your code joins the two:

• store.read(file) returns the file's contents as FileContents (Other built-in types), for passing the bytes on to a provider (store.read(file)). The format helpers don't accept it. It reads the whole file into memory, so it is capped: reading a field whose max_size is over the whole-read limit of 10 MB is a compile error pointing at the format helpers or batches.each.
• Rows in a file, of any size, are read and written only through the batch helpers: csv.read, csv.write, tsv, json_lines, json.read and json.write (Batches). Data formats lists their operations. There is no loop over a file's rows, and no parse of a whole file into a list. A provider's CSV report is put in a store and read the same way.
• json.read(text, as:) is only for single JSON documents in memory, such as request bodies and provider answers.

Media services, such as Cloudinary and Uploadcare, are ordinary packages. You call them through their own operations and pure functions, such as one that builds a transformation URL, because they can't do the whole files job.

Files in development and tests

jadpo dev keeps every store's files in a folder under .jadpo/files/, whatever the store's driver, so nothing leaves your machine. A test keeps them in memory (The disk driver in jadpo dev).

Deleting and erasing

Deleting the record, or clearing the field, deletes the bytes after the change commits, through a generated subscriber, because an action writes only one store (One store per action). A File field can be marked personal:. Erasure and retention then reach it like any field (Personal data).

Bytes go only once no record holds them, and never before the change that let go of them has committed:

• What counts. Deleting a record, a cascade from another record included, clearing its File field and setting the field to another file each let go of the file the field held. A File copied into a second record is held twice, and its bytes stay until both let go. A lifecycle's delete that only hides the record still holds the file.
• When. The commit that leaves a file held by no record also keeps an event in the outbox, and its delivery deletes the bytes, retried like any delivery. A crash after the commit leaves the event for the tick's sweep, so the bytes still go. A change that leaves no event, such as a purge, is found by the tick within a minute.
• Bytes a put kept. A put records its file as held by none before its bytes are sent. Bytes no record holds a day later, such as those of an action that failed after its put, are deleted by the tick.
• Never held again. Once its bytes are being deleted, a File can't be kept in a record: a write that tries stops with a fault, keeping nothing.
• A column or table dropped. A retired File column's rows, and a retired table's, still hold their files. A migration that drops the column or table lets go of each in the same transaction, and the tick deletes the bytes no record holds any more, as long as a File field still names their store of files (Removing data).
• One database per store of files. The holders are counted in the database whose records hold the files. Fields naming one store of files in entities of two databases are toolchain.not_built_yet, since one database's records couldn't see the other's.

The disk driver

Driver.disk keeps files on the server's own disk. It works in production on the native target only, and on Workers it is a compile error. The build manifest records that the app then runs on one server, since a second server wouldn't see the files: each deployable that reaches a disk store lists those stores as its one_server (The manifest).

app.jadpo:

application {    connections: {        uploads: {            driver: Driver.disk            path: "/var/lib/acme/uploads"            free_space_floor: 5%        }    }}

• Where. path: is the folder the store's files are kept in, written in full with no . or .. in it, and never inside the project's own folder, where git, jadpo build and jadpo dev would reach them: jadpo check refuses any other (connection.disk_path_invalid), following a link that leads into the project. As a deployable starts it makes the folder, readable by the app's user only, when it is missing, and doesn't build a store whose folder other users can write to, listing it in runtime.started's unbuilt, since they could plant files it reads; so is a store with no path:, answering misconfigured when used.
• Files never change. Each is kept under a key the runtime generates, a UUID sharded into two directory levels, never one derived from a file's name. The driver refuses any key that doesn't match its own pattern, opens each folder and file relative to the one above it without following symbolic links, reads only a regular file, and writes folders and files readable by the app's user only.
• Space. A write that runs out of space fails as storage_full. free_space_floor:, a percentage from 0% up to, but not, 100%, 5% by default, is the share of the disk kept free: a put, an upload or a join that would leave less free than it answers storage_full and writes nothing, so imports and the database aren't starved of space. The floor is a monitoring threshold.
• Commands. jadpo files ls, show, get, put and copy aren't built yet. jadpo files copy will move a store's files to another, which is how you will move off one server. Until then, jadpo files is cli.unknown_command.

A deployable built for Workers that uses Driver.disk is refused (deployable.target_unsupported). jadpo dev's disk driver answers a full disk with storage_full.

Not built yet

The rest of this topic is the design for what comes next, and none of it works today. Today your app writes files with put and keeps them on the disk driver. A client's file can reach your app only inside a request, within the 1 MB limit on a request body (What is refused), and a client can't fetch a file back.

The S3 driver

driver: Driver.s3 is the built-in driver for any S3-compatible service. A File field naming one today is data.file_field_invalid, since it keeps no files yet. Cloudflare R2, Amazon S3, Backblaze B2, DigitalOcean Spaces and MinIO will all use it, each with its own host and settings, listed on the driver's page. No package wraps the driver with presets.

Uploading

Files go straight from the client to the store, not through your app, so the 1 MB limit on a request body (What is refused) isn't in the way:

1. An action calls the generated User.avatar_upload(), which returns an UploadSlot: a signed address valid for 15 minutes that accepts only max_size bytes.
2. The client sends the file to that address.
3. The client sends the slot's token in an ordinary create or update, in an Input field of type FileUpload. Before the write commits, the runtime checks that the file is there, its size, and its type from its first bytes rather than what the client claimed. A mismatch is invalid_value.

Asking for a slot needs the create or update grant on the entity (Grants). Uploads that are never attached are deleted after a day.

On the disk driver, an upload is streamed to a temporary file as it arrives. It is counted, with a 413 at the first byte over max_size, hashed and type-checked, then synced and renamed into place, and the directory is synced.

Large uploads go in parts

A field whose max_size is over 100 MB gets a multipart slot instead of a single address, the same shape on S3, R2 and disk. The slot names a part size, with equal parts as R2 requires, and gives an address for each part: a signed UploadPart address on S3 and R2, and PUT /files/<store>/uploads/<slot>/parts/<n> on disk. The client sends the parts, and may resend any part after a dropped connection. Attaching the FileUpload completes the upload and runs the same size and type checks. A generated client helper hides the parts.

POST /imports/slot                          an app route calling CustomerImport.sheet_upload()<- 200 { "token": "upl_…", "part_size": 16777216, "parts_url": "/files/uploads/upl_…/parts" }PUT /files/uploads/upl_…/parts/1            16 MiB<- 204PUT /files/uploads/upl_…/parts/2            the connection drops; the client resends part 2<- 204POST /imports { "sheet": "upl_…" }          the ordinary create joins the parts, checks size and type<- 201

Compressed uploads

An upload sent with a Content-Encoding is refused. A gzip file, such as .csv.gz, is read only by a batch helper given max_unpacked_size:, and going past that size fails the file, so a small upload can't unpack to gigabytes. That reading is built (Reading gzip). Jadpo unpacks no other compression and no archives, such as zip.

Downloading

An Output that picks a File field sends a signed link valid for five minutes, with the file's name, size and type. The read has already passed policy, field restrictions included, so there is no separate download operation to protect.

Downloads aren't built yet. Until they are, a File in a route's output: is toolchain.not_built_yet, directly or through what it holds (What a File holds).

On the disk driver your app serves downloads from its own origin. It answers single-range Range requests and If-Range, with an ETag from the file's hash, and reads in fixed sizes, so a 2 GB download uses the memory a 2 KB one does. Because the origin is your app's own, every download is sent with Content-Disposition: attachment (except for an allow-list of image types), X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox. The client's file name only ever goes in filename* (RFC 6266).

Image variants and scanning

Both come when an app needs them.

• Image variants, such as a thumbnail, are declared on a File field, variants: { thumb: 200x200 }, and made by the runtime over any store, so no store needs to offer them and the files job stays the same everywhere. The runtime checks an image's pixel dimensions before decoding it. A variant is read as user.avatar.variants.thumb, a File that becomes a signed link in an Output like the original.
• Virus scanning is the scanning capability, named in application { } as scanning: { connection: clamav }. A ClamAV container package fills it (Services), or a scanning API's package. With it set, Jadpo scans every upload itself. A file stays in a scanning state, and can't be attached, until it passes. An infected file is refused like one of the wrong type, and deleted.

Details

The exact rules, for when you hit an edge case.