# Files

**Partly built.** Missing: uploads, downloads, the S3 driver and jadpo files. See [What is built](/docs/built-in-services-built.md).

[Export records to a file](/guides/guide-export-records-to-a-file.md) writes a file your app makes, start to finish.

To keep a file, such as a price sheet a supplier sends, give an entity a
field of type `File`. The record holds the file's key, name, size and
type, and the bytes live in a store of files. A store is a connection that
provides the `files` capability, which today is the built-in disk driver,
on the native target ([The disk driver](/docs/files.md#the-disk-driver)).

A bakery keeps its price sheets in a folder on its server. `app.jadpo`
names the store:

```jadpo
application {
    connections: {
        sheets: {
            driver: Driver.disk
            path: "/var/lib/bakery/sheets"
        }
    }
}
```

`sheets` is your name for the connection, and `Driver.disk` and its `path:` are
built in.

`domains/bakery/entities/price_sheet.jadpo` keeps a file in that store. `File`
and `FileType` are built in:

```jadpo
/// A supplier's price sheet, kept as it was sent.
entity PriceSheet {
    id: Uuid
    supplier: Text {
        min_length: 1
        max_length: 80
    }
    sheet: File {
        store: sheets
        max_size: 5mb
        types: [FileType.csv]
    }

    policy: {
        create: [Access.public]
        read: [Access.public, Access.system]
    }
}
```

The field's settings:

- `store:` names the connection, as it does for entities and caches. Two
  folders are two connections.
- `max_size:` is the largest file the field accepts. You must write it.
- `types:` lists the file types it accepts. Leave it out and any type goes.

A file is kept in the field only when it fits all three, which every
write checks ([What a write checks](/docs/files-put-read-and-the-disk-driver-in-development.md#what-a-write-checks)).

## Writing a file your app makes: `put`

Your app writes a file with `attempt sheets.put(bytes, FileType.csv)`,
which returns a `File` to store in a field. `put` is an outside call, so you make it before the
action's writes, while no transaction is open
([Outside calls hold no transaction](/docs/all-or-nothing-actions-and-concurrent-changes.md#outside-calls-hold-no-transaction)).
In an action with several changing calls it is a keyed step, like a
provider call, and its key is derived as any step's is, so a rerun writes
the same object again
([Several changing calls in one action](/docs/several-changing-calls-in-one-action.md)).
Bytes that are never attached to a record are deleted after a day.

Here `sheets.put` keeps the bytes before the action reads or writes
anything, and gives the `File` that the record keeps. An action's parameter is
always a named type, so `SheetText` is one of your app's.

`domains/bakery/types/sheet_text.jadpo`:

```jadpo
/// A price sheet as a supplier sends it.
type SheetText = Text {
    min_length: 1
    max_length: 100000
}
```

A store with no room answers `storage_full`, which only a failure of kind
`unavailable` can map, and the action has to say which of your failures that is.

`domains/bakery/failures/sheet_not_kept.jadpo`:

```jadpo
/// The sheets store had no room for a sheet.
type SheetNotKept = Failure {
    kind: FailureKind.unavailable
    message: "There was no room to keep the sheet."
}
```

`domains/bakery/actions/keep_prices.jadpo`. Its `use` lines import from
your app's other files, `use app { sheets }` the connection in `app.jadpo`:

```jadpo
use app { sheets }
use bakery/entities { PriceSheet }
use bakery/types { SheetText }
use bakery/failures { SheetNotKept }

/// Keeps a price sheet as it was sent.
action keep_prices(
    supplier: PriceSheet.supplier,
    sheet_text: SheetText
) fails SheetNotKept -> PriceSheet {
    var sheet: File = attempt sheets.put(encoding.to_utf8(sheet_text), FileType.csv) {
        storage_full: SheetNotKept
    }
    return PriceSheet.create {
        set: {
            supplier: supplier
            sheet: sheet
        }
    }
}
```

A test runs it against an in-memory store, so nothing touches the disk.

`domains/bakery/actions/keep_prices.test.jadpo`:

```jadpo
use bakery/entities { PriceSheet }

test "a supplier's sheet is kept on its record" {
    var price_sheet: PriceSheet = attempt keep_prices("Millers", "flour,1.20\nrye,2.40\n")
    assert price_sheet.supplier == "Millers"
}
```

## Reading a file: `read`, and formats

A store moves bytes in and out and knows nothing of what is inside.
Formats (CSV, TSV, JSON, JSON Lines) turn bytes into your app's types and
back, whatever store the bytes came from. Your code joins the two:

- `store.read(file)` returns the file's contents as
  `FileContents` ([Other built-in types](/docs/built-in-types.md#other-built-in-types)),
  for passing the bytes on to a provider
  ([`store.read(file)`](/docs/files-put-read-and-the-disk-driver-in-development.md#storereadfile)). The format helpers don't accept it. It reads
  the whole file into memory, so it is capped: reading a field whose
  `max_size` is over the whole-read limit of 10 MB is a compile error
  pointing at the format helpers or `batches.each`.
- Rows in a file, of any size, are read and written only through the
  batch helpers: `csv.read`, `csv.write`, `tsv`, `json_lines`,
  `json.read` and `json.write`
  ([Batches](/docs/batches-work-over-many-items.md)).
  [Data formats](/docs/data-formats-json-csv-tsv-and-json-lines.md)
  lists their operations. There is no loop over a file's rows, and no
  parse of a whole file into a list. A provider's CSV report is put in a
  store and read the same way.
- `json.read(text, as:)` is only for single JSON documents in memory,
  such as request bodies and provider answers.

Media services, such as Cloudinary and Uploadcare, are ordinary packages.
You call them through their own operations and pure functions, such as
one that builds a transformation URL, because they can't do the whole
`files` job.

## Files in development and tests

`jadpo dev` keeps every store's files in a folder under `.jadpo/files/`,
whatever the store's driver, so nothing leaves your machine. A test keeps
them in memory
([The disk driver in `jadpo dev`](/docs/files-put-read-and-the-disk-driver-in-development.md#the-disk-driver-in-jadpo-dev)).

## Deleting and erasing

Deleting the record, or clearing the field, deletes the bytes after the
change commits, through a generated subscriber, because an action writes
only one store ([One store per action](/docs/one-store-per-action.md)).
A `File` field can be marked `personal:`. Erasure and retention then
reach it like any field ([Personal data](/docs/personal-data.md)).

Bytes go only once no record holds them, and never before the change
that let go of them has committed:

- **What counts.** Deleting a record, a cascade from another record
  included, clearing its `File` field and setting the field to another
  file each let go of the file the field held. A `File` copied into a
  second record is held twice, and its bytes stay until both let go. A
  lifecycle's `delete` that only hides the record still holds the file.
- **When.** The commit that leaves a file held by no record also keeps
  an event in the outbox, and its delivery deletes the bytes, retried
  like any delivery. A crash after the commit leaves the event for the
  tick's sweep, so the bytes still go. A change that leaves no event,
  such as a purge, is found by the tick within a minute.
- **Bytes a put kept.** A `put` records its file as held by none before
  its bytes are sent. Bytes no record holds a day later, such as those of
  an action that failed after its `put`, are deleted by the tick.
- **Never held again.** Once its bytes are being deleted, a `File` can't
  be kept in a record: a write that tries stops with a fault, keeping
  nothing.
- **A column or table dropped.** A retired `File` column's rows, and a
  retired table's, still hold their files. A migration that drops the
  column or table lets go of each in the same transaction, and the tick
  deletes the bytes no record holds any more, as long as a `File` field
  still names their store of files
  ([Removing data](/docs/migrations.md#removing-data)).
- **One database per store of files.** The holders are counted in the
  database whose records hold the files. Fields naming one store of files
  in entities of two databases are `toolchain.not_built_yet`, since one
  database's records couldn't see the other's.

## The disk driver

`Driver.disk` keeps files on the server's own disk. It works in
production on the native target only, and on Workers it is a compile
error. The build manifest records that the app then runs on one server,
since a second server wouldn't see the files: each deployable that
reaches a disk store lists those stores as its `one_server`
([The manifest](/docs/the-manifest.md)).

`app.jadpo`:

```jadpo
application {
    connections: {
        uploads: {
            driver: Driver.disk
            path: "/var/lib/acme/uploads"
            free_space_floor: 5%
        }
    }
}
```

- **Where.** `path:` is the folder the store's files are kept in,
  written in full with no `.` or `..` in it, and never inside the
  project's own folder, where git, `jadpo build` and `jadpo dev` would
  reach them: `jadpo check` refuses any other
  ([`connection.disk_path_invalid`](/docs/errors/connection.disk_path_invalid.md)),
  following a link that leads into the project. As a deployable starts it
  makes the folder, readable by the app's user only, when it is missing,
  and doesn't build a store whose folder other users can write to,
  listing it in `runtime.started`'s `unbuilt`, since they could plant
  files it reads; so is a store with no `path:`, answering
  `misconfigured` when used.
- **Files never change.** Each is kept under a key the runtime
  generates, a UUID sharded into two directory levels, never one derived
  from a file's name. The driver refuses any key that doesn't match its
  own pattern, opens each folder and file relative to the one above it
  without following symbolic links, reads only a regular file, and
  writes folders and files readable by the app's user only.
- **Space.** A write that runs out of space fails as `storage_full`.
  `free_space_floor:`, a percentage from `0%` up to, but not, `100%`, 5%
  by default, is the share of the disk kept free: a put, an upload or a
  join that would leave less free than it answers `storage_full` and
  writes nothing, so imports and the database aren't starved of space.
  The floor is a [monitoring threshold](/docs/monitoring-thresholds.md#thresholds-across-requests).
- **Commands.** `jadpo files ls`, `show`, `get`, `put` and `copy` aren't
  built yet. `jadpo files copy` will move a store's files to another, which
  is how you will move off one server. Until then, `jadpo files`
  is `cli.unknown_command`.

A deployable built for Workers that uses `Driver.disk` is refused
([`deployable.target_unsupported`](/docs/errors/deployable.target_unsupported.md)).
`jadpo dev`'s disk driver answers a full disk with `storage_full`.

## Not built yet

The rest of this topic is the design for what comes next, and none of it
works today. Today your app writes files with `put` and keeps them on the disk
driver. A client's file can reach your app only inside a request, within the
1 MB limit on a request body ([What is refused](/docs/one-input.md#what-is-refused)),
and a client can't fetch a file back.

### The S3 driver

`driver: Driver.s3` is the built-in driver for any S3-compatible service. A
`File` field naming one today is `data.file_field_invalid`, since it keeps no
files yet. Cloudflare R2, Amazon S3, Backblaze B2, DigitalOcean Spaces and MinIO will
all use it, each with its own host and settings, listed on the driver's
page. No package wraps the driver with presets.

### Uploading

Files go straight from the client to the store, not through your app, so
the 1 MB limit on a request body ([What is refused](/docs/one-input.md#what-is-refused))
isn't in the way:

1. An action calls the generated `User.avatar_upload()`, which returns an
   `UploadSlot`: a signed address valid for 15 minutes that accepts only
   `max_size` bytes.
2. The client sends the file to that address.
3. The client sends the slot's token in an ordinary create or update, in
   an `Input` field of type `FileUpload`. Before the write commits, the
   runtime checks that the file is there, its size, and its type from its
   first bytes rather than what the client claimed. A mismatch is
   `invalid_value`.

Asking for a slot needs the `create` or `update` grant on the entity
([Grants](/docs/grants.md)). Uploads that are never attached are
deleted after a day.

On the disk driver, an upload is streamed to a temporary file as it
arrives. It is counted, with a `413` at the first byte over `max_size`,
hashed and type-checked, then synced and renamed into place, and the
directory is synced.

### Large uploads go in parts

A field whose `max_size` is over 100 MB gets a multipart slot instead of
a single address, the same shape on S3, R2 and disk. The slot names a
part size, with equal parts as R2 requires, and gives an address for each
part: a signed `UploadPart` address on S3 and R2, and
`PUT /files/<store>/uploads/<slot>/parts/<n>` on disk. The client sends
the parts, and may resend any part after a dropped connection. Attaching
the `FileUpload` completes the upload and runs the same size and type
checks. A generated client helper hides the parts.

```text
POST /imports/slot                          an app route calling CustomerImport.sheet_upload()
<- 200 { "token": "upl_…", "part_size": 16777216, "parts_url": "/files/uploads/upl_…/parts" }
PUT /files/uploads/upl_…/parts/1            16 MiB
<- 204
PUT /files/uploads/upl_…/parts/2            the connection drops; the client resends part 2
<- 204
POST /imports { "sheet": "upl_…" }          the ordinary create joins the parts, checks size and type
<- 201
```

### Compressed uploads

An upload sent with a `Content-Encoding` is refused. A gzip file, such as
`.csv.gz`, is read only by a batch helper given `max_unpacked_size:`, and
going past that size fails the file, so a small upload can't unpack to
gigabytes. That reading is built ([Reading gzip](/docs/reading-gzip.md)). Jadpo
unpacks no other compression and no archives, such as zip.

### Downloading

An `Output` that picks a `File` field sends a signed link valid for five
minutes, with the file's name, size and type. The read has already passed
policy, field restrictions included, so there is no separate download
operation to protect.

Downloads aren't built yet. Until they are, a `File` in a route's `output:`
is `toolchain.not_built_yet`, directly or through what it holds
([What a `File` holds](/docs/files-put-read-and-the-disk-driver-in-development.md#what-a-file-holds)).

On the disk driver your app serves downloads from its own origin. It
answers single-range `Range` requests and `If-Range`, with an ETag from the
file's hash, and reads in fixed sizes, so a 2 GB download uses the memory a
2 KB one does. Because the origin is your app's own, every download is sent
with `Content-Disposition: attachment` (except for an allow-list of image
types), `X-Content-Type-Options: nosniff` and
`Content-Security-Policy: sandbox`. The client's file name only ever goes in
`filename*` (RFC 6266).

### Image variants and scanning

Both come when an app needs them.

- **Image variants**, such as a thumbnail, are declared on a `File`
  field, `variants: { thumb: 200x200 }`, and made by the runtime over any
  store, so no store needs to offer them and the `files` job stays the
  same everywhere. The runtime checks an image's pixel dimensions before
  decoding it. A variant is read as `user.avatar.variants.thumb`, a
  `File` that becomes a signed link in an `Output` like the original.
- **Virus scanning** is the `scanning` capability, named in
  `application { }` as `scanning: { connection: clamav }`. A ClamAV
  container package fills it ([Services](/docs/running-a-service-in-a-container.md#services)),
  or a scanning API's package. With it set, Jadpo scans every upload
  itself. A file stays in a scanning state, and can't be attached, until
  it passes. An infected file is refused like one of the wrong type, and
  deleted.

## Details

The exact rules, for when you hit an edge case.

| Page | What it covers |
|---|---|
| [File fields, put, read and the disk driver](/docs/files-put-read-and-the-disk-driver-in-development.md) | A `File` field, `store.put` and `store.read` each refuse what doesn't fit. |
| [How an export writes its file](/docs/export-file-pieces.md) | A large export is written a page of rows at a time, and each page is a piece of the file. |
| [Reading gzip](/docs/reading-gzip.md) | Some uploads arrive as gzip files, such as a supplier's `.csv.gz`. |