# Monitoring

**Partly built.** Missing: telemetry, recording the path each request takes, and `jadpo traces`, `metrics`, `logs` and `path`. See [What is built](/docs/built-in-services-built.md).

A threshold says when something your app does is too slow, too frequent
or failing too often, such as a route's p95 duration. You declare
thresholds in your app's source, in a monitoring file for each domain, so
they are reviewed with the code and versioned with each build rather
than configured in a monitoring tool.

`jadpo check` holds each threshold to what it watches and writes rules for
your monitoring tool. Nothing sends those metrics yet, since `telemetry:`
isn't built ([Telemetry](/docs/telemetry.md)), so a threshold today is checked, reviewed
and turned into rules, and doesn't yet raise an alert by itself.

| Page | What it covers |
|---|---|
| [Telemetry](/docs/telemetry.md) ○ not built | The generated runtime reports what it does as OpenTelemetry traces, metrics and logs, the open standard most monitoring tools accept. |
| [Path IDs](/docs/path-ids.md) ○ not built | Every request records the path it took through your app: the route, each action and query, and each branch taken inside them. |

## Writing a threshold

A domain's thresholds are its `thresholds { }`, in
`domains/<domain>/monitoring.jadpo`. Each is `name: { }` giving one
thing it watches and its limits, and its `///` line, what it protects,
is the summary of the alerts made from it. Here one threshold holds an action's
p99 to 50 ms.

This builds on a project from [`jadpo new`](/docs/the-jadpo-command.md#jadpo-new), which has
a `hello` domain with an action called `greet` and a route, `POST /hello`, that
runs it. Add this one file. `thresholds { }`, `watches:` and `p99:` are built
into Jadpo. `greeting` is your name for the threshold, and the `use` line
imports `jadpo new`'s `greet`.

`domains/hello/monitoring.jadpo`:

```jadpo
use hello/actions { greet }

thresholds {
    /// Working out a greeting stays quick.
    greeting: {
        watches: greet
        p99: 50ms
    }
}
```

A threshold can watch a route instead, and hold it to several limits at once.
This one watches the `POST /hello` route.

`domains/hello/monitoring.jadpo`, with `greeting_route` new:

```jadpo
use hello/actions { greet }

thresholds {
    /// A visitor is greeted quickly.
    greeting_route: {
        route: {
            method: HttpMethod.post
            path: "/hello"
        }
        duration: 800ms
        queries: 2
        p95: 300ms
        server_failures: 1%
        within: 5m
    }

    /// Working out a greeting stays quick.
    greeting: {
        watches: greet
        p99: 50ms
    }
}
```

What it watches is in its own domain:

- `route:` names a route by its `method:` and its path as `jadpo
  routes` prints it, placeholders by name, as a route test names one;
- `watches:` names an action, query, schedule or subscriber, imported
  like any name the file uses. Their names are unique across the
  project, so the name says which.

Its limits, each an upper bound:

| Limit | Kind | Watches | Value |
|---|---|---|---|
| `duration:` | Per request | Anything | The longest one request or run may take; a route's is shorter than its `deadline:` |
| `queries:` | Per request | All but a query | The most queries one may run |
| `p50:`, `p90:`, `p95:`, `p99:` | Across requests | Anything | That share of the requests within `within:` takes at most this long |
| `server_failures:` | Across requests | Anything | The share within `within:` that may end in a fault or a failure of a kind answered with a 5xx: `unavailable`, `timed_out`, `outcome_unknown`, `misconfigured` or `internal_fault`; never one a client caused |
| `oldest_waiting:` | Across requests | A subscriber | The age its oldest waiting delivery may reach |

`within:` is the window the across-request limits are judged over, a
whole number of minutes, hours or a day from `1m` to `1d`, and `5m`
when left out. A percentile is no shorter than a lower one beside it,
and `server_failures:` is above `0%` and below `100%`. A threshold that
names nothing in its domain is
[`threshold.subject_unknown`](/docs/errors/threshold.subject_unknown.md);
any other mistake in one, such as a limit that doesn't apply to what it
watches, is
[`threshold.declaration_invalid`](/docs/errors/threshold.declaration_invalid.md).

## Thresholds per request

These judge one request: its duration and its number of queries. They are
checked and kept in the runtime's plan, and nothing reads them yet. They come
into use with `telemetry:`, when the runtime will keep each request's spans in
memory until the request ends, then send:

- the full trace when a threshold was crossed, the request failed or
  faulted, or its path is new to this process;
- otherwise metrics only, plus one sample trace for each path each
  minute, so there is always a normal request to compare a bad one with.

`telemetry: { detail: }` sets both ([Telemetry](/docs/telemetry.md)):
`full_traces:` names the first list and `samples:` the second.

A process that crashes loses the requests it hasn't sent yet, and long
jobs send on a timeout. This keeps the volume low without losing
anything structural: the compiler knows every path, so the number of
per-path metrics can't grow without bound.

## Thresholds across requests

These judge many requests together: a p95 over a window, an error rate,
the oldest waiting delivery. One instance can't judge them for a fleet,
so `jadpo check` writes rules for your monitoring tool into
`build/monitoring/`: `prometheus.yaml` and `openslo.yaml`. The build manifest
carries the same thresholds, so a tool that reads it needs no generated rules.
You load the files into your tool yourself. They are generated, so a change
belongs in the thresholds, not in the file.

The `p99:` above becomes this rule in `build/monitoring/prometheus.yaml`:

```yaml
- alert: hello_greeting_p99
  expr: "histogram_quantile(0.99, sum by (le) (rate(jadpo_run_duration_seconds_bucket{jadpo_kind=\"action\", jadpo_name=\"hello.greet\"}[5m]))) > 0.05"
  labels:
    jadpo_threshold: "hello.greeting.p99"
  annotations:
    summary: "Working out a greeting stays quick."
    description: "hello.greet p99 above 50ms within 5m"
```

A disk store's `free_space_floor:` ([The disk driver](/docs/files.md#the-disk-driver))
is one of these too: the rules raise an alert when the store's disk has
less free than its floor, as puts start answering `storage_full`.

## Setting and changing them

- **Loosening needs a person.** Raising or removing a threshold is how
  an agent would silence an alert, so it needs an
  [acceptance](/docs/acceptances.md) and appears in
  the approval report. Tightening one doesn't. `jadpo check` compares
  each threshold with the one of the same name on the before side, read
  from git: a limit raised, or a longer `within:`, is
  `planning.todo_list.p95: ThresholdChange.raised`; a limit gone is
  `ThresholdChange.removed` on the limit, and a threshold gone, or
  watching something else, on the threshold, `planning.todo_list`; a
  disk store's floor lowered is `uploads.free_space_floor:
  ThresholdChange.lowered`. A new threshold, a lower limit and a higher
  floor need nothing.
- **Per build.** Thresholds belong to a build.

Not built yet: starting values worked out from the query report and from
production, an agent proposing changes from what it observes through the MCP
server ([The developer console and MCP](/docs/the-developer-console-and-mcp.md)),
and metrics for each flag value's path ([Path IDs](/docs/path-ids.md)).

## Details

The exact rules, for when you hit an edge case.

| Page | What it covers |
|---|---|
| [What Jadpo generates from thresholds](/docs/monitoring-thresholds-generated.md) | `jadpo check` reads each domain's thresholds and writes rules a monitoring tool can load. |