Monitoring

A threshold says when something your app does is too slow, too frequent or failing too often, such as a route's p95 duration. You declare thresholds in your app's source, in a monitoring file for each domain, so they are reviewed with the code and versioned with each build rather than configured in a monitoring tool.

jadpo check holds each threshold to what it watches and writes rules for your monitoring tool. Nothing sends those metrics yet, since telemetry: isn't built (Telemetry), so a threshold today is checked, reviewed and turned into rules, and doesn't yet raise an alert by itself.

Writing a threshold

A domain's thresholds are its thresholds { }, in domains/<domain>/monitoring.jadpo. Each is name: { } giving one thing it watches and its limits, and its /// line, what it protects, is the summary of the alerts made from it. Here one threshold holds an action's p99 to 50 ms.

This builds on a project from jadpo new, which has a hello domain with an action called greet and a route, POST /hello, that runs it. Add this one file. thresholds { }, watches: and p99: are built into Jadpo. greeting is your name for the threshold, and the use line imports jadpo new's greet.

domains/hello/monitoring.jadpo:

use hello/actions { greet }thresholds {    /// Working out a greeting stays quick.    greeting: {        watches: greet        p99: 50ms    }}

A threshold can watch a route instead, and hold it to several limits at once. This one watches the POST /hello route.

domains/hello/monitoring.jadpo, with greeting_route new:

use hello/actions { greet }thresholds {    /// A visitor is greeted quickly.    greeting_route: {        route: {            method: HttpMethod.post            path: "/hello"        }        duration: 800ms        queries: 2        p95: 300ms        server_failures: 1%        within: 5m    }    /// Working out a greeting stays quick.    greeting: {        watches: greet        p99: 50ms    }}

What it watches is in its own domain:

• route: names a route by its method: and its path as jadpo routes prints it, placeholders by name, as a route test names one;
• watches: names an action, query, schedule or subscriber, imported like any name the file uses. Their names are unique across the project, so the name says which.

Its limits, each an upper bound:

within: is the window the across-request limits are judged over, a whole number of minutes, hours or a day from 1m to 1d, and 5m when left out. A percentile is no shorter than a lower one beside it, and server_failures: is above 0% and below 100%. A threshold that names nothing in its domain is threshold.subject_unknown; any other mistake in one, such as a limit that doesn't apply to what it watches, is threshold.declaration_invalid.

Thresholds per request

These judge one request: its duration and its number of queries. They are checked and kept in the runtime's plan, and nothing reads them yet. They come into use with telemetry:, when the runtime will keep each request's spans in memory until the request ends, then send:

• the full trace when a threshold was crossed, the request failed or faulted, or its path is new to this process;
• otherwise metrics only, plus one sample trace for each path each minute, so there is always a normal request to compare a bad one with.

telemetry: { detail: } sets both (Telemetry): full_traces: names the first list and samples: the second.

A process that crashes loses the requests it hasn't sent yet, and long jobs send on a timeout. This keeps the volume low without losing anything structural: the compiler knows every path, so the number of per-path metrics can't grow without bound.

Thresholds across requests

These judge many requests together: a p95 over a window, an error rate, the oldest waiting delivery. One instance can't judge them for a fleet, so jadpo check writes rules for your monitoring tool into build/monitoring/: prometheus.yaml and openslo.yaml. The build manifest carries the same thresholds, so a tool that reads it needs no generated rules. You load the files into your tool yourself. They are generated, so a change belongs in the thresholds, not in the file.

The p99: above becomes this rule in build/monitoring/prometheus.yaml:

- alert: hello_greeting_p99  expr: "histogram_quantile(0.99, sum by (le) (rate(jadpo_run_duration_seconds_bucket{jadpo_kind=\"action\", jadpo_name=\"hello.greet\"}[5m]))) > 0.05"  labels:    jadpo_threshold: "hello.greeting.p99"  annotations:    summary: "Working out a greeting stays quick."    description: "hello.greet p99 above 50ms within 5m"

A disk store's free_space_floor: (The disk driver) is one of these too: the rules raise an alert when the store's disk has less free than its floor, as puts start answering storage_full.

Setting and changing them

• Loosening needs a person. Raising or removing a threshold is how an agent would silence an alert, so it needs an acceptance and appears in the approval report. Tightening one doesn't. jadpo check compares each threshold with the one of the same name on the before side, read from git: a limit raised, or a longer within:, is planning.todo_list.p95: ThresholdChange.raised; a limit gone is ThresholdChange.removed on the limit, and a threshold gone, or watching something else, on the threshold, planning.todo_list; a disk store's floor lowered is uploads.free_space_floor: ThresholdChange.lowered. A new threshold, a lower limit and a higher floor need nothing.
• Per build. Thresholds belong to a build.

Not built yet: starting values worked out from the query report and from production, an agent proposing changes from what it observes through the MCP server (The developer console and MCP), and metrics for each flag value's path (Path IDs).

Details

The exact rules, for when you hit an edge case.