Several changing calls through a package

Some jobs need several calls that each change something at a provider, in order: create a customer, attach a card, start a subscription. If the action stops halfway, a rerun must not do the first steps again. Jadpo records each step before it is sent, replays the recorded answers on a rerun, and stops the action as diverged if anything differs.

These rules apply to an app's action and to the package actions it calls, which count as one sequence. Several changing calls in one action says where such an action may run, what's allowed in it, and when it's warned; this topic adds what a package and its provider bring, and how a record ends.

• Words. A step is one outside call made by an action from its first changing call on, numbered along the path the action runs, package actions' calls included. The record is the rows holding an action's steps, keyed by the action's key. A step is in doubt when it was recorded as sent with no answer recorded.
• How many, and which. An action may make up to 8 changing calls in order, counted on its longest path with package actions' calls included. Each must be idempotent: true in its package: it takes the key, or it is safe to repeat as it is (a PUT that replaces, a DELETE). A changing call with neither stays the only changing call in its action. A package action over the limit cannot be published.
• The record. Before a step is sent the runtime records its position, operation, a fingerprint of its request (over the typed value, not the bytes) and the time sent; when the answer comes, success or failure, it records the answer as the provider sent it, encrypted, at most 1 MB (a larger one is a fault naming the step). Provider reads after the first changing step are recorded the same way. Database reads are not, and reads before the first changing step are not.
• Replay. A rerun replays answered steps in order without sending them, decoding each stored answer with the current package. At each step the operation and the fingerprint must match, and at the end every recorded step must have been replayed; otherwise the action stops as diverged before anything is sent, naming the step, its operation and the top-level request fields that differ. A diverged action is held for an operator, not failed: deploying matching code and retrying finishes it.
• Time. Once a record exists, clock.now is the time stored in it (the first run's start) for every rerun, including the rerun after a failed commit (Reads and writes). A resumed action writes timestamps from when it began, which is also when the provider acted. Before the first changing step a rerun takes a fresh time. The current time: clock.now holds the same.
• Keys and windows. Each step's key is derived as "Calling" says, the step number and key generation included. A keyed step in doubt inside its provider's window: (less a margin, the smaller of a tenth of the window and one hour) ends the run retry-later, and the next attempt sends it again under the same key. It parks as outcome_unknown when it has no key, when its window would pass, or when the package says replays_failures: true and the provider answered 5xx. Only the package's new_key_on: answer makes the runtime send again under the next key generation by itself. The window rule holds for single calls too. A resumable single call with a store already available keeps only its operation, first dispatch time, key generation and runner generation in the step tables, with no request fingerprint or answer and no steps encryption setting. It keeps a fresh code clock and its existing provider key; host time bounds retries. A saved record cannot switch between encrypted replay and dispatch metadata: its saved mode disagrees with the new build's, so it stops as diverged before sending anything, with the problem kept for the operator. A route with no store keeps no durable history across independent requests.
• One runner per record. Routes claim the client's key before the action runs (HTTP). A delivery's lease carries a generation that every record write checks, so a runner whose lease lapsed stops before sending another step. The provider's "key in use" answer means wait: retried within the call's budget, then retry-later, never a failure. The provider's "different request" answer is diverged, never misconfigured.
• Undo is code. There is no built-in compensation. An undo is a step like any other: it needs a key, gets the same warning, and a failure mapped as "already undone" (Stripe's resource_missing on a delete) counts as handled.
• Ending. The record is removed in the action's commit transaction, with the delivery's handled marker or the route's stored response. A record that ends any other way (failed, diverged, parked, or a route's key lapsing after 24 hours) becomes an operator item listing each step and its answer, with retry (--not-done to send the step in doubt under a new key once the operator has checked it did not happen) and abandon (end the action as failed, keeping the list of effects for tidying up by hand). It is kept at most 30 days, after which the answers are dropped and the step list stays.
• Personal data. Requests are kept only as fingerprints. Answers are encrypted with application { steps: { encryption_key: } } (Background work). Nothing links them to a person, so the personal-data report lists "provider answers for unfinished actions, at most 30 days", as it does for logs. A metric and a no-setup alert watch the number of unfinished records and the age of the oldest.
• Deploys. Each record keeps the build id it started under and a digest of its action's step shape (the operations on each path, package steps included). No deploy command is involved: a rerun under another build sees the different build id in the manifest, compares the step shape, and stops as diverged before sending anything if it changed. Code after the last recorded step, new steps after it, writes and failure mappings are not changes to the shape.
• Run limits. A run cut off with a record (a Workers limit, a deploy, a crash) resumes; only its step in doubt follows the window rule. The deadline warning sums the timeout times the attempts of every outside call on the longest path, a provider's operation and an ai call alike, against the route's deadline:. A single call is checked the same way.
• Tests and tools. Fakes keep keys as the provider does (the same key and request return the first answer; a different request returns the package's mismatch answer) and gain two scripted answers, lost (the provider acted and the answer never arrived) and in_use. jadpo test reruns every test that reaches a multi-step action with a crash at each step boundary (before sending, after sending but before recording, after recording) and checks that the end state and the fake's effects match the run without one; this is on by default. jadpo dev and jadpo deliveries show list each step as sent (with its key), answered, replayed or in doubt, and jadpo watch will once it is built.

$ jadpo deliveries show d_7Q2billing.subscribe, from subscriber start_subscriptionOutcome unknown at step 3 of 4  1  stripe.customers.create         answered     10:02:11  cus_P4x…  2  stripe.payment_methods.attach   answered     10:02:12  pm_1N…  3  stripe.subscriptions.create     no answer    10:02:12  Stripe answered 500 and repeats it for this key until 10:02 tomorrow  4  stripe.invoices.finalize        not reachedCheck in Stripe whether customer cus_P4x… has a subscription.  It doesn't:  jadpo deliveries retry d_7Q2 --not-done  It does:     jadpo deliveries abandon d_7Q2, then tidy up in Stripe

Calls without idempotency keys (GitHub, Twilio Messages, Postmark) stay one per action. A receipt sent after a charge belongs in a subscriber of the action's event, not as an unkeyed last step.

How the compiler keeps these rules

jadpo check keeps these rules with these errors: step.context, step.large_output, step.store_required, step.too_many, step.unkeyed, step.late_failure, effect.call_in_loop and route.calls_past_deadline. It reads an operation's large_output:. A provider's idempotency: window:, replays_failures: and new_key_on: are read into the plan, and an operation's changes: marks a step. The provider call records, replays and keys each step through the capability seam, whose exchange writes a step before it is sent.

Where else this is covered

• What an operator sees, and the metric and the alert on unfinished records: "What an operator sees" and "Several changing calls in one action" in Background work. Each step's state shows under its request's or delivery's line in jadpo dev.
• jadpo runs unfinished --build: Tools.
• The fakes and crash points: "Provider calls and capabilities in tests" in Testing.
• The choices behind these rules: the design notes for Jobs and events.