conform.failed
A package's answers to jadpo package conform don't fit the capability it fills, or the suite couldn't run.
Area 14, Tools; not declinable; the next step is usually the agent's.
Why Jadpo refuses it
jadpo package conform <package> ai runs the ai capability's
conformance suite through the package's provide ai mapping: a typed
answer, a refusal, a cut-off, a stream, an embedding and the usage
reported. Without --live it answers from the package's recordings,
served by a local fake, so nothing leaves the machine. A case that fails
means the app would misread what the provider sends, such as taking an
answer the model stopped short as whole. So the suite fails, says which
case and why, and a package isn't published until it passes.
What it looks like
An app asking its models through its own provider package, Acme Models, whose mapping says how a refusal looks but not a cut-off:
config.jadpo:
config { acme_host: Url { default: "https://api.acme-models.example" } acme_key: Text { secret: true } drafting_model: ModelId { default: "acme/acme-small-1" }}
app.jadpo:
application { connections: { acme: { package: AcmeModels.v1 domains: [support] credential: config.acme_key host: config.acme_host } } ai: { connection: acme models: { default: config.drafting_model } limits: { per_day: { tokens: 2000000 } } }}
The package's recording of messages.create is in its recordings/
folder, which the local fake answers from:
mkdir -p shared/providers/acme_models/v1/recordingscat > shared/providers/acme_models/v1/recordings/messages.create.json <<'EOF2'{ "stop_reason": "{Text}", "text": "{Text}", "usage": { "input_tokens": "{Int}", "output_tokens": "{Int}" }}EOF2jadpo package conform acme_models ai
error: `AcmeModels.v1`'s cut-off doesn't conform to `ai` reason: The mapping says nothing of a cut-off, so an answer stopped at `max_output:` would be read as whole. next: Correct `provides/ai.jadpo` against the provider's documentation and the package's recordings, then run it again (agent) rule: conform.failed help: https://jadpo.dev/docs/errors/conform.failedConformance suite typed answer passed decodes as ConformanceAnswer refusal passed refused, for its mark cut-off failed no cut_off_when: stream skipped AcmeModels.v1 maps no stream: embedding skipped AcmeModels.v1 maps no embed: usage passed 12 in, 7 out, 0 cachedai doesn't conform: 1 failed, 3 passed, 2 skipped, its recordings and a local fake
(agent) at the end of the next: line means your coding agent can make the fix itself.
A case without a recording to answer from is refused the same way, and
its next: line says to record it against the provider's sandbox; a
suite that couldn't start, for lack of memory or of a free loopback
port, says what to free.
How to fix it
If the provider marks a cut-off in its answer, as Acme's
stop_reason does, say how in the mapping:
provide ai { generate: { ... refused_when: { path: "stop_reason" equals: "refusal" } cut_off_when: { path: "stop_reason" equals: "max_tokens" } ... } ...}
The suite's cut-off case then passes, and an answer the model stopped at
max_output: is read as cut off rather than whole.
Related
Markdown for agents: /docs/errors/conform.failed.md.Built from Jadpo main at aee2e8f.