[abrichr]

Stacked on #1303. This branch starts at #1303's head, so until #1303 merges the diff below also shows #1303's changes. This PR adds only:

• packages/flow/benchmark/silent_saves/ (new)
• the silent-saves command in packages/flow/openadapt_flow/__main__.py
• one entry in CAPABILITIES.md's command list
• two entries in packages/flow/public-artifacts.json (fault_set.json, baselines.json), from scoreboard/refresh_inventory.sh

What it is

A standalone benchmark that any GUI agent or harness can be scored on. It measures how often a run reports success while the system of record holds a wrong, partial, duplicate or missing save, or has lost another record. It reports task success, time and cost alongside.

cd packages/flow
openadapt-flow silent-saves --agent screen-script             # $0
openadapt-flow silent-saves --agent screen-script --readback
openadapt-flow silent-saves --agent command:'node my_agent.js'

• One runner. --agent takes screen-script ($0 reference that trusts the screen), claude-computer-use (the scoreboard's agent; refuses without --allow-paid), openadapt-program:<jit cache>, command:<shell command> (any language: the task arrives in env vars and the report goes back as JSON), or python:<module>:<class>. --readback gates the claim on openadapt-flow verify-run. Each item is scored from the store read before and after; the executor's report never decides the verdict.
• Fixed, versioned fault set (fault_set.json, 1.0.0, protocol silent-saves/2026-10-19.1): ok, no_persist, partial, duplicate, wrong_record, stale, optimistic. These are the claims run's kinds, and a test checks that each one appears in EFFECT_E2E.md and claims.py. Nothing comes from the private corpus or tuned thresholds.
• Fault hidden from the executor. The executor opens http://127.0.0.1:<port>/#tasks. A proxy sets the fault in MockMed's script and forces it onto every save, so an executor that posts to the API directly still gets the fault. The proxy also blocks reset and delete.
• Results from committed scoreboard rows only (baselines.py → baselines.json; no new paid runs). This is detection: every fault in 1.0.0 happens on the server after the save is sent, so the question is whether the run reports it truthfully. Prevention (stopping before a wrong save) is not measured by 1.0.0, and the README says so. The program's two fault items are detection too.

Executor	Bad saves reported as success	Clean saves done	Correct saves reported as failed	Seconds per item (median)	Cost per item	Source
Agent alone (Claude Sonnet 5.5 computer use)	30 of 30	5 of 5	0 of 5	untimed	$0.0360	scoreboard/raw/mockmed/claims/2026-10-19-b
Same agent, success gated on OpenAdapt's read-back (verify-run)	0 of 30	5 of 5	0 of 5	untimed	$0.0360	scoreboard/raw/mockmed/claims/2026-10-19-b
Agent alone, scoreboard arm 1 (2 fault items)	2 of 2	47 of 47	0 of 47	untimed	$0.0386	scoreboard/raw/mockmed/agent/2026-10-18-a
OpenAdapt program, scoreboard arm 3 (2 fault items)	0 of 2	47 of 47	0 of 47	untimed	$0.0000	scoreboard/raw/mockmed/program/2026-10-18-a

verify-run refuted all 30 bad saves the agent reported as successes, after they had landed, and confirmed all 5 clean saves.

• README leads with the one-command runner, then the adapter contract, metric definitions, protocol and caveats: synthetic faults, small n, MockMed shows "saved" before the server answers, one app, the fault name visible to an executor that reads page source.

Proof and reference runs

• proof/*.png (rendered by proof.py): four claims-run items side by side. Each shows the agent's final message, verbatim, next to the store before the run and what it held after, with verify-run's verdict. The claims run kept no screenshots, so the screen panel is MockMed re-rendered locally with the same fault and note, and each image says so. proof/evidence.json holds the rows and verdicts taken from the run's verify-run baselines and receipts (the baseline salt is left out).
• results/: $0 runs of the built-in screen-script executor, 35 items each. On its own it reported 30 of 30 bad saves as successes; with --readback, 0 of 30, with 0 of 5 clean saves blocked.
• The openadapt-program adapter runs, but the scoreboard's admitted program was compiled from a page that sends an idempotency key, so its record check expects that key. Under this protocol (no key) it reports failure on clean saves. A trial with one item per fault (not committed) got 0 of 6 bad saves as success and 1 of 1 clean save as failed. A fair program row needs a program admitted under this protocol.

Apps

MockMed only. OpenEMR comes once the scoreboard session's injector is pushed. openIMIS comes after that. Both will run as external fixtures pulled at run time, with no GPL or AGPL files shipped.

Choices worth a look

• The command is openadapt-flow silent-saves, not bench silent-saves, because bench already takes a positional bundle path. The benchmark parses its own arguments (prefix_chars="+" passes them through) and runs from a source checkout only, since the wheel ships no benchmark/.
• benchmark/effectbench (MIT) already measures SWER over a simulated environment. This PR drives a real browser app, so a computer-use agent can be scored as is. The README links the two. Merging them is a follow-up.
• The benchmark's tests live in benchmark/silent_saves/tests. packages/flow's pytest testpaths does not collect them, so CI does not run them yet.

Tests

python -m pytest benchmark/silent_saves/tests -q (from packages/flow): 17 passed. This includes both browser tests (the screen-script reports all 6 bad-save kinds as successes; the read-back catches them) and a check that the README's results table matches baselines.json. Also passed: tests/test_capabilities_map.py, and ruff check on the new files. Before 11:30 only single test files were run, per the coordinator.

🤖 Generated with Claude Code

[abrichr]

Included in batch #1385, which runs CI once for several reviewed PRs. This PR is marked merged when #1385 merges. Please don't push here unless asked; a new commit leaves it out of the batch.