[abrichr]

Stacked on #1352 (itself stacked on #1303). Until those merge, the diff here includes them. This PR adds fault set 1.1.0 of silent-saves: repeated execution under UI change. It touches only packages/flow/benchmark/silent_saves/ and public-artifacts.json.

What 1.1.0 is

openadapt-flow silent-saves --fault-set 1.1.0 --agent <adapter>: the same MockMed task 100 times, with one UI change switched on per block, all MockMed's existing drift modes: renamed controls, display scaling, moved controls, a What's New screen, latency, a reordered queue, a moved New Encounter control, a look-alike referral, a relabeled type, and the target referral gone (where the right answer is to stop). Five items get a 1.0.0 server fault, including a duplicate that lands despite the page's save key. Versioned in fault_set_1.1.0.json.

Per arm it reports verified saves, silent wrong saves, false stops, items left for a person, human minutes per 100 items (only minutes a person logged; scripted approvals count as 0-minute events), and cost per verified item. Detection (server-fault bad saves reported as success) is kept apart from prevention (trap items where a wrong record landed).

Results (MockMed, 100 items per arm, 2026-10-19)

Arm	Verified saves	Silent wrong saves	False stops	Items left for a person	Human minutes per 100 items	Cost per verified item
1. Agent alone	90 of 90	5 of 100	0 of 90	5	0.0	$0.0433
1 + read-back (same runs)	90 of 90	0 of 100	0 of 90	10	0.0	$0.0433
2. Agent's first run under OpenAdapt	90 of 90	1 of 100	0 of 90	9	0.0 (93 scripted events)	$0.0140
3. Program	62 of 90	0 of 100	28 of 90	38	0.0	$0.0000
3r. Program with self-repair	62 of 90	0 of 100	28 of 90	38	0.0	$0.0077

Arm	Detection: server-fault bad saves reported as success	Prevention: trap items where a wrong record landed	Stopped when no correct save existed
1. Agent alone	5 of 5	0 of 25	5 of 5
1 + read-back (same runs)	0 of 5	0 of 25	5 of 5
2. Agent's first run under OpenAdapt	1 of 5	0 of 25	5 of 5
3. Program	0 of 5	0 of 25	5 of 5
3r. Program with self-repair	0 of 5	0 of 25	5 of 5

• Arm 1 adapted to every change and stopped when Jane Sample was gone, but reported all 5 server-fault bad saves as successes. Gated on verify-run, the same runs reported none of them, with no clean save blocked. That is detection; the rows had already landed.
• No arm wrote a wrong record on a trap item. The $0 screen-script reference, which clicks by position, wrote the wrong record on 20 of 25 trap items and reported every one as a success, so the traps do bite.
• Arms 3 and 3r stopped on every item of three blocks (display scaling, What's New, moved New Encounter). Self-repair changed none of those: it declined the display-scaling halts, and no proposal passed its probe for the other two. No approval was asked.
• Spend: $5.69 (arm 1 $3.90, arm 2 $1.31, arm 3r $0.48), within the approved caps. Every call is in each run's ledgers/; totals are in results/spend.json. Combined with the scoreboard's ledger, about $18.23, under the $20 stop.

Issues filed from these runs

• [arm 2, item 60] compile_agent recording runs are not gated by the read-back; a duplicate save got through silently #1376 (arm 2, item 60): compile_agent recording runs are not gated by the read-back; a duplicate got through silently.
• [arm 2, items 46-55] What's New dismiss click gets record_written from effect mining, so the canary refuses #1377 (arm 2, items 46-55): the What's New dismiss click gets record_written from effect mining, so the canary refuses.
• [arm 2, items 56-65] latency makes certification refuse navigation clicks (business-side effect not proven) #1378 (arm 2, items 56-65): latency makes certification refuse navigation clicks.

Runner changes

• compile_arm.py (arm 2) and repair_arm.py (arm 3r, a minimal MockMed SelfRepair target over the shipped ReplayTarget).
• spend.py: a locked, per-arm cost ledger that reads the scoreboard's ledger and stops before an arm's cap or a $20 combined total.
• A 300 s per-item watchdog that stops a hung Playwright driver. Arm 3 once hung for 2.5 hours at item 42; it was resumed from there.
• table_1_1.py builds the README tables from the runs, and a test fails if they drift.

Honest notes

• Arm 2's item 60 first ran without its fault: a bug in this benchmark, where scripted admission reset the fault. It was fixed and the item re-run alone. The invalid row is kept.
• Runs were in parallel on a loaded machine, so seconds are rough.
• One run per arm in a fixed order, one change per block, scripted approvals, and 5 fault items; the README caveats say what each limits.
• Since these runs, feat(flow): repair every stop that wrote nothing by default, and measure it #1360 (repair by default) repairs the moved New Encounter block (9 of 9) and display scaling (9 of 9), which changes the 3r result for those blocks. What's New stays open pending a person's classification ([arm 2, items 46-55] What's New dismiss click gets record_written from effect mining, so the canary refuses #1377).

Tests

python -m pytest benchmark/silent_saves/tests -q from packages/flow: 25 passed (including the browser tests). Also passed: ruff check on the package, and the prospect-name check.

🤖 Generated with Claude Code

[abrichr]

Included in batch #1385, which runs CI once for several reviewed PRs. This PR is marked merged when #1385 merges. Please don't push here unless asked; a new commit leaves it out of the batch.