feat(flow): repair every stop that wrote nothing by default, and measure it

[abrichr]

This work was built stacked on #1325 (the bridge); #1325 has since merged, so this diff is against main alone.

Compiled programs stop rather than guess when the app changes. Before this PR, a stop waited for a person to start self-repair. Now repair is the default response to a stop, and its success rate is measured, failures included.

What changes

Repair by default (openadapt_flow.repair.auto_repair). When a deployment names a repair agent (a new repair: section in deploy.yaml, or --repair-agent), openadapt-flow run hands every stop that wrote nothing to it:

1. The agent gets the stop, the screen evidence and the record check. The record check is new on RepairRequest: each save's effect contracts with literal values masked, and its record identity as digests.
2. The agent proposes a re-anchored step, a bridge of up to three steps (feat(flow): self-repair a halt with a short bridge of new steps #1325), or a fix a signed policy pre-approves (feat(flow): standard changes - repairs a person approved in advance #1294).
3. Flow probes the fix up to the save and runs the replay and fault cases with the same read-back.
4. One person approves on the phone or at Desktop, unless a standard change covers the fix.
5. The item resumes. A worklist resumes through WorklistQueue.

How each stop is routed:

The run ended	The stop goes to
HALTED_BEFORE_EFFECT	Repair. A halt self-repair doesn't handle (a wrong record, for example) goes to a person.
RECONCILIATION_REQUIRED	Reconciliation: the change ledger and undo list. Never repair.
Anything else	A person.

How to turn it off, and what happens when a run can't be repaired:

• --no-auto-repair, or repair.auto: false, turns it off.
• In the default mode, a run that can't be repaired (pinned, managed, qualification-bound, or a worklist run) runs plainly and prints why repair is off for it.
• --self-repair still means "require repair", and then such a run is refused.

Every stop is one row in <store>/auto-repair/ledger.jsonl, and summarize() reports the funnel from stop to resumed item.

Program reuse bound to intent (jit/signature.py, jit/compile_agent.py). The admission manifest now pins two things:

• the record check's digest: the contracts of every saving step, with anchors left out so a repair keeps the digest;
• the canary's environment: the origin, plus an optional environment_id such as the app version, or a callable that reads it.

A run refuses a mismatch. A new app version at the same origin is refused instead of silently reusing the old program. TaskSignature(record_check=, environment=) puts both in the key. Keys without pins are unchanged, so existing caches keep working.

Measured (packages/flow/benchmark/auto_repair/)

The agent was a real Claude agent (claude-sonnet-5-5, one call per proposal). The approver was scripted and approves everything that reaches it, so a bad fix that passed the probe would have been saved. Total model spend was $0.99 in the committed runs (from the agent's own call logs), plus about $0.15 in uncommitted diagnostic probes.

Where	Stops	Reached repair	Proposed	Passed probe	Approved	Resumed and verified	Wrong saves	Person decisions
MockMed, 22 drift modes × 3 trials (66 runs)	30	21	12	6	6	6 / 21	0	6
MockMed worklist (relink, redesign; 5 items each; drift ships after the pre-flight)	10	10	2	2	2	10 / 10	0	2
OpenEMR, #1347's recorded runs (offline routing)	8	0				0
openIMIS #1331, before #1337 (offline routing)	3	0				0
openIMIS #1331 live, #1337 merged locally + scripted risk review of step_002/step_012	3	3	2	0	0	0	0	0

What these numbers show:

• Repaired: relink 3/3 and redesign 3/3. Each took one call ($0.013–0.015) and one tap.

• Not repaired, nothing written:

  • menu: the agent can't see inside the closed menu.
  • notice: the fix needs two clicks, and "Continue" reads as a possible save to the fail-closed classifier.
  • provider: the agent declined to choose a provider.
  • font and zoom: false stops from a pixel region_stable postcondition. No click fixes these.

  The first three need a look at more than one screen, which is feat(flow): search repair bridges on a staging copy, scored by read-back #1302's staging search.

• OpenEMR: all 8 recorded stops are RECONCILIATION_REQUIRED, because "Create New Patient" (step_039) counts as a save and had already acted. Repair can't help these; OpenEMR: identity check at 'Confirm Create New Patient' reads a background label behind the modal and refuses every 9-17 items #1347 needs its cause fixed.

• openIMIS: with fix(flow): halt a relabelled Save on the Save, not the scroll before it #1337 and the risk review, the relabelled Save halts on the Save before any effect and reaches repair, 3 of 3 (0 of 3 before). The agent found "SAVE CLAIM" in 2 of 3. No fix passed the probe: in the replay the Save is still disabled (grey) while the scroll looks for it, and the new template was cropped from the enabled button. Nothing was written.

• Person time: not timed (the approver was scripted). The measure is the count of decisions: one per repaired stop, or one per worklist group.

Silent-saves 1.1.0: the 28 items arms 3 and 3r always stopped on (#1380)

Re-run on the same admitted MockMed program, items and store judge (benchmark/auto_repair/measure_silent_saves.py, which needs #1380's benchmark/silent_saves):

Block	Items with no server fault	Correct in #1380 (3r)	Correct now	How
zoom (display scaling)	9	0	9	No stop: region_stable now also matches at the page scale its step's own click showed (both axes must agree; ±1%; same thresholds; a covered region still fails).
relink (moved New Encounter)	9	0	9	1 repair with one approval; the other 8 ran the repaired program.
notice (What's New)	10	0	0	Not fixed; see below.
Total	28	0	18	0 wrong saves. The server-fault items 30 and 80 still end flagged or stopped.

• relink: the agent had proposed the right control in bench: silent-saves 1.1.0, repeated runs under UI change, four arms (stacked on #1352) #1380, but a repaired copy of an admitted agent-compiled (JIT) program could never run. Its admission pins the digest, which every patch changes, so the probe failed with DraftNotAdmitted. A repair keeps the record check, so the admission now carries over (jit.lifecycle.admit_repair):

  • first repair_probe, which runs only behind a gate that stops before the save;
  • then repair, after the person's approval or a signed standard change, embedding the prior admission and verifying it again.

  A copy that changes the record check or the parameters is refused.

• zoom: a false stop on a correct screen at 125%. bench: silent-saves 1.1.0, repeated runs under UI change, four arms (stacked on #1352) #1380's older engine also declined the halt as not repairable.

• notice / [arm 2, items 46-55] What's New dismiss click gets record_written from effect mining, so the canary refuses #1377: the fix is two clicks, "Continue to tasks" then "Open". The fail-closed write classifier reads "Continue" as a possible save, so the patch is refused, and clicking it on a real record to find out is exactly what the probe avoids. The same classification is [arm 2, items 46-55] What's New dismiss click gets record_written from effect mining, so the canary refuses #1377: arm 2 attaches a placeholder record_written to that click, so the canary refuses. Both need a person's one-time classification of that control (as qualify set-risk does for openIMIS); neither is loosened here.

• Spend: 21 agent calls, $0.31. 10 of them were on notice, one per item: a single run repairs only its own stop, and WorklistQueue grouping would ask once.

Bugs the live openIMIS run found (fixed here)

• Patch failed to load. A re-targeted save that carries a submit_binding (openIMIS's icon-only Save, recorded on the visual path) got the DOM locator under the new point, and a step may not carry both. The patched bundle failed to load even though the agent found the right control. The patch now keeps such a step visual and keeps its binding.

• Probe couldn't open a session. The halted session stays open while a probe opens another, and two Playwright drivers can't share a thread. The tests hid this behind their fixture's shared browser. run now shares one browser for every session of a repair. A new test runs the CLI without the fixture's browser; it fails without the fix.

• Give-ups weren't counted. An agent that gave up returned None, so the call that decision took never reached the ledger. The MockMed ledger shows 18 calls and $0.24; the agent's own log has 30 calls and $0.43. An agent can now give up with an action-less proposal that still reports its calls.

Not done

• openIMIS repair still fails at the probe. The template comes from the enabled Save, so the probe can't find the disabled one. Repairing it needs a template that matches the control in either state, or a probe that waits for the control to become actionable. The openIMIS VM is stopped again.
• OpenEMR live repair runs: not attempted, because no recorded OpenEMR stop is one repair may take, and the scoreboard's fixture is reserved.
• fix(flow): halt a relabelled Save on the Save, not the scroll before it #1337 rebase. When fix(flow): halt a relabelled Save on the Save, not the scroll before it #1337 rebases onto main (after feat(flow): self-repair a halt with a short bridge of new steps #1325), its test_reviewed_risk_makes_the_halt_repairable must expect (step, "replace", result); the bridge changed _repairable's return value.

Tests

• New tests:
  • tests/test_auto_repair.py: routing, ledger and funnel.
  • tests/test_auto_repair_cli.py: live MockMed. It covers on-by-default from deploy.yaml, --no-auto-repair, and the plain-run fallback.
  • tests/test_jit_reuse_binding.py: 16 tests.
  • Ledger assertions in the WorklistQueue test in tests/test_standard_change.py.
• Existing suites pass locally: self-repair, bridge, self-repair CLI, standard changes, JIT (177), CLI config and deployment, repair lifecycle, and the capability map.

🤖 Generated with Claude Code

[abrichr]

Included in batch #1384, which runs CI once for several reviewed PRs. This PR is marked merged when #1384 merges. Please don't push here unless asked; a new commit leaves it out of the batch.