feat(flow): code-as-action agent for compile-agent run 1

[abrichr]

Stacked on #1277 (compile-agent). Merge that first; this PR then retargets to main.

What

ClaudeAgent(vision="code"): the run-1 agent acts through code instead of a screenshot after every click. It gets one run tool that takes a short script:

• read the structure: tree, find, text, read (the DOM of the page and every frame; a desktop accessibility tree; OCR text on RDP and Citrix);
• act by role and name: click, fill, select, press, scroll;
• screenshot only when it asks;
• sh: a read-only shell in a working directory (allow-listed text tools, | only, no path outside the directory, sandbox-exec with network denied on macOS), and save FILE to keep state on disk.

Every action is a structured step (element role, name and row; action; value) in code_action_trace.json, checked against the locator the recorder stored and then against the compiled program (check_trace_against_ir; steps distill dropped are reported as distilled). The identity the agent read before the save and the inputs it read back after it go to the person who confirms the record checks.

Design follows CUAWright (arXiv:2610.04116), OpenGCUA (arXiv:2610.22200) and ANIS (arXiv:2610.15765), with one difference: the script is not general code. Each line is a read or one recordable action, so every action goes through the save gate and becomes a program step. Docs: packages/flow/docs/CODE_AS_ACTION.md.

Save-gate fixes (all compile-agent agents)

Found running OpenEMR. The first three are also in #1277 (ported there):

• a Save inside an iframe was never gated; the harness now reads through the backend's frame chain;
• Enter with focus inside an iframe read the top page;
• a click on a "Postal code" field was gated as a save ("post");
• Enter on a native drop-down was gated as a save;
• a layout table row with many controls was taken as the record's scope.

Measured

Quotable figures will be scoreboard arm 7 (#1307), once the scoreboard's single runner has run it.

The earlier exploratory measurement in packages/flow/benchmark/code_as_action/ (its own runner, before the scoreboard) is kept as raw data: OpenEMR referral code 8/8 at $0.055 against the screenshot loop's 6/8 at $0.33 (both stops by the save gate, no wrong save); MockMed code $0.012, structure-first $0.014, screenshot loop $0.075. Spend there: $4.06 of its $15 cap.

Not done here

• compile-agent on desktop surfaces needs jit.SurfaceApp (feat(flow): compile_agent runs on Windows, macOS, Linux, RDP and Citrix apps #1283). The tool and the gate already work without a page (unit tests on a fake desktop surface).
• No native backend implements authoring_tree, so desktop listings fall back to OCR text; acting by exact role and name works through locate_structural.
• The root CAPABILITIES.md row waits for docs: one map of what OpenAdapt can do #1260 (the file is only there).

Tests

tests/test_code_action.py (shell sandbox, script language on a fake desktop surface, gate hints, trace check, the Claude loop with a fake client) and tests/e2e/test_code_action_e2e.py (MockMed through the whole pipeline with a scripted agent, wrong patient refused, a form inside an iframe), added to the flow-critical CI job. No paid calls.

🤖 Generated with Claude Code