feat(flow): compile-agent — compile an agent's first verified run into a zero-token program

[abrichr]

openadapt-flow compile-agent "<task>" compiles an agent's first verified run: a computer-use agent does the task once against --url (it writes there, so point it at a test copy), OpenAdapt checks the record it wrote in the system of record, and compiles the run into a program. Later runs make no model calls. openadapt-flow compile-agent run <dir> runs it.

What it does

1. Run 1 in a harness (openadapt_flow/compile_agent/harness.py). Claude (claude-opus-5-5, the customer's own key) gets the page's structure as short text lines with element refs, not a screenshot every turn. It can ask for a screenshot, and gets one after a save. Every action goes through the JIT CaptureComputer, so the run is recorded and compiled like any JIT capture.
2. Save gate. Every action that may commit a write (a commit-labelled control of any role, an unreadable element, Enter, a typed line break, Space off a text field, any shortcut) is checked first: the record's identity (--on-screen "MRN P1") must appear as whole words in the control's own row, dialog, form or section. The harness keeps a screenshot as evidence and asks the hold when there is one. A save on the wrong record is never performed, and nothing is compiled.
3. Run-1 record check. The system of record is read when run 1 starts and after it ends. It passes only when exactly one record appeared, it has every --expect field, and every input landed in it.
4. Checks a person confirms. The change becomes record --verify proposals. A person confirms them (and the parameters); they become the program's verify.yaml.
5. Admission through the JIT lifecycle: canaries (values the operator supplies, since a canary is a real save), then an admitter key, a registered agent under a signed policy (self-admission when the driving agent is that registered agent and the policy allows it, feat(jit): let the recording agent admit its own draft when the policy allows it #1249), or a person at a terminal who is shown the steps and the canary results. Each canary also gets a verify-run receipt.
6. Emit an Agent Skill, an MCP server with one tool, and a one-line CLI command, all calling run_compiled.
7. Later runs: no model calls, every save read back, and a verify-run receipt against the confirmed checks.

Python API (openadapt_flow.compile_agent): compile_first_run(...), run_compiled(dir, **params), and FirstRunCompiler for the steps one at a time. ClaudeAgent(vision="selective" | "every_turn"), ScriptedAgent for no-model runs. Docs: packages/flow/docs/COMPILE_AGENT.md.

Measured (packages/flow/benchmark/compile_agent/)

Ten live trials per arm on MockMed, alternating order, a distinct note per trial, prompt caching on in both arms. Raw rows are committed. (Measured at 2d5e4c1, before the rename.)

Per run 1 (median)	Structure first	Screenshot every turn
Passed the record check	10 / 10	10 / 10
Screenshots sent to the model	1	15
Prompt tokens	10,332	76,053 (7.4x)
Cost	$0.028	$0.148 (5.3x)
Model time	8.9 s	11.6 s

Wall time mostly tracked the shared host's load (1-minute load average 60 to 480 during the run). At low load (trials 1 to 3), run 1 took 15 to 18 s structure first and 36 to 59 s every turn. Ten replays of the admitted programs: 10/10 VERIFIED, 0 model calls. Total spend under $2.30 of the $20 cap.

Tests

• tests/test_compile_agent.py (no browser): record check, commit labels, listing, scrubbing that keeps the task's own values, the Claude loop's token accounting, cost cap and refusal, every gated way to commit and the plain actions that are not, whole-word scoped identity, stale refs by row, the hold, the emitted MCP server.
• tests/e2e/test_compile_agent_e2e.py (MockMed, no network): the real ClaudeAgent loop with a fake client compiles run 1, the registered agent self-admits under its policy after two canaries, the replay makes 0 model calls with a VERIFIED receipt, compiling again does not call the agent and run 1 refuses to run on an admitted task, a replay on an app that files the note on another patient halts, an agent that opens the wrong patient is refused at the save with nothing written or compiled, and the CLI path (no terminal stops at draft, a person confirms at a terminal, an admitter key admits, compile-agent run). Added to the flow-critical CI job, which is the job that runs tests/e2e files.

Other changes

• CaptureComputer gains page, pointer and screenshot() (read-only).
• docs/JIT.md: the live-loop sketch now points at the shipped loop.
• docs/CAPABILITIES.md (flow): a "From a prompt to a program" section. The root CAPABILITIES.md does not exist on main yet; docs: one map of what OpenAdapt can do #1260 adds it (with a test that every verb is listed); compile-agent goes into its command list once it is on main.

Web only, and says so

compile-agent runs web apps only in this PR. --backend windows|macos|linux|rdp|citrix, a --config whose backend.kind is not web, or a non-http(s) URL is refused with exit 2 before anything runs (UnsupportedTarget in Python). There is no silent browser fallback. Desktop, RDP and Citrix need the JIT to drive the record --backend backends and a non-DOM observer; that is left for the backend-parity work.

Limits

Listed in COMPILE_AGENT.md: web apps only; the gate recognizes commits by label and Enter in a form; on a wrong-record replay the verify-run receipt says HALTED_BEFORE_EFFECT rather than RECONCILIATION_REQUIRED until the guarded save (#1258) lands; canaries write test records; MockMed's demo idempotency key needs a reset between runs in tests.

Review

An independent full-diff review found gaps in the save gate (newline, Space, shortcuts, role=button, unreadable elements, whole-page substring identity), person admission filing the review for the person, the admitter stamped as recorder, and default canary values. All are fixed in 98a3223 with tests; the remaining items are listed as limits.

🤖 Generated with Claude Code