feat(flow): code-as-action agent for compile-agent run 1

.github/workflows/packages.yml

@@ -398,6 +398,7 @@ jobs:
             tests/test_admitted_composition_runtime.py \
             tests/e2e/test_structural_action.py \
             tests/e2e/test_compile_agent_e2e.py \
+            tests/e2e/test_code_action_e2e.py \
             --basetemp=runs/ci-critical --junitxml=runs/critical-junit.xml
       - name: Fail if any critical test was skipped or none ran
         run: |

packages/flow/benchmark/code_as_action/README.md

@@ -0,0 +1,129 @@
+# Code as action against the screenshot loop
+
+> **Quotable figures come from the scoreboard** (`benchmark/scoreboard`, arm 7,
+> [#1307](https://github.com/OpenAdaptAI/OpenAdapt/pull/1307)), built by
+> `python -m scoreboard.build` from rows its single runner commits. This folder is the earlier,
+> exploratory measurement made before the scoreboard existed. It used its own
+> runner and is kept as raw data.
+
+Measured 2026-10-18 with `claude-sonnet-5-5` at effort `medium`, on the
+operator's own key, in the compile-agent harness. Every arm runs through the
+same harness, save gate, JIT capture and compiler; only what the agent sees
+and how it acts differ:
+
+- **code**: `ClaudeAgent(vision="code")`, one `run` tool that takes a script
+  ([docs/CODE_AS_ACTION.md](../../docs/CODE_AS_ACTION.md)).
+- **every_turn**: Claude's computer toolset (`computer_toolset_20260801`), a
+  screenshot after every action. The screenshot loop.
+- **selective** (MockMed only): the structural listing after every action, a
+  screenshot on request (compile-agent's default).
+
+Correct is decided by reading the system of record back, never by the agent's
+own report. Synthetic data only.
+
+**Load.** The host (12 cores) was shared with other sessions; its 1-minute
+load was between 115 and 420 at the start and end of every run, so every row is marked
+`under_load: true` and every time below was taken under load. Tokens, model
+calls, actions and cost do not depend on load.
+
+## MockMed (5 trials per arm, `results/mockmed/`)
+
+Task: "Open the referral for Jane Sample, start a Triage encounter, and save
+this note: {note}", with a distinct note per trial, arms in rotating order,
+MockMed reset before each run. Correct = the record check passed (exactly one
+new encounter, on patient p1, type Triage, with the note).
+
+| median per run | code | selective | every_turn |
+| --- | ---: | ---: | ---: |
+| Correct (read back) | 5 / 5 | 5 / 5 | 5 / 5 |
+| Model calls | 6 | 5 | 5 |
+| Actions | 6 | 6 | 13 |
+| Prompt tokens | 12,738 | 10,332 | 73,303 |
+| Output tokens | 350 | 359 | 533 |
+| Cost | $0.012 | $0.014 | $0.075 |
+| Model time | 5.9 s | 5.8 s | 9.7 s |
+| Run time (under load) | 21.5 s | 24.0 s | 26.0 s |
+| Screenshots sent | 0 | 1 | 14 |
+
+On a five-step task, code as action and the structural listing are about the
+same; both read about a sixth of the screenshot loop's tokens and cost about
+a sixth as much. The program compiled from a code-arm run was admitted and
+replayed twice: VERIFIED both times, 0 model calls (`replays.jsonl`).
+
+## OpenEMR referral (8 referrals per arm, `results/openemr/`)
+
+The referral task from [openemr_referral_timing](../openemr_referral_timing/):
+register a referred, made-up patient from 12 structured fields in OpenEMR's
+New Patient form (it sits in an iframe; a duplicate-check dialog opens in a
+second one), and do not create a duplicate. r6 matches an existing chart's
+name with a different birth date: correct means no new chart. Sign-in and the
+blank form are set up before the clock starts. One block per arm from a reset
+fixture (pinned image, `realrun` state), referrals r1 to r8 in order. Correct
+= SQL read-back by the test setup: exactly one new chart with 12 of 12 fields
+matching (r6: no new chart).
+
+| median per run | code | every_turn |
+| --- | ---: | ---: |
+| Correct (read back) | 8 / 8 | 6 / 8 |
+| Model calls | 12 | 14 |
+| Actions | 36 | 56 |
+| Prompt tokens | 90,777 | 619,498 |
+| Output tokens | 1,024 | 2,028 |
+| Cost | $0.055 | $0.332 |
+| Model time | 18.5 s | 48.6 s |
+| Run time (under load) | 45.4 s | 75.2 s |
+| Screenshots sent | 2 | 56 |
+
+- The code agent read 6.8 times fewer prompt tokens, cost 6 times less, and
+  took 36 actions to the screenshot loop's 56. It asked for a screenshot in
+  every run (1 to 4 each), mostly to find a section below the fold. Both
+  stopped correctly on r6.
+- The two screenshot-loop misses were stopped by the save gate before any
+  save: Enter on the "Contact" button (a `<button>` in a form submits it) and
+  a Ctrl+End shortcut. The gate refuses an action it can't rule out as a save
+  when the record identity isn't around it. Neither arm wrote a wrong or
+  duplicate chart.
+- In the earlier referral timing study (its own harness, no save gate), a
+  `claude-sonnet-5-5` screenshot agent saved 62 of 63 at a median $0.083 and
+  49 actions. The screenshot arm here cost more per run; it is a different
+  harness and toolset (`computer_toolset_20260801`, a screenshot after every
+  action), so the two are not a matched comparison.
+- The gate's identity for this task is "Create New Patient" in the scope of
+  the save (the confirm dialog): a new patient's identity is typed, not
+  shown, so this checks that the save happens in the new-patient dialog. It
+  is weaker than an MRN check and is the same for both arms.
+
+**Trace and program.** For each code run the recording was compiled and
+checked with `check_trace_against_ir`. In 4 of 8 runs the first version of
+the check reported 2 steps missing; in each, they were a pair of clicks that
+opened and closed the form's Contact section, which distill drops as a dead
+end. The check now reads `distill.json` and reports such steps as distilled.
+Two more code runs with the fixed check (`results/openemr-distill-check/`,
+r4 and r7, from a reset fixture): both correct with 12 of 12 fields, both
+pass, and r4 shows one step distilled as a dead end. That r4 run also shows
+the spread: 26 model calls, 8 failed lines, 8 screenshots and $0.148, against
+11 calls and $0.048 for r4 in the main block.
+
+**Set aside.** `results/openemr/superseded.jsonl` holds three runs from
+blocks stopped to fix save-gate bugs that refused the screenshot agent (see
+`SUPERSEDED.md`). They are not in the tables.
+
+**Redacted.** During these runs the page URL the tool printed still had its
+query string, so OpenEMR's per-session `token_main` (a local test fixture,
+since stopped) reached the model and the rows. The committed rows replace it
+with `REDACTED`, and the tool now prints URLs without query strings.
+
+## Spend
+
+`results/spend.json` lists every paid run, including four debugging runs
+whose rows were not kept. Total: see the file; the cap was $15.
+
+## Reproduce
+
+```bash
+PYTHONPATH=. python benchmark/code_as_action/measure_mockmed.py --env-file .env --trials 5
+# OpenEMR: a running local fixture (benchmark/openemr_local) and its state dir.
+DOCKER_HOST=... OPENEMR_FIXTURE_STATE=<state dir> OPENEMR_FIXTURE_LOCK=<lock> PYTHONPATH=. \
+  python benchmark/code_as_action/measure_openemr.py --env-file .env --reset --arms code
+python benchmark/code_as_action/summarize.py
+```

packages/flow/benchmark/code_as_action/common.py

@@ -0,0 +1,129 @@
+"""Shared pieces of the code-as-action measurements: the key, the spend
+ledger, host load, and how a run-1 result becomes a committed row.
+
+The key is read from an env file into this process only and never printed.
+The spend ledger (``results/spend.json``) is shared by every measurement
+script here, so the total cap holds across them.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import statistics
+import subprocess
+import time
+from pathlib import Path
+from typing import Any, Optional
+
+HERE = Path(__file__).resolve().parent
+RESULTS = HERE / "results"
+SPEND = RESULTS / "spend.json"
+TOTAL_CAP_USD = 15.0
+#: Timed runs count as unloaded only below this 1-minute load (12 cores).
+LOAD_LIMIT = 12.0
+
+
+def load_key(env_file: Path) -> None:
+    for line in env_file.read_text().splitlines():
+        name, sep, value = line.strip().partition("=")
+        if sep and name.strip() == "ANTHROPIC_API_KEY":
+            os.environ["ANTHROPIC_API_KEY"] = value.strip().strip('"').strip("'")
+            return
+    raise SystemExit(f"no ANTHROPIC_API_KEY in {env_file}")
+
+
+def spent() -> float:
+    if not SPEND.is_file():
+        return 0.0
+    return float(json.loads(SPEND.read_text()).get("total_usd", 0.0))
+
+
+def may_start(run_cap: float, cap: float = TOTAL_CAP_USD) -> Optional[str]:
+    """None when a run capped at ``run_cap`` can't pass the total cap."""
+
+    total = spent()
+    if total + run_cap > cap:
+        return f"${total:.2f} spent; a ${run_cap:.2f} run could pass the ${cap:.2f} cap"
+    return None
+
+
+def add_spend(label: str, usd: float) -> float:
+    RESULTS.mkdir(parents=True, exist_ok=True)
+    data = json.loads(SPEND.read_text()) if SPEND.is_file() else {"runs": []}
+    data["runs"].append({"label": label, "usd": round(usd, 6), "at": now()})
+    data["total_usd"] = round(sum(r["usd"] for r in data["runs"]), 6)
+    data["cap_usd"] = TOTAL_CAP_USD
+    SPEND.write_text(json.dumps(data, indent=2) + "\n")
+    return data["total_usd"]
+
+
+def now() -> str:
+    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
+
+
+def load_1m() -> float:
+    return round(os.getloadavg()[0], 1)
+
+
+def commit() -> str:
+    return subprocess.run(
+        ["git", "rev-parse", "--short", "HEAD"],
+        capture_output=True,
+        text=True,
+        cwd=HERE,
+    ).stdout.strip()
+
+
+def append(path: Path, row: dict[str, Any]) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open("a") as handle:
+        handle.write(json.dumps(row, sort_keys=True) + "\n")
+
+
+def agent_metrics(agent: dict[str, Any], harness: dict[str, Any]) -> dict[str, Any]:
+    """The comparison columns, from an agent ledger and harness stats."""
+
+    return {
+        "model_calls": agent.get("model_calls"),
+        "prompt_tokens": agent.get("prompt_tokens"),
+        "input_tokens": agent.get("input_tokens"),
+        "cache_read_input_tokens": agent.get("cache_read_input_tokens"),
+        "cache_creation_input_tokens": agent.get("cache_creation_input_tokens"),
+        "output_tokens": agent.get("output_tokens"),
+        "cost_usd": agent.get("cost_usd"),
+        "model_seconds": agent.get("model_seconds"),
+        "agent_seconds": agent.get("wall_seconds"),
+        "actions": harness.get("actions"),
+        "screenshots_sent": (
+            harness.get("screenshots_requested", 0)
+            + harness.get("screenshots_at_gate", 0)
+            + harness.get("screenshots_every_turn", 0)
+        ),
+        "images_in_requests": agent.get("images_in_requests"),
+        "tool_errors": agent.get("tool_errors"),
+    }
+
+
+def median(rows: list[dict[str, Any]], key: str) -> Optional[float]:
+    values = [float(r[key]) for r in rows if r.get(key) is not None]
+    return round(statistics.median(values), 3) if values else None
+
+
+def summarize(
+    rows: list[dict[str, Any]], arms: list[str], keys: list[str]
+) -> dict[str, Any]:
+    out: dict[str, Any] = {}
+    for arm in arms:
+        mine = [r for r in rows if r["arm"] == arm]
+        unloaded = [r for r in mine if not r.get("under_load")]
+        out[arm] = {
+            "runs": len(mine),
+            "correct": sum(1 for r in mine if r.get("correct")),
+            "runs_under_load": len(mine) - len(unloaded),
+            "total_cost_usd": round(
+                sum(float(r.get("cost_usd") or 0) for r in mine), 4
+            ),
+            **{f"median_{k}": median(mine, k) for k in keys},
+        }
+    return out