Skip to content

scoreboard: arm 7, the code-as-action agent's first run - #1307

Open
abrichr wants to merge 31 commits into
claude/launch-demo-cua-to-program-8fa902from
claude/scoreboard-arm7
Open

abrichr wants to merge 31 commits into
claude/launch-demo-cua-to-program-8fa902from
claude/scoreboard-arm7

Conversation

@abrichr

@abrichr abrichr commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

Adds arm 7 (code_agent) to the scoreboard runner, per PROTOCOL.md. Code only: the rows are run by the scoreboard's single runner under the load gate.

python -m scoreboard.run code_agent <task> --run-id <id> --n N runs one first run of ClaudeAgent(vision="code") (claude-sonnet-5-5, effort medium) per item, inside the compile-agent harness and save gate. Each run is recorded by the same JIT wrapper as arm 2, read back from the system of record, compiled, checked against the agent's trace (check_trace_against_ir), and taken through the same _admission(). It uses add_spend(), schema.Row, schema.timed() and write_meta(model=...) unchanged. OpenEMR uses tasks/openemr.py (item_values, check, oracle_verifier, setup, entry_url). Arms 1 to 3 are unchanged; the two JIT builders take an optional dialect.

Merge order: #1277, then #1305 (the agent itself), then this. Until those land, this diff also shows their commits (this branch has #1305 merged in).

Trial runs set aside (superseded/, not in the scoreboard, untimed, $0.31 on the scoreboard key, not yet in raw/spend.json): MockMed 5 of 5 correct and admitted; OpenEMR items 701 to 705: 4 correct and 1 safe stop on hand review, with admission refused for all 5 ("a write step lacks an independent read-back": no system-of-record change bound to the confirm-dialog save step); 5 earlier OpenEMR runs refused at the first drop-down by a gate rule since fixed in #1305. Arm 7's OpenEMR items now start at 721, clear of the four charts the trial saved.

public-artifacts.json: the inventory writer also registered this branch's existing raw files (arms 1 to 3), which were not listed yet.

🤖 Generated with Claude Code

abrichr and others added 24 commits October 18, 2026 13:24
…ecomes a zero-token program

Live agent loop with selective vision (structural listing first, screenshots
on request or at the save gate), right-record check and hold before every
save, run-1 record check against the system of record, person-confirmed
record checks, JIT admission (policy self-admission), emitted skill, MCP
server and CLI line, and replays with a verify-run receipt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, CI

- Gate a click at the current pointer (no coordinate) like any other click.
- docs/CRYSTALLIZE.md; CAPABILITIES.md and JIT.md point at it.
- benchmark/crystallize/measure.py: live selective vs every-turn runs.
- Run tests/e2e/test_crystallize_e2e.py in the flow-critical CI job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ten live claude-opus-5-5 trials per arm on MockMed: structure first used
10,332 prompt tokens and $0.028 per run against 76,053 and $0.148 with a
screenshot every turn; 10/10 passed in each arm; 10/10 replays VERIFIED with
0 model calls. A new run 1 clears checks confirmed for an earlier draft.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s.json

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gaps from review

Rename: openadapt-flow compile-agent / compile-agent run, package
openadapt_flow.compile_agent (compile_first_run, run_compiled,
FirstRunCompiler), COMPILE_AGENT.md, benchmark/compile_agent, cache dir
.openadapt-compile-agent. The help says the agent writes to --url.

Review fixes:
- Gate every action that may commit: commit-labelled controls of any role,
  unreadable elements, Enter, typed line breaks, Space off text fields,
  shortcuts. More commit words (delete, update, done, ...).
- Identity must appear as whole words in the control's row, dialog, form or
  section, not anywhere on the page; refs are re-checked by row.
- A refused run's draft is set aside even if the agent carried on; stale
  proposals are cleared; run 1 refuses on a task with a draft or program.
- The recording is signed with the driving agent's principal, not the admitter.
- Person admission shows steps and canary results and asks at each stage.
- Canary values are required (a canary is a real save); no defaults.
- Unknown models need explicit prices for the cost cap.
- confirm_params runs before verify.yaml is written; receipt baseline is reset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--backend (record's choices), a --config whose backend.kind is not web, or a
non-http(s) URL stop with exit 2 and a message before anything runs. The
Python API raises UnsupportedTarget. No silent browser fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ClaudeAgent(vision="code") acts through one run tool that takes a short
script: read the DOM, accessibility tree or OCR text, act by role and
name, scroll, screenshot only on request, and a sandboxed read-only shell
in a working directory. Every action is a structured step checked against
the locator the recorder stored and the compiled program.

The save gate now reads inside iframes, takes a tool's hint for frames and
desktop surfaces, and no longer treats a click on a text field such as
Postal code as a save.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ocus in frames

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e not saves

- A click over an iframe is read inside the frame through the backend's
  hit-tested frame chain; a frame it can't map counts as unreadable (gated).
- Enter or Space with focus inside a frame reads the frame that has focus.
- A click on an editable field ("Postal code") only focuses it; not a commit.

Found by the code-as-action session (claude/code-as-action d3b8efc4, 8bbcad02);
ported here so #1277 does not merge with the gaps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed OpenEMR runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, not missing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… result files

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- shell: per-tool flag allow-list, no long options, attached values checked,
  quoted operators are words, process-group kill; macOS profile denies other
  programs and reads outside the working and system directories
- gate: on a surface with no page, Enter and Space are always gated; select
  only on lists, and a drop-down that submits on change is gated
- a bad number is a line error; a line that acted before failing keeps its
  step in the trace and reports a possible commit; error text is scrubbed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…save

Any onchange counted before, so OpenEMR's Title drop-down was gated like a
save and refused every run at the first select.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nEMR referral)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr and others added 4 commits October 18, 2026 17:51
…runner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a path not on this branch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr and others added 3 commits October 18, 2026 20:19
- harness: unpack the click point before reading it.
- _HarnessedAgent.dialect is a plain attribute, so it meets ComputerUseAgent.
- canaries run through functools.partial instead of a defaulted lambda.
- compile-agent run calls FirstRunCompiler.load(...).run so --param values
  can't collide with run_compiled's reset= keyword.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant