[abrichr]

Stacked on #1300 (and #1299). Merge those first.

What this is

One small measured run on synthetic MockMed data. It compares Harness² as published with Harness² using the read-back as both evidence and selection. Full write-up: packages/flow/benchmark/harness2_scorer/RESULTS.md.

• Tasks: seven MockMed tasks. This PR adds slow: the form stays up after a save that already landed.
• Models: the agent is gpt-4.1-nano. Proposers and the selector are gpt-4.1-mini, one call per stage.
• Search: Harness² agg mode with k=2.
• Measure: each chosen harness is re-run 10 times per task on fresh copies, and every re-run is scored by the reward worker. Selecting by reward picks a run that already succeeded, so the chosen run's own outcome would flatter that arm; fresh re-runs avoid that bias.
• Spend: $0.60 against the $15 cap.

Arm	Right outcome	Verified	Wrong record	Duplicate	Safe stop
Base harness	39/70	37	8	17	3
Harness²	41/70	27	0	22	13
Harness² + read-back	40/70	27	0	23	10

At this scale the read-back did not help. Both searches removed every wrong-patient write and fixed the missing-patient task. Both also added duplicates and broke typelabel. The write-up names two causes:

1. With the read-back in view, proposers wrote rules the agent cannot follow, such as "verify the system of record shows exactly one new encounter". The agent has no read-back of its own, so it saved again.
2. Harness² ranks each candidate on one rollout, which mostly ranks luck. Read-backs cost about $0.001 a rollout here, so scoring each candidate on several rollouts is the cheap next step.

Files

• benchmark/harness2_scorer/run.py: the arms, a hard cap, and --more-reruns to top up re-runs.
• summarize.py: the tables. It copies raw data without the reward workers' private keys and tokens.
• evidence/: per-arm results.json, summary.json, and raw.tar.gz (2.3 MB). These are listed in public-artifacts.json, and the archive was checked for keys and tokens.
• docs/HARNESS2.md: a short "Measured run" section.

🤖 Generated with Claude Code