feat(flow): Harness² benchmark adapter scored by read-back

[abrichr]

What this adds

openadapt_flow.integrations.harness2 lets OpenAdapt act as the scorer for Harness² harness search on GUI tasks. Harness² handles the search: proposers edit the agent's instructions from full trajectories. OpenAdapt supplies the task environment and the score. The score comes from the reward worker reading the system of record of a fresh app copy after each rollout.

• Rollout: each one starts a fresh MockMed copy and registers the episode, which reads the baseline. The agent then drives the copy under the harness being tested. Afterwards the worker reads /api/db again and signs a tier-2 receipt. Production is never involved.
• Search reward: verified 1, safe stop 0, wrong value -0.5, wrong record or duplicate -1. Outcomes that a person must settle get no number.
• MockMed task family: six variants: plain, lookalike, sort, typelabel, timeout (the save lands, then the client shows an error) and missing (the right outcome is to stop).
• Adapter: OpenAdaptGuiAdapter implements Harness²'s BenchmarkAdapter. It imports Harness² and copies no Harness² code. It has two evidence modes:
  • screen: Harness² as published, with no scorer.
  • readback: each trajectory ends with what the store shows.
• Selection: reward_selection picks the final by read-back reward. run_task runs one task through Harness²'s own recursion.run_mode.
• Agents and proposers:
  • Scripted agent and scripted proposer for tests (no model calls).
  • ModelGuiAgent and ModelProposer for measured runs. They call models through a small HTTP client with a hard cost cap. Every request first passes privacy.prepare_outbound.
• Docs: packages/flow/docs/HARNESS2.md.

Install and CI

Harness² is not on PyPI, and a published wheel cannot name a git URL. So the extra openadapt-flow[harness2] brings Playwright only, and the docs give the pinned pip install command for Harness². flow-unit installs Harness² at commit 1cfe254e. It has no dependencies.

One integration point uses Harness² internals. Harness² builds proposer prompts per surface name and knows only opencode and codex. The adapter registers a third surface text, built from Harness²'s own INV and DOM templates.

Tests

tests/test_harness2_adapter.py: 22 tests, about 60 s locally.

• The scorer judges eight store states.
• Scripted rollouts produce wrong-record, duplicate, verified, reconciliation and safe-stop outcomes.
• With the read-back, the Harness² loop moves lookalike from a wrong-record base to a verified final.
• The model-selector path runs, the screen mode hides the read-back, and the model proposer writes only harness files.
• The privacy gate refuses to send without a scrubber or an opt-out.

Follow-ups (separate PRs)

• Meta-Harness-style trace export.
• Multi-step repair search against a staging copy.
• Desktop and RDP fixtures.
• A small measured run with and without the scorer.

🤖 Generated with Claude Code