Skip to content
35 changes: 35 additions & 0 deletions packages/flow/docs/HARNESS2.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,41 @@ end with Harness虏's recursion code: `k` base rollouts, one mode (`seq` or
`agg`), selection by reward or by Harness虏's model selector, and the
read-back of the chosen run.

## Trace export

Meta-Harness ([arXiv:2603.28052](https://arxiv.org/abs/2603.28052)) gives
its proposer a filesystem with every earlier candidate's harness, full
execution traces and scores. In its ablation, full traces beat summaries,
56.7 to 38.7. `openadapt_flow.integrations.harness2.traces` writes that
filesystem for GUI work. `run_task` writes one for every task under
`<work_dir>/traces`.

```
traces/INDEX.md one line per trace, best reward first
traces/index.jsonl the same, one JSON object per trace
traces/candidates/<label>/harness/ the harness each candidate ran
traces/traces/<id>/trace.md the full trace, readable
traces/traces/<id>/trace.json the full trace, structured
```

Each trace keeps every step in full: the screen, the action, the reason the
agent gave, and any error. At each write it records whether the patient on
screen was the requested one (`verified`, `mismatch`, `unreadable`). It ends
with the read-back: the records added, whose they are, the outcome, the
reward and the receipt id.

`from_flow_run` reads a compiled program's `report.json` instead. Its
identity verdicts are the runtime's own gate results, and a read-back can be
attached. The repair search uses this to read a halted run beside the last
verified ones.

Screen text passes through `privacy.scrub_text` before it is written, like
`REPORT.md`. Identity verdicts are computed before scrubbing, so a scrubbed
export still says which write hit the wrong patient.

In `readback` mode the same identity verdict also appears in the trajectory
Harness虏 proposers read, as an observation after each write.

## Install

Harness虏 needs Python 3.12 and is not on PyPI. Install it from its
Expand Down
17 changes: 16 additions & 1 deletion packages/flow/openadapt_flow/integrations/harness2/adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,15 @@ def delivered(self, run_dir: Path) -> bool:

def trajectory(self, run_dir: Path, *, obs_trunc: int) -> str:
run_dir = Path(run_dir)
run = load_run(run_dir)
identity: dict[int, Any] = {}
if self.evidence == "readback" and run.get("task_id") in self._tasks:
# OpenAdapt's identity check at each write: was the record on
# screen the requested one? Part of the read-back evidence.
from openadapt_flow.integrations.harness2.traces import from_rollout

trace = from_rollout(run_dir, self._tasks[run["task_id"]])
identity = {s.index: s.identity for s in trace.steps if s.identity}
lines: list[str] = []
for step in load_steps(run_dir):
obs = str(step.get("observation", ""))
Expand All @@ -313,9 +322,15 @@ def trajectory(self, run_dir: Path, *, obs_trunc: int) -> str:
+ _action_text(action)
+ (f"\n reason given: {thought}" if thought else "")
)
verdict = identity.get(int(step.get("index", -1)))
if verdict is not None:
lines.append(
f"[OBSERVATION] identity check at this write: {verdict.status}; "
f"requested {verdict.expected}; on screen "
f"{verdict.observed or 'no patient banner'}"
)
if step.get("error"):
lines.append(f"[ERROR] {step['error']}")
run = load_run(run_dir)
if run.get("error"):
lines.append(f"[ERROR] {run['error']}")
if not any(line.startswith("[ACTION] done") for line in lines):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@

:func:`run_task` is the loop for one task: ``k`` base rollouts, one Harness虏
recursion mode (its code, imported), a selection by reward or by a model, and
the read-back of the chosen run.
the read-back of the chosen run, and a trace export of every rollout
(``<work_dir>/traces``, see :mod:`.traces`).
"""

from __future__ import annotations
Expand All @@ -23,6 +24,7 @@
from harness2.core.proposer import Proposer
from harness2.core.spec import RunView

from openadapt_flow.integrations.harness2 import traces
from openadapt_flow.integrations.harness2.adapter import OpenAdaptGuiAdapter

#: Where an unscored candidate ranks: below a safe stop, above any penalty.
Expand Down Expand Up @@ -168,6 +170,20 @@ def run_id_of(label: str) -> str:
"mode_result": result.to_dict(),
"proposer_usd": round(sum(s.usd for s in result.stages) + sel.usd, 6),
}
labelled = [(lbl, rv) for lbl, rv in result.rollouts.items() if rv.ok]
traces.export(
work_dir / "traces",
[
traces.from_rollout(rv.run_dir, adapter.task(task_id), candidate=lbl)
for lbl, rv in labelled
],
harnesses={
lbl: work_dir / "harnesses" / lbl
for lbl, _rv in labelled
if (work_dir / "harnesses" / lbl).is_dir()
},
)
summary["traces"] = str(work_dir / "traces")
(work_dir / "summary.json").write_text(
json.dumps(summary, indent=2, default=str), encoding="utf-8"
)
Expand Down
Loading
Loading