Repository navigation
Workbench: live measurements leave no versioned trace #459
Description
Split out of #428, which is being closed — its core ask is done, this is the adjacent point that was never in scope for it.
What's left
workbench/runs/ and workbench/reports/ are gitignored, and only three baselines are committed (consent.json, describe-project.json, wizard-enhance.json). Every live measurement made against this bench — including the ones behind #431, #432, #440, #452, #453, #454 and #456 — leaves no trace anyone else can replay without re-running it themselves, on their own key, against whatever the provider resolves that day.
Concretely, that means:
- A cassette recorded during a PR review (e.g.
workbench/cassettes/judge-migration-*.json) replays offline for whoever has the file, but nothing forces it to reachmain— it's a byproduct of one session, not a tracked measurement. - The three real-take scenarios (
real-wizard-enhance,real-zooms,real-zoom-grounding) have never been judged at all, because the take itself (workbench/fixtures/) is gitignored and absent from every clone. See removeModifier deletes a whole pill but reports removing one id #457's sibling problem andworkbench/README.md§ "La prise réelle" for what that costs on a fresh checkout (44 L0 tests fail on the missing fixture, silently, since CI never runs the bench). - A baseline's
behaviour/dslnumbers, once written, are never re-verified against anything — nothing reads them back (baseline.tsonly compares check ids), so they drift into being an unlabelled archive, which is exactly what happened towizard-enhance.jsonbefore fix(workbench): retire beh.sandbox from the baseline, confirmed at n=10 #453 said so explicitly.
First step, as noted when this was flagged
Freeze the real-take scenario baselines (--update-baseline, verified by hand rather than trusted), then capture the provider's usage block in the recording proxy (workbench/lib/cassette.ts) — the token count every context-size argument in this codebase wants is already in the response the recorder sees pass through, uncaptured.
Also worth deciding here, not in #428
Whether cassettes from a judged PR review should be committed as a matter of course (the way judge-migration-*.json were, by convention rather than by rule) or whether that's noise the repo doesn't want. workbench/README.md doesn't currently say either way.
Activity
auberginewly commentedon Sep 26, 2026
Hey! I'd love to help move this forward!
I'm currently working on an internal Coding Agent during my internship, and I'm particularly interested in agent evaluation and reliability.
I noticed that the original real-take baselines are still missing. I'd be happy to take ownership of this part, including building reproducible evaluation scenarios, preparing test fixtures, and validating the results.
Are the original recording fixtures available to contributors, or would it make more sense to create a new set?
I'd love to contribute more regularly to OpenScreen, especially on the Agent side. Happy to take on more work in this area over time!
EtienneLescot commentedon Oct 24, 2026
Hi @auberginewly, sorry for the slow reply, and thanks for offering! The original take stays local and isn't distributed, so a new, small fixture set that can be committed is the better route: it also fixes fresh clones missing the fixture. Good first steps: freeze the real-take scenario baselines (checked by hand), then capture the provider's
usageblock inworkbench/lib/cassette.ts. Happy to review, ask anything here.auberginewly commentedon Oct 27, 2026
Thanks for clarifying, @EtienneLescot! I'll use a new, small fixture set that can be committed, with distinct scenario IDs and separately checked annotations, while keeping the original private take and its baselines explicitly unavailable.
I also checked the merged #653: the cassette usage capture and versioned measurement/replay workflow are already there, so I'll build on those rather than duplicate them. The remaining work is the public material and scenarios, recorded measurements, and reviewed baselines. Thanks for the direction!
auberginewly commentedon Oct 27, 2026
Thanks again for the guidance, @EtienneLescot. I've opened draft #1403 with a small publicly committable local recording, its project/cursor sidecar, capture source and licensing, and three new scenario IDs. It reuses #653's measurement workflow, with an explicit
public-recordedregistration rather than labelling a recording synthetic. The original private take andreal-*failures are preserved.The scoped offline checks and measurement lifecycle controls pass, and the PR includes a candid independent agent review record. This fixture is a silent, scripted browser capture, so it does not yet cover narrated enhancement or speech preservation. Real provider measurements are also still pending local
.env.workbenchconfiguration; there are no claimed provider usage numbers, judged model results or approved baselines yet. I'll keep the PR in draft while those gaps remain, and I'm not treating this as completion of #459. Thank you for offering to review.auberginewly commentedon Oct 28, 2026
Thanks again for the direction, @EtienneLescot. I've updated draft #1403 with actual bounded
deepseek-flashruns, provider usage, agent/judge cassettes and an independent review of the edits and evidence. The public recording, cursor sidecar and three new IDs remain separate from the private take.The results exposed a strict replay problem: later
setZoomcalls reference region UUIDs returned during recording, whereas replay generates fresh UUIDs. Export verification passes, but strict replay fails. The reviewer withdrew provisional baseline approval; all committed receipts now reject baseline eligibility, and no baseline has been frozen. I have retained the real model failures, an indeterminate judge response, and transport-failure evidence rather than weakening freshness/privacy checks. The PR also documents the stored-focus oracle's limitation for automatic pointer-following zooms.The scoped offline checks pass (78 tests, one existing skip). Enhancement measurements still suffer transport failures; narrated material and compositor-output review are also outstanding. The PR contains reproduction commands and the remaining gaps and stays in draft. This is progress on #459, not a completion claim. Thank you for offering to review it.
auberginewly commentedon Oct 28, 2026
Thanks again for the guidance, @EtienneLescot. I've updated #1403 and marked it ready for review after completing the replay and independent-review work.
It now includes nine normally completed real
deepseek-flashturns, nine judge requests, observed provider usage, three approved measurement bundles and three bound baselines on the pinned current-main source. Both independent preflights and final public agent-plus-judge strict replay/baseline verification pass. The prospective ID-allocation ledger preserves exact tool targeting and genuine wrong-ID errors; older rejected evidence remains unchanged. Actual edited documents and production-compositor samples were independently inspected.The runbook and reproduction commands and independent review record retain the genuine sweep failures, automatic-focus unknowns, one indeterminate judge and a documented judge false positive caused by incomplete transcript facts. Approval means faithful observed evidence, not every quality check passing.
The new fixture is still a silent scripted browser recording, with separate IDs and explicit provenance. Human annotation review, narrated/speech coverage and the original private take remain unvalidated; #459 stays open, with no closing reference. Latest CI is running. Thank you for offering to review this.