Skip to content

scoreboard: timed OpenEMR arms 3-6, arm 5 API step, agent/screen/API comparison - #1365

Merged
abrichr merged 17 commits into
mainfrom
claude/scoreboard-openemr-arms-3-6
Oct 20, 2026
Merged

abrichr merged 17 commits into
mainfrom
claude/scoreboard-openemr-arms-3-6

Conversation

@abrichr

@abrichr abrichr commented Oct 19, 2026

Copy link
Copy Markdown
Member

Follow-up to #1303, which was merged at 05dd032. These are the scoreboard rows and changes made after that commit. Every figure comes from python -m scoreboard.build over rows committed under packages/flow/benchmark/scoreboard/raw/.

What's in it

Checks

  • ruff check and ruff format --check on benchmark/scoreboard.
  • The public artifact inventory was regenerated from a clean checkout of the commit (refresh_inventory.sh).
  • Grepped for prospect names; none found.
  • Synthetic data only.

🤖 Generated with Claude Code

abrichr and others added 9 commits October 19, 2026 09:55
…y: triggers and a poller)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…down in the tables

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ngine and throughput lines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ff runs, Amendment 6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e labels per run, claims runs shown only when complete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread packages/flow/benchmark/scoreboard/claims_openemr.py Fixed
abrichr and others added 2 commits October 19, 2026 14:25
…n the fixture database), Amendment 7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr and others added 2 commits October 19, 2026 14:41
…e every median, Amendment 8

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr and others added 4 commits October 19, 2026 15:13
… on rebuilt state (5/5 caught, 0 false alarms); claims scripts never read a stale receipt; Amendment 9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er in a file (CodeQL py/clear-text-storage-sensitive-data); label the re-judge block 'rebuilt state, judged by fixed verify-run'

The claims scripts wrote each item's (made-up, patient-shaped) values to a
params.json beside the baseline. verify-run now reads them from /dev/stdin,
so no file holds them; the run record keeps only the executor and its
claim (claims.py no longer copies the agent's final text into it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	packages/flow/public-artifacts.json
@abrichr

abrichr commented Oct 19, 2026

Copy link
Copy Markdown
Member Author

Included in batch #1385, which runs CI once for several reviewed PRs. This PR is marked merged when #1385 merges. Please don't push here unless asked; a new commit leaves it out of the batch.

@abrichr
abrichr merged commit b91c4af into main Oct 20, 2026
33 of 41 checks passed
abrichr added a commit that referenced this pull request Oct 20, 2026
@abrichr
abrichr deleted the claude/scoreboard-openemr-arms-3-6 branch October 20, 2026 02:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants