Repository navigation
bench(flow): referral API step under each_item approval + durable, 20 runs per arm - #1374
Merged
Merged
Conversation
… + durable
benchmark/openemr_api_tier/approval_durable.py runs the OpenEMR referral
with API steps off and on, both under ApprovalGate("each_item") and a
durable run, interleaved, with the setup's own SQL read-back. The harness
answers each approval as a stand-in for the person and its wait is
measured and excluded. Every number it writes carries the label "API step
under each_item approval + durable, no run gate; harness answers
approvals; approval wait excluded": the referral bundle doesn't pass the
run gate yet, so this is not the governed production profile.
The approval view now lists each record-check selector once: the real
bundle's 11 record checks select the same record, which repeated the
rows 11 times and pushed the phone view past its row limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runs per arm Label: API step under each_item approval + durable, no run gate; harness answers approvals; approval wait excluded. Not the governed production profile: the referral bundle doesn't pass the run gate yet on its GUI path. On the local OpenEMR test copy (synthetic patients), interleaved blocks: - API step: 20/20 correct by SQL read-back, 18 saved through the API (r6, the possible duplicate, fell back and created nothing, twice), median 1.12 s engine time, 1 approval per item. - API steps off: 19/20 correct plus 1 fail-closed safe stop (nothing saved), median 15.95 s, 2 approvals per new referral. - No wrong or duplicate chart. The run paused for a shared quiet window after 19 runs and resumed into a second directory; approval_durable.py gains --from-block, --skip-done and a summarize command that rebuilds runs.csv and summary.json from the items. Only the CSV, the summary and each item's result.json are committed (the 21 MB of screenshots and engine reports are kept outside the repository). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1364 (the API-step stack). Item 6 of the production-profile task: the measurement.
Label: API step under each_item approval + durable, no run gate; harness answers approvals; approval wait excluded. These are not numbers for the governed production profile. The referral bundle doesn't pass the run gate yet, because of its GUI path (see the admission list in the session notes, and the local-edit proposal #1366).
Results (local OpenEMR test copy, synthetic patients, 2026-10-19)
engine_s, so the two numbers aren't directly comparable.Not covered
The engine is the API-step stack (#1358, #1361, #1362, #1364) at
352af585b, without the review fixes #1367–#1372. With #1370 and #1371, this referral's API step is refused:POST /library/ajax/execute_background_services.phpinside the span;submitme(...)on submit andcapitalizeMeon each name field, and uses a custompastDaterule.Also in this PR
benchmark/openemr_api_tier/approval_durable.py: the harness. Both arms run underApprovalGate("each_item")anddurable=True, interleaved; a harness approver stands in for the person and the approval wait is measured separately. It gains--from-block,--skip-doneand asummarizecommand. The run paused for a shared quiet window after 19 runs and resumed into a second directory;summarizerebuilds the CSV and summary from the per-item results.results/approval_durable/2026-10-19-combined/(runs.csv,summary.json, and 40 per-item JSON files), plus a README. The 21 MB of screenshots and engine reports are kept outside the repo, in~/oa/scratch/api-step-measure/.public-artifacts.jsonwas regenerated withcheck_release_consistency.py --write-public-artifact-inventory.🤖 Generated with Claude Code