Skip to content

bench(flow): referral API step under each_item approval + durable, 20 runs per arm - #1374

Merged
abrichr merged 3 commits into
mainfrom
flow/api-step-measure
Oct 20, 2026
Merged

abrichr merged 3 commits into
mainfrom
flow/api-step-measure

Conversation

@abrichr

@abrichr abrichr commented Oct 19, 2026

Copy link
Copy Markdown
Member

Stacked on #1364 (the API-step stack). Item 6 of the production-profile task: the measurement.

Label: API step under each_item approval + durable, no run gate; harness answers approvals; approval wait excluded. These are not numbers for the governed production profile. The referral bundle doesn't pass the run gate yet, because of its GUI path (see the admission list in the session notes, and the local-edit proposal #1366).

Results (local OpenEMR test copy, synthetic patients, 2026-10-19)

Arm Runs Correct by SQL read-back Saved through Median engine time Approvals per item
API steps off 20 19, plus 1 safe stop (nothing saved) GUI 15.95 s 2 (1 for r6)
API step 20 20 API 18, GUI 2 (r6) 1.12 s 1
  • r6 is the possible duplicate of a seeded chart. The API step's duplicate pre-check found the chart, so the API step was not used; the GUI steps ran and created nothing. That is the right outcome.
  • The GUI arm's safe stop was the save's identity check refusing at coverage 0.81 before any click. The earlier spike saw the same stop once in 20 runs.
  • No run saved a wrong or duplicate chart.
  • For reference, the scoreboard's arm 5 (API step, no gate, not durable) measured 3.2–3.9 s per item. Its timing boundaries differ from this harness's engine_s, so the two numbers aren't directly comparable.

Not covered

The engine is the API-step stack (#1358, #1361, #1362, #1364) at 352af585b, without the review fixes #1367–#1372. With #1370 and #1371, this referral's API step is refused:

  • OpenEMR sends a background POST /library/ajax/execute_background_services.php inside the span;
  • the form runs submitme(...) on submit and capitalizeMe on each name field, and uses a custom pastDate rule.

Also in this PR

  • benchmark/openemr_api_tier/approval_durable.py: the harness. Both arms run under ApprovalGate("each_item") and durable=True, interleaved; a harness approver stands in for the person and the approval wait is measured separately. It gains --from-block, --skip-done and a summarize command. The run paused for a shared quiet window after 19 runs and resumed into a second directory; summarize rebuilds the CSV and summary from the per-item results.
  • A fix found by the real bundle: the approval view listed the same record rows once per record check (11 times), which pushed the phone view past its 24-row limit. Each record-check selector is now listed once, both in the view and in the digest. Test included.
  • Committed data: results/approval_durable/2026-10-19-combined/ (runs.csv, summary.json, and 40 per-item JSON files), plus a README. The 21 MB of screenshots and engine reports are kept outside the repo, in ~/oa/scratch/api-step-measure/. public-artifacts.json was regenerated with check_release_consistency.py --write-public-artifact-inventory.

🤖 Generated with Claude Code

abrichr and others added 3 commits October 19, 2026 13:20
… + durable

benchmark/openemr_api_tier/approval_durable.py runs the OpenEMR referral
with API steps off and on, both under ApprovalGate("each_item") and a
durable run, interleaved, with the setup's own SQL read-back. The harness
answers each approval as a stand-in for the person and its wait is
measured and excluded. Every number it writes carries the label "API step
under each_item approval + durable, no run gate; harness answers
approvals; approval wait excluded": the referral bundle doesn't pass the
run gate yet, so this is not the governed production profile.

The approval view now lists each record-check selector once: the real
bundle's 11 record checks select the same record, which repeated the
rows 11 times and pushed the phone view past its row limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runs per arm

Label: API step under each_item approval + durable, no run gate; harness
answers approvals; approval wait excluded. Not the governed production
profile: the referral bundle doesn't pass the run gate yet on its GUI path.

On the local OpenEMR test copy (synthetic patients), interleaved blocks:
- API step: 20/20 correct by SQL read-back, 18 saved through the API
  (r6, the possible duplicate, fell back and created nothing, twice),
  median 1.12 s engine time, 1 approval per item.
- API steps off: 19/20 correct plus 1 fail-closed safe stop (nothing
  saved), median 15.95 s, 2 approvals per new referral.
- No wrong or duplicate chart.

The run paused for a shared quiet window after 19 runs and resumed into a
second directory; approval_durable.py gains --from-block, --skip-done and a
summarize command that rebuilds runs.csv and summary.json from the items.
Only the CSV, the summary and each item's result.json are committed (the
21 MB of screenshots and engine reports are kept outside the repository).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr

abrichr commented Oct 19, 2026

Copy link
Copy Markdown
Member Author

Included in batch #1383, which runs CI once for several reviewed PRs. This PR is marked merged when #1383 merges. Please don't push here unless asked; a new commit leaves it out of the batch.

abrichr added a commit that referenced this pull request Oct 20, 2026
batch: API-step fixes, compatibility matrix and local-edit classification (#1370 #1371 #1372 #1364 #1374 #1366)
Base automatically changed from flow/compatibility-matrix to main October 20, 2026 00:24
@abrichr
abrichr merged commit 0a30daf into main Oct 20, 2026
35 checks passed
@abrichr
abrichr deleted the flow/api-step-measure branch October 20, 2026 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant