Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b53a57f
bench(flow): exclusive-time profiler for one replay run
abrichr Oct 18, 2026
6ee27d6
Merge remote-tracking branch 'origin/main' into claude/throughput-c6bed5
abrichr Oct 18, 2026
7624301
feat(flow): worklist pool with one claim per item and pipelined read-…
abrichr Oct 18, 2026
4fd89d1
feat(flow): --workers N and --readback-window K for worklist runs
abrichr Oct 18, 2026
feee79c
perf(flow): DOM text check ignores punctuation before falling back to…
abrichr Oct 18, 2026
fa2c6c4
feat(flow): worklist.entry_bundle returns each worker to the start pa…
abrichr Oct 18, 2026
58a55dc
fix(flow): a handed-off read-back always settles through the pending …
abrichr Oct 18, 2026
d6067e2
feat(flow): per-step timing spans in report.json (settle, screenshot,…
abrichr Oct 18, 2026
4774294
perf(flow): fast path with every check kept: native fill for plain te…
abrichr Oct 18, 2026
cfe3519
fix(flow): match transaction outcomes case-insensitively in the workl…
abrichr Oct 18, 2026
0b1aa37
feat(flow): process-isolated worklist workers; OCR thread cap; harnes…
abrichr Oct 18, 2026
d6558cb
docs(flow): fast path, tradeoffs and worker processes in THROUGHPUT.md
abrichr Oct 18, 2026
f214dce
feat(flow): named speed profiles (standard, fast) and FunctionWorker …
abrichr Oct 18, 2026
ba7bb2a
bench(flow): import OpenEMR worklist runs into the scoreboard's rows;…
abrichr Oct 18, 2026
d6d19c6
fix(flow): review fixes for parallel worklists and pipelined read-back
abrichr Oct 18, 2026
7d1d294
Merge remote-tracking branch 'origin/main' into claude/throughput-c6bed5
abrichr Oct 18, 2026
b82131e
docs(flow): link the scoreboard PR until benchmark/scoreboard is on main
abrichr Oct 18, 2026
6ee2e37
fix(flow): read the deployment's worklist section with a default (fak…
abrichr Oct 19, 2026
f34d177
Merge remote-tracking branch 'origin/main' into claude/throughput-c6bed5
abrichr Oct 19, 2026
d6cf225
docs: capability rows for parallel worklists, the fast profile and st…
abrichr Oct 19, 2026
ff52df0
Merge remote-tracking branch 'origin/main' into claude/throughput-c6bed5
abrichr Oct 19, 2026
d2df1e3
fix(flow): mypy in the worklist pool and the read-back hand-off
abrichr Oct 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion CAPABILITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,10 @@ improvising or retrying when something is off.
| Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` |
| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` |
| Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) |
| Run a worklist on several workers at once, one signed-in session each, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--workers` |
| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) |
| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) |
| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) |
| Record where each step's time goes in every run report | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` |

## When the app changes

Expand Down
57 changes: 57 additions & 0 deletions packages/flow/benchmark/openemr_throughput/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# OpenEMR referral worklist runner

Runs the OpenEMR referral program over a worklist on a local OpenEMR 8.0.0.3
test copy, with 1 to 8 workers, either speed profile, and optional pipelined
read-back, and judges every item from the database afterwards. Every patient,
address, phone number (555-01xx) and email (`example.invalid`) is made up.
OpenAdapt isn't affiliated with the OpenEMR project.

Speed figures are published only through the
scoreboard (`benchmark/scoreboard/`, added in [#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) (arm 4, `program_fast`, and arm 6, `parallel`).
This folder holds the runner and the profiler, not results.

| File | What it is |
| --- | --- |
| `harness.py` | Runs one worklist on N workers against the fixture and judges every item afterwards |
| `profiler.py` | Exclusive-time profiler for one replay, deeper than the per-step `timings` in `report.json` |

## What one run does

1. **Setup, not timed.** Reset the database to the fixture's verified
baseline, add 8 clerk accounts cloned from the fixture's actor user, seed
the existing chart Rosa Delgado, and generate M distinct synthetic
referrals. First names have 4 or more letters (see LIMITS.md, "Short
parameter values").
2. **The worklist.** `run_worklist_pool` over the M referrals on N workers.
Each worker opens its own browser context and signs in once as its own
clerk. For each item it claims, it closes any open patient chart (OpenEMR
keeps the last patient open after a save, which moves the form), opens a
blank New Patient form, and replays the program with the engine's own
read-back of the saved chart through the OpenEMR REST API. `--processes`
runs each worker in its own process.
3. **Judge, not timed.** The test setup reads every item back from the
database with SQL. An item is correct when exactly one new chart exists
for it with 12 of 12 fields matching; an unrun item must have no chart.

`--halt-at I` replaces item I with a possible duplicate of the seeded chart;
the engine must halt on it and the pool must stop every worker.
`--speed fast` applies the `fast` profile (`openadapt_flow.runtime.speed`).
`--profile` (one worker) writes an exclusive-time profile per item.

Each run writes `summary.json` (counts, wall time, sign-in time per worker,
host load, engine commit, speed settings) and `items.json` (worker, state,
judge's verdict and seconds per item) into `--out/<tag>`, for the
scoreboard's importer.

## Run

Needs podman, the fixture's baseline state, and the compiled referral bundle
(not in this repository). Timed runs wait until the host's 1-minute load
average is under 12.

```bash
THROUGHPUT_UI_PORT=9911 THROUGHPUT_API_PORT=9910 \
python -m benchmark.openemr_throughput.harness run --bundle BUNDLE \
--fixture-state STATE --out OUT --results OUT/runs.jsonl \
--tag n4 --workers 4 --items 32 --speed fast --processes --gate-load 12
```
Loading
Loading