Skip to content

feat(flow): parallel worklists with one session per worker, measured - #1296

Merged
abrichr merged 22 commits into
mainfrom
claude/throughput-c6bed5
Oct 19, 2026
Merged

abrichr merged 22 commits into
mainfrom
claude/throughput-c6bed5

Conversation

@abrichr

@abrichr abrichr commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

An outside review said OpenAdapt's throughput is serial and no faster than careful human entry. This PR makes worklists parallel, makes one entry faster without removing a check, and records where every step's time goes. Speed figures are published only by the scoreboard (benchmark/scoreboard, launch-demo session): this PR provides its arm 4 (fast speed profile) and arm 6 (--workers N).

What changes

Parallel worklists

  • replay and run take --workers N for a for-each bundle (one loop over a worklist). Each item is the same governed run a serial loop makes for that row, with every check, in its own run directory with its own one-row authorization.
  • Each worker signs in once with its own account from the deployment file's new worklist: section (env var names only). The run refuses N above worklist.max_workers, too few accounts, two accounts for the same user, two workers sharing a session (same cookies), rows not declared items_independent, and programs with steps outside the loop.
  • Items are claimed once from a durable SQLite queue in the run directory. An item claimed before a crash becomes uncertain and is never re-run automatically.
  • A halt, failure, worker error or RECONCILIATION_REQUIRED on any worker stops new claims on all workers; in-flight items finish their own checks. worklist_pool.json and WORKLIST.md show per-worker and per-item results.
  • --readback-window K: a worker may start item n+1 while item n's system-of-record read-back runs (only the item's last action, only read-backs that don't use the screen, never for durable runs). The item report is rewritten with the verdict. A failed read-back stops all workers and names the items written after it.
  • worklist.entry_bundle: a recorded navigation replayed before each item, for apps that keep state between items.
  • ProcessWorker: the same worker in its own process (same queue and stop flag). RDP workers use the existing FreeRDP SessionPool (unit tests only).

Where the time goes

  • Every step in report.json has timings: exclusive milliseconds for settle_before, screenshot, resolve, identity, ocr, actuate, settle_after, postcondition, readback, artifact_write, other. They add up to elapsed_ms.

One entry, faster (every check kept)

Always on (no tradeoff found): scroll readiness asks the DOM first and keeps scrolling while the target exists off-screen, without a full-frame OCR search; DOM text postconditions match letters and digits and accept rendered text when hidden option lists interleave textContent (a miss still falls back to OCR); a structural lookup checks box and frame chain once, after binding.

Named speed profiles (openadapt_flow.runtime.speed, runtime.speed), default standard:

Setting standard fast Tradeoff
Typing into plain text fields key by key one insert (input, then change on blur, fire) Per-key handlers in a plain text field don't run; comboboxes/autocomplete/datalist and the guarded keyboard path always get every key; input read-back and DB read-back still check the value
Page-quiet window 100 ms 50 ms Less time for the app to start reacting before the next check; a later reaction becomes a halt, not a wait
OCR threads default 2 Helps many workers, can slow one

Not touched: the guarded click's re-checks, identity checks, input read-back, the system-of-record read-back.

Found along the way

  • The identity check refuses parameter values under 4 characters (first name "Bo" halts). Now in LIMITS.md.
  • The final confirm button's OCR identity band includes form text behind the dialog, so it can halt when the page scrolled a little differently. Fails closed.
  • The referral bundle halts at a scroll step in installed Google Chrome (the default auto browser); bundled Chromium works.
  • Transaction outcomes are upper case in reports; the pool now compares case-insensitively.

Still open

Tests

test_worklist_pool.py (claims, a forced halt on one worker stopping the others, refusals, resume, pipelined window, process workers), test_readback_pipeline.py (real Replayer against MockMed), test_cli_worklist_pool.py (CLI wiring, refusals, entry bundle), test_fast_path.py (native fill, speed profiles, off-screen probe, hidden option lists, step timings), test_page_settle.py (punctuation case).

🤖 Generated with Claude Code

abrichr and others added 16 commits October 18, 2026 14:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…back

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… OCR; committed throughput harness

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ge; throughput and limits docs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…path; benchmark README and figure script

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… resolve, identity, ocr, actuate, postcondition, readback, artifact_write)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xt fields, off-screen scroll probe, one fewer structural round trip, configurable page-quiet window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ist pool; harness --page-quiet-ms

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s builds workers from a module-level factory

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for benchmark runners; speed figures move to the scoreboard

Native fill is now off by default and on in the fast profile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… per-item host load

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ProcessWorker: the child replies before attaching the settle callback, the
  parent keeps early settlements, and a read-back wait has a timeout.
- Pipelined read-back only for REST, FHIR, GraphQL and SQL verifiers; wrapped
  and screen-bound verifiers verify inline.
- A failed pipelined read-back stops claims at once; a stopping item sets the
  stop flag before its state; the stop is timestamped under the write lock.
- report.json stays unsuccessful until a pipelined read-back settles.
- DOM text check keeps separators that are part of a number.
- Accounts that sign in through a bundle must name their user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr marked this pull request as ready for review October 18, 2026 20:31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr and others added 5 commits October 18, 2026 21:13
…e configs in managed-runner tests have none)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ep timings; regenerate the public artifact inventory

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	packages/flow/openadapt_flow/deployment.py
#	packages/flow/openadapt_flow/runtime/replayer.py
#	packages/flow/openadapt_flow/vision/ocr.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant