[abrichr]

Qualifying a new workflow now takes a verify.yaml instead of a 1,100-line script.

openadapt-flow qualify path/to/bundle --verify verify.yaml --out runs/q1 \
  --certify clinical-write [--cases ...] [--run-params inputs.json]

For review only. Do not merge. Flow is frozen for the monorepo cutover. After OpenAdapt #1178 lands, carry this branch over with git format-patch origin/main..feat/declarative-qualify, then git am -3 --directory=packages/flow in a worktree from origin/monorepo/phase1-import.

What changed

• openadapt_flow/qualify/ (new library, plus the qualify CLI default form; the subcommand is qualify campaign). Moved here from the scripts:
  • project init, risk classification, and identity and effect policies;
  • the six cases, case scoping, and per-campaign local test keys (Ed25519, held in memory, never a production trust root);
  • signed case results, certification, the run report, and an optional certified run.
  • qualify checks verify.yaml and the bundle before it runs and lists every problem it finds. It stops at the first case that does not behave as required, with the reason.
• Generic web fault driver (qualify/faults.py). It finds the save step's button through the bundle's own structural selector and stages the faults there:
  • ambiguity: clones the button;
  • wrong identity: relabels the button, and rotates record-identity text when the step binds to it;
  • stale identity: blanks the button's document;
  • weak effect: swaps in a screen-only check;
  • missing effect: removes the check.
  • It ships with a generic web environment observer. A pixel-surface driver is not built: qualify refuses app.target other than web, and the docs list it as the next piece.
• verify.yaml (qualify/config.py) covers:
  • the system-of-record read-back, reusing the deployment effects schema (rest today; sql, fhir and graphql go through the same path);
  • mapping record fields to parameters, and a duplicate/idempotency key;
  • save_step: auto or an explicit step id, plus test inputs;
  • reset (command, http or python) and Python hooks for sign-in and API sessions;
  • ${VAR} / ${VAR:-default} expansion.
• Four default compiler fix-up passes (compiler/fixup_passes.py). Each change is recorded in the bundle's compile_notes.json:
  • drop Enter after typing a select option (web only);
  • drop Ctrl+A on an empty field, using a pixel test of the clicked field box;
  • add REGION_STABLE to a field click (on pixel surfaces, "click then type" counts as a field click);
  • add an identifier crop to an unlabeled button.
  • Recompiling the OpenEMR recording reproduces the hand-prepared bundle: same 29 steps, checks, crops and armed flags. The 16th old fix-up, email as the duplicate key, now comes from verify.yaml.
• Scripts:
  • scripts/openemr_qualification.py is now a thin wrapper over benchmark/openemr_local/verify.yaml and qualify_hooks.py. It keeps the direct-SQL second oracle as an optional, demo-only check (--no-sql-oracle turns it off).
  • The MockMed exporter qualifies through the library via QualifyHooks (public-demo/mockmed-verify.yaml). Its pack layout, media and validation code are unchanged.
• Docs: docs/QUALIFICATION_TERMS.md defines qualify, certify and certification policy, with every built-in policy and pack read from the code. docs/QUALIFY_A_WORKFLOW.md is the how-to with an example verify.yaml. qualify --help opens with a one-line definition and links both pages, and the README links them too.

Line counts

	before	after
scripts/openemr_qualification.py	1133	190
scripts/export_public_demo_evidence.py	3297	2671 (the rest is pack, media and validation)
new: qualify library		2367
new: fixup passes		327
per-app config (OpenEMR verify.yaml + hooks; MockMed verify.yaml)		57 + 123; 25

Results (Mac, local fixtures, machine load average 220–300 on 12 cores from other jobs)

• OpenEMR, via the CLI, 6 cases × 3 trials, --certify clinical-write:
  • All 18 trials behaved as required: representative VERIFIED by exact REST read-back 3/3, and every fault HALTED with no mutation, 15/15. 0 model calls.
  • Certified under clinical-write: identity coverage 1/1, effect coverage 1/1, cases 6/6.
  • Campaign took 1320 s; mean trial 71 s; resets 326 s in total.
  • The certified run that followed was VERIFIED, production_eligible, 0 model calls, 50 s engine time.
  • The old script's campaign (qual13) took 586 s, with 25–31 s per representative trial. The new run was much slower: 35–88 s per representative trial under the current load. This is not a same-load A/B.
• OpenEMR, via the wrapper, smoke run (representative + wrong-identity, 1 trial each, with the SQL oracle): the SQL oracle passed on both (exact_record 12/12 fields; no_mutation).
• MockMed exporter: new pack mockmed-triage-declarative-1, 3 VERIFIED + 15 HALTED, 596 files. --validate reports VALID. The export took 153 s wall-clock, including recording, compiling and the idempotency campaign.
• Tests: compiler, CLI, public-demo evidence, qualification project, drills, and the new tests all pass after merging main: 334 passed.

Still per-app

• Sign-in, fixture reset and the OAuth read session for OpenEMR, as Python hooks.
• openemr_local_demo.compile_recording still binds OpenEMR's duplicate-dialog postcondition. This was already there; it is a benchmark compile helper.

Limits

• Web only. The pixel-surface fault driver is next.
• The certification policy does not hold the fault list, minimum effect tier or trial count. qualify requires all six cases, sets the tier from the configured read-back, and takes trials from campaign.trials.
• Keys are local test keys.
• A pass covers only the environment it ran in.

Review status (2026-10-17)

Fixed in 55b075e: the mypy error in qualify/faults.py, the unregistered verify.yaml files in public-artifacts.json, and tests/test_desktop_check_it_cli_mockmed.py, which still called helpers this PR removed from scripts/export_public_demo_evidence.py.

Fixed in f060c6c: test_attached_browser_run failed because static_control_identifier_crop gave an unlabeled web button a pixel crop as its only identity evidence. The pixel tier's VERIFY path is off by default, so that check could only abstain or reject, and the irreversible step halted on every run. runtime/identity.verifiable_identity_evidence(surface_kind, pixel_verify_enabled=...) now says which evidence a runtime can verify on a surface. The pass asks it per step (surface_kind_for hook for multi-surface bundles, #533) and adds the crop only for a runtime with pixel_verify_enabled (compile --pixel-verify-enabled). The runtime ladder is unchanged: a mismatching crop halts, and a crop the runtime can't verify never passes.

[abrichr]

Carried over into the monorepo as OpenAdaptAI/OpenAdapt#1186 (merged as 093b66e11569f0b62eb400f71646febf2cd0a68d). openadapt-flow now lives in OpenAdapt/packages/flow; please continue there.