[abrichr]

scaffold-verifier used to write a draft that checked nothing until a person filled it in. It now builds the record check from the reads the app made after the save, and tries to break it before anyone reviews it.

What it does

openadapt-flow scaffold-verifier rec/ [--db app.sqlite] [--verifier verify.yaml] [--prove mockmed|effect-e2e] [--bundle bundle/] [--complete-read]

1. Find the read. It looks for a GET after the save whose JSON holds a record with the recorded values (meta.json params): REST lists and FHIR Bundles from the network capture (network.jsonl, the openadapt.network-capture/1 format from feat(flow): API steps compiled from a recording's network capture #1269, read without importing the recorder side), a SQLite database (--db, every table), or a reviewed verifier section (--verifier).
2. Fill in the check. It writes record_check.proposed.yaml (the verify.yaml shape). The check expects exactly one new record, matched on the recorded values bound as {param: ...} plus fixed values the save sent. No other record may change or disappear. A read whose records carry an id gets a content twin (records_drop_fields: [id]), so a changed row with the same id is caught. With --db, every other table gets a guard: no row added, changed or removed. A write no read covers, or a table that couldn't be read, keeps the check a draft.
3. Prove it. record_check.proof.json holds the results of the effect_e2e fault classes applied to the recorded records. With --prove, the same check also runs end to end through the real replayer against MockMed or the effect_e2e service, judged by that fixture's own ground truth. Free-text values change on every run, so a check that froze the demo values fails the clean save. If the check misses any fault, it's written as # DRAFT with each MISS: listed, and the command exits 1.
4. A person approves. Every effect carries needs_operator_confirmation: true. verify-run and qualify refuse the file until a person removes those lines.
5. Identity coverage. With --bundle, it reports how many clicks have the right-record check armed, before and after evidence from the recording's DOM or accessibility text at each unarmed click. It only proposes evidence the runtime checks on that surface. The evidence text goes to identity.proposed.json (mode 0600).

New engine piece: effects.kind: surfaces (runtime/effects/surfaces.py) gives one read per table or resource. Each effect names its read in probe. An effect that names no read, or one that isn't configured, halts. When surfaces is unset, existing verifier identity digests don't change.

Results (synthetic, no model calls)

Fixture	Reads	Result
effect_e2e service	one REST read (/api/records)	6 fault classes caught; the stray billing write is missed, the same blind spot as EFFECT_E2E.md's 9 of 72. Written as a draft.
effect_e2e service	--db (encounters, banner, billing)	All 9 bad writes stopped end to end; clean save passes; timeout is an actuator stop. Working.
MockMed --break-it server	/api/db	Draft until --complete-read (declared, recorded as such); then 6 of 6 faults caught end to end.
OpenEMR-shaped FHIR fake	Observation?patient=	Scoped by the patient param; partial, duplicate, phantom, wrong patient and lost update refuted.
OpenEMR Standard API shape	/apis/default/api/patient	Same read, key and fields as the hand-written benchmark/openemr_local/verify.yaml.

Review

An independent review found two fail-open bugs, both fixed with regression tests. Skipped tables still counted as "every table read". A stale fault was credited when the selector couldn't tell the records apart. The same pass fixed:

• guards that missed changed or deleted rows;
• name collisions across sources;
• save timing (events are stamped after the screen settles);
• record ids in URLs leaking into names;
• generated reads that were never built or run;
• a broken capture that crashed the CLI;
• a read-only demo that got a check.

Not done

• The root CAPABILITIES.md row and scripts/build_llms_full.py exist only in docs: one map of what OpenAdapt can do #1260 (open). I'll update the row and re-run the script here once docs: one map of what OpenAdapt can do #1260 lands.
• Saves that update an existing record aren't handled yet; the check expects one new record.
• --db reads SQLite only. Other databases go through --verifier with kind: sql.

Docs: EFFECT_KIT.md (new section and the surfaces kind), claims.yaml → VERIFICATION.md (new claim record-check-from-app-reads; the identity caveat now names the coverage report).

🤖 Generated with Claude Code