diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 570e4eae85..b171b5d9c0 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -124,6 +124,7 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile one or more demonstrations into a deterministic program | Released | Real app (OpenEMR referral, 62 timed runs) | `openadapt-flow record`, `compile`; [packages/flow](packages/flow/README.md) | +| Classify common local edits without a person (select-all, Enter in a field or dropdown, an in-form tab, an informational dialog's dismiss), only when the recording's network capture shows no write and a runtime check confirms only the expected change | In review ([#1366](https://github.com/OpenAdaptAI/OpenAdapt/pull/1366)) | Synthetic; OpenEMR referral recording re-admitted offline (violations 12 to 1 under `permissive`) | [LOCAL_EDIT_CLASSIFICATION.md](packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md), `compiler/local_edit.py` | | Right-record check before any write (structure, then OCR, then pixels) | Released | Real app and synthetic | [RESOLUTION_LADDER.md](packages/flow/docs/RESOLUTION_LADDER.md), `runtime/identity.py` | | Read the saved record back from the system of record (API, database, FHIR, separate session) | Released | Real app (OpenEMR 12 of 12 fields) | [VERIFICATION.md](packages/flow/docs/VERIFICATION.md), `runtime/effects/` | | Read values out of an app as a workflow step (READ), with a records format, a signed review queue and export to CSV or a database | On main ([#1229](https://github.com/OpenAdaptAI/OpenAdapt/pull/1229), [#1215](https://github.com/OpenAdaptAI/OpenAdapt/pull/1215), [#1218](https://github.com/OpenAdaptAI/OpenAdapt/pull/1218)–[#1224](https://github.com/OpenAdaptAI/OpenAdapt/pull/1224)) | Synthetic (mock desktop apps) | `openadapt_flow/interchange/` | @@ -191,6 +192,7 @@ improvising or retrying when something is off. | Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#in-a-durable-run) | | Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | In review ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-governance) | | Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | In review ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | +| Say, for every feature under every run profile, whether it works or is refused and why, checked against the runtime in CI | In review ([#1364](https://github.com/OpenAdaptAI/OpenAdapt/pull/1364)) | Synthetic | [COMPATIBILITY.md](packages/flow/docs/COMPATIBILITY.md) | | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | diff --git a/llms-full.txt b/llms-full.txt index 8766f100b6..807a2c2ac8 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -481,6 +481,7 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile one or more demonstrations into a deterministic program | Released | Real app (OpenEMR referral, 62 timed runs) | `openadapt-flow record`, `compile`; [packages/flow](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/README.md) | +| Classify common local edits without a person (select-all, Enter in a field or dropdown, an in-form tab, an informational dialog's dismiss), only when the recording's network capture shows no write and a runtime check confirms only the expected change | In review ([#1366](https://github.com/OpenAdaptAI/OpenAdapt/pull/1366)) | Synthetic; OpenEMR referral recording re-admitted offline (violations 12 to 1 under `permissive`) | [LOCAL_EDIT_CLASSIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md), `compiler/local_edit.py` | | Right-record check before any write (structure, then OCR, then pixels) | Released | Real app and synthetic | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md), `runtime/identity.py` | | Read the saved record back from the system of record (API, database, FHIR, separate session) | Released | Real app (OpenEMR 12 of 12 fields) | [VERIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION.md), `runtime/effects/` | | Read values out of an app as a workflow step (READ), with a records format, a signed review queue and export to CSV or a database | On main ([#1229](https://github.com/OpenAdaptAI/OpenAdapt/pull/1229), [#1215](https://github.com/OpenAdaptAI/OpenAdapt/pull/1215), [#1218](https://github.com/OpenAdaptAI/OpenAdapt/pull/1218)–[#1224](https://github.com/OpenAdaptAI/OpenAdapt/pull/1224)) | Synthetic (mock desktop apps) | `openadapt_flow/interchange/` | @@ -548,6 +549,7 @@ improvising or retrying when something is off. | Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#in-a-durable-run) | | Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | In review ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-governance) | | Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | In review ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | +| Say, for every feature under every run profile, whether it works or is refused and why, checked against the runtime in CI | In review ([#1364](https://github.com/OpenAdaptAI/OpenAdapt/pull/1364)) | Synthetic | [COMPATIBILITY.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/COMPATIBILITY.md) | | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | diff --git a/packages/flow/benchmark/openemr_api_tier/approval_durable.py b/packages/flow/benchmark/openemr_api_tier/approval_durable.py new file mode 100644 index 0000000000..7b3e57f929 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/approval_durable.py @@ -0,0 +1,445 @@ +"""The OpenEMR referral with the API step under each_item approval and a durable run. + +Label every number from this harness exactly as: "API step under each_item +approval + durable, no run gate; harness answers approvals; approval wait +excluded". It is not the governed production profile. The referral bundle +doesn't pass the run gate yet (GUI-path admission; see docs/API_TIER.md and +the admission list in the session notes), so this measures the parts that +do run: the person approving the exact request (docs/APPROVAL_MODES.md) and +the durable checkpoints (docs/API_TIER.md "In a durable run"). + +- Arms: ``browser`` (API steps off) and ``api`` (API steps on), both under + ``ApprovalGate("each_item")`` with ``durable=True``. Interleaved blocks; each + block resets the fixture and seeds the chart r6 must not duplicate. +- Approvals: the harness stands in for the person and answers each request + as soon as it is written (``decided_by="human"``, route ``desktop_run_it``, + approver ``harness-approver``). Time spent waiting for the answer is + measured and subtracted: ``engine_s`` excludes it, ``approval_wait_s`` is it. +- Read-back: the test setup's own direct SQL check (``common.record_check``), + never the arm under test. r6 is right when no chart is created. +- Load: each item waits until the 1-minute load is under GATE_LOAD1 (12). + + PYTHONPATH=packages/flow python -m benchmark.openemr_api_tier.approval_durable \\ + compile --recording --out + PYTHONPATH=packages/flow python -m benchmark.openemr_api_tier.approval_durable \\ + run --bundle --out [--blocks 5 --per-block 4] +""" + +from __future__ import annotations + +import argparse +import csv +import json +import shutil +import statistics +import subprocess +import sys +import time +from pathlib import Path +from typing import Any + +from benchmark.openemr_api_tier.common import ( + REFERRALS, + load_demo, + record_check, + reset_and_seed, + spec_for, + sql_patients, +) +from benchmark.openemr_api_tier.feature_e2e import NO_MATCH_TEXT, _load1, wait_quiet +from benchmark.openemr_api_tier.spike_fixture import SpikeFixture + +LABEL = ( + "API step under each_item approval + durable, no run gate; harness answers " + "approvals; approval wait excluded" +) +APPROVER = "harness-approver" +COLS = [ + "arm", + "block", + "pos", + "ref", + "mismatch", + "correct", + "readback", + "new_charts", + "engine_success", + "transaction_outcome", + "actuation", + "api_fallback", + "approvals", + "approval_kinds", + "approval_wait_s", + "engine_s", + "wall_s", + "checkpoints", + "error", + "load_start", + "load_end", + "gate_waited_s", + "engine_sha", +] + + +def cmd_compile(a: argparse.Namespace) -> None: + """The harness compile (feature_e2e), admitted with a right-record parameter.""" + + from openadapt_flow.compiler.api_proposal import ( + admit_api_step, + describe_api_step, + propose_api_step, + ) + from openadapt_flow.ir import Postcondition, PostconditionKind, Workflow + + demo = load_demo() + rec, bundle = Path(a.recording), Path(a.out) + demo.compile_recording(rec, bundle) + wf = Workflow.load(bundle) + dialog = [ + s + for s in wf.steps + if s.anchor is not None + and s.anchor.structural is not None + and s.anchor.structural.selector == "#create" + ] + assert len(dialog) == 1 + dialog[0].expect = list(dialog[0].expect) + [ + Postcondition( + kind=PostconditionKind.TEXT_PRESENT, text=NO_MATCH_TEXT, timeout_s=10.0 + ) + ] + wf, step, reasons = propose_api_step(wf, rec, replace=True) + if step is None: + raise SystemExit(f"no API step: {reasons}") + # The harness admits under the fixture's name so the matrix runs + # unattended; in use a person admits with `openadapt-flow api admit`. + wf, _ = admit_api_step( + wf, admitted_by="benchmark-fixture", identity={"subject_name": "lname"} + ) + wf.save(bundle) + print(describe_api_step(next(s for s in wf.steps if s.id == step.id))) + + +class HarnessPerson: + """Answers every waiting approval request at once, and times the wait.""" + + def __init__(self, store: Any) -> None: + self.store = store + self.waited_s = 0.0 + self.answered: list[str] = [] + + def sleep(self, seconds: float) -> None: + from openadapt_flow.approval_mode.item_approval import make_answer + + t0 = time.monotonic() + for request in self.store.requests(): + if request.request_id in self.answered: + continue + if self.store.status(request) != "waiting": + continue + self.store.write_answer( + make_answer( + request, + decision="approve", + decided_by="human", + route="desktop_run_it", + approver_id=APPROVER, + auth_method="cloud_aal2", + ) + ) + self.answered.append(request.request_id) + time.sleep(min(seconds, 0.02)) + self.waited_s += time.monotonic() - t0 + + +def run_one( + env: dict[str, Any], arm: str, ref_idx: int, bundle: Path, out: Path +) -> dict[str, Any]: + from openadapt_flow.approval_mode.gate import ApprovalGate + from openadapt_flow.approval_mode.item_approval import ItemApprovalStore + from openadapt_flow.benchmark.openemr_local import OpenEMRPatientOracle + from openadapt_flow.ir import Workflow + from openadapt_flow.runtime import Replayer + from openadapt_flow.runtime.durable.checkpoint import CheckpointStore + + demo, f, browser, ocr = env["demo"], env["f"], env["browser"], env["ocr"] + ref = REFERRALS[ref_idx - 1] + spec = spec_for(ref) + out.mkdir(parents=True) + engine_dir = out / "engine" + before_ids = {r["id"] for r in sql_patients(f)} + oracle = OpenEMRPatientOracle(f.api_base_url, f.token_session("oracle"), spec) + res: dict[str, Any] = { + "arm": arm, + "ref": ref.key, + "mismatch": ref.mismatch, + "error": None, + } + store = ItemApprovalStore(engine_dir) + person = HarnessPerson(store) + gate = ApprovalGate( + "each_item", + approval_store=store, + approval_wait_s=120.0, + approval_poll_s=0.05, + sleep=person.sleep, + monotonic=time.monotonic, + ) + t0 = time.time() + ctx = browser.new_context( + viewport={"width": 1280, "height": 800}, device_scale_factor=1 + ) + try: + page = ctx.new_page() + page.goto(f.ui_base_url) + backend = demo._BasePlaywrightBackend(page) + demo._preauthenticate_browser(backend, f) + t_setup = time.time() + ocr.clear_ocr_memo() + rep = Replayer( + backend, + effect_verifier=oracle.verifier, + api_steps=(arm == "api"), + approval_gate=gate, + durable=True, + ).run( + Workflow.load(bundle), + params=spec.params(), + bundle_dir=bundle, + run_dir=engine_dir, + ) + t_task = time.time() + raw = rep.model_dump(mode="json") + last = raw["results"][-1] if raw["results"] else {} + res.update( + engine_success=raw.get("success"), + transaction_outcome=raw.get("transaction_outcome"), + actuation=last.get("actuation"), + api_fallback=next( + (r["api_fallback"] for r in raw["results"] if r.get("api_fallback")), + None, + ), + step_errors=[ + (r["step_id"], (r.get("error") or "")[:300]) + for r in raw["results"] + if r.get("error") + ], + ) + except Exception as exc: # noqa: BLE001 + t_setup = locals().get("t_setup", time.time()) + t_task = time.time() + res["error"] = f"{type(exc).__name__}: {exc}"[:500] + time.sleep(demo.ORACLE_SETTLE_S) + res.update(record_check(f, ref, before_ids)) + t_end = time.time() + ctx.close() + requests = store.requests() + res.update( + approvals=len(requests), + approval_kinds="+".join(r.form_state_kind for r in requests), + approval_wait_s=round(person.waited_s, 3), + engine_s=round(t_task - t_setup - person.waited_s, 3), + wall_s=round(t_end - t0, 3), + checkpoints=len(CheckpointStore(engine_dir).checkpoints()), + ) + (out / "result.json").write_text(json.dumps(res, indent=1, default=str)) + shutil.rmtree(engine_dir / "steps", ignore_errors=True) + return res + + +def cmd_run(a: argparse.Namespace) -> None: + from playwright.sync_api import sync_playwright + + import openadapt_flow.vision.ocr # noqa: F401 + + ocr = sys.modules["openadapt_flow.vision.ocr"] + out, bundle = Path(a.out), Path(a.bundle) + out.mkdir(parents=True, exist_ok=True) + sha = subprocess.run( + ["git", "rev-parse", "HEAD"], capture_output=True, text=True + ).stdout.strip()[:9] + demo, f = load_demo(), SpikeFixture() + pw = sync_playwright().start() + browser = pw.chromium.launch(headless=True) + ocr._get_engine() + env = {"demo": demo, "f": f, "browser": browser, "ocr": ocr} + csv_path = out / "runs.csv" + rows: list[dict[str, Any]] = [] + done = ( + { + d.name + for d in (Path(a.skip_done) / "items").iterdir() + if (d / "result.json").is_file() + } + if a.skip_done + else set() + ) + try: + for b in range(a.from_block - 1, a.blocks): + for arm in ["browser", "api"] if b % 2 == 0 else ["api", "browser"]: + names = [ + f"b{b + 1:02d}-{arm}-{pos + 1}-" + f"{REFERRALS[(b * a.per_block + pos) % len(REFERRALS)].key}" + for pos in range(a.per_block) + ] + if all(n in done for n in names): + continue + reset_and_seed(f) + for pos in range(a.per_block): + idx = ((b * a.per_block + pos) % len(REFERRALS)) + 1 + if names[pos] in done: + continue + gate = wait_quiet() + load_start = _load1() + res = run_one( + env, + arm, + idx, + bundle, + out + / "items" + / f"b{b + 1:02d}-{arm}-{pos + 1}-{REFERRALS[idx - 1].key}", + ) + row = {k: res.get(k) for k in COLS} | { + "block": b + 1, + "pos": pos + 1, + "load_start": load_start, + "load_end": _load1(), + "gate_waited_s": gate["gate_waited_s"], + "engine_sha": sha, + } + rows.append(row) + print( + json.dumps( + { + k: row[k] + for k in ( + "arm", + "block", + "ref", + "correct", + "actuation", + "approvals", + "engine_s", + "approval_wait_s", + "error", + ) + } + ), + flush=True, + ) + finally: + browser.close() + pw.stop() + with open(csv_path, "w", newline="") as fh: + w = csv.DictWriter(fh, fieldnames=COLS) + w.writeheader() + w.writerows(rows) + (out / "summary.json").write_text( + json.dumps(summarize(rows, sha), indent=1) + "\n" + ) + + +def summarize(rows: list[dict[str, Any]], sha: str) -> dict[str, Any]: + """Per arm: correctness by read-back, engine time, approvals.""" + + out: dict[str, Any] = {"label": LABEL, "engine_sha": sha, "arms": {}} + for arm in ("browser", "api"): + mine = [r for r in rows if r["arm"] == arm] + if not mine: + continue + times = [r["engine_s"] for r in mine if r["engine_s"] is not None] + out["arms"][arm] = { + "runs": len(mine), + "correct": sum(bool(r["correct"]) for r in mine), + "errors": sum(bool(r["error"]) for r in mine), + "api_saves": sum(r["actuation"] == "api" for r in mine), + "approvals_per_item": sorted({r["approvals"] for r in mine}), + "approval_kinds": sorted({r["approval_kinds"] for r in mine}), + "median_engine_s": round(statistics.median(times), 2) if times else None, + "median_approval_wait_s": round( + statistics.median(r["approval_wait_s"] for r in mine), 3 + ), + "max_load_start": max( + (r["load_start"] for r in mine if r["load_start"] is not None), + default=None, + ), + # Stopped before saving, nothing written (a fail-closed halt or a + # refused duplicate): not a wrong save. + "safe_stops": sum( + not r["correct"] and r.get("new_charts") == 0 for r in mine + ), + "wrong_saves": sum( + not r["correct"] and (r.get("new_charts") or 0) > 0 for r in mine + ), + } + return out + + +def rows_from_items(out: Path, sha: str) -> list[dict[str, Any]]: + """Rebuild the rows from each item's result.json (an interrupted run).""" + + rows = [] + for item in sorted((out / "items").iterdir()): + result = item / "result.json" + if not result.is_file(): + continue # interrupted before the item finished + res = json.loads(result.read_text()) + block, arm, pos = item.name.split("-")[:3] + rows.append( + {k: res.get(k) for k in COLS} + | { + "block": int(block[1:]), + "pos": int(pos), + "load_start": None, + "load_end": None, + "gate_waited_s": None, + "engine_sha": sha, + } + ) + return rows + + +def cmd_summarize(a: argparse.Namespace) -> None: + """Combine the finished items of one or more --out directories into --to.""" + + out = Path(a.to) + out.mkdir(parents=True, exist_ok=True) + rows = [row for d in a.out for row in rows_from_items(Path(d), a.engine_sha)] + with open(out / "runs.csv", "w", newline="") as fh: + w = csv.DictWriter(fh, fieldnames=COLS) + w.writeheader() + w.writerows(rows) + (out / "summary.json").write_text( + json.dumps(summarize(rows, a.engine_sha), indent=1) + "\n" + ) + print(json.dumps(summarize(rows, a.engine_sha), indent=1)) + + +def main() -> None: + ap = argparse.ArgumentParser() + sub = ap.add_subparsers(dest="cmd", required=True) + sm = sub.add_parser( + "summarize", help="Rebuild runs.csv and summary.json from items/" + ) + sm.add_argument("--out", required=True, action="append") + sm.add_argument("--to", required=True) + sm.add_argument("--engine-sha", required=True) + c = sub.add_parser("compile") + c.add_argument("--recording", required=True) + c.add_argument("--out", required=True) + r = sub.add_parser("run") + r.add_argument("--bundle", required=True) + r.add_argument("--out", required=True) + r.add_argument("--blocks", type=int, default=5) + r.add_argument("--per-block", type=int, default=4) + r.add_argument("--from-block", type=int, default=1) + r.add_argument( + "--skip-done", default=None, help="An earlier --out; skip its finished items" + ) + a = ap.parse_args() + {"compile": cmd_compile, "run": cmd_run, "summarize": cmd_summarize}[a.cmd](a) + + +if __name__ == "__main__": + main() diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-1-r1-nadia-brennan.json new file mode 100644 index 0000000000..f716662abd --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.023, + "engine_s": 1.114, + "wall_s": 4.369, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-2-r2-marcus-feld.json new file mode 100644 index 0000000000..21282d886e --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-2-r2-marcus-feld.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Marcus", + "got": "Marcus", + "match": true + }, + { + "field": "lname", + "want": "Feld", + "got": "Feld", + "match": true + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": "1958-09-02", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": "114 QUARRY HILL DRIVE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0124", + "got": "555-0124", + "match": true + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": "marcus.feld@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.023, + "engine_s": 1.092, + "wall_s": 3.806, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-3-r3-ines-carvalho.json new file mode 100644 index 0000000000..96a6aa09ff --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.021, + "engine_s": 1.053, + "wall_s": 3.758, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-4-r4-theo-lindqvist.json new file mode 100644 index 0000000000..b7d10d91cc --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.026, + "engine_s": 1.1, + "wall_s": 3.783, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-1-r1-nadia-brennan.json new file mode 100644 index 0000000000..78095f21cb --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.043, + "engine_s": 18.324, + "wall_s": 21.29, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-2-r2-marcus-feld.json new file mode 100644 index 0000000000..1f8d8bbe93 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-2-r2-marcus-feld.json @@ -0,0 +1,99 @@ +{ + "arm": "browser", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": false, + "transaction_outcome": "RECONCILIATION_REQUIRED", + "actuation": null, + "api_fallback": "API steps are off for this run", + "step_errors": [ + [ + "step_040", + "Identity check failed for step 'step_040' (click recorded visual target): a target was found positionally (rung 'structural', confidence 1.00) but its surrounding text does not match the recorded target's \u2014 expected '', observed 'Emergency Contact: Confirm Create New Patient Cance" + ] + ], + "new_charts": 0, + "readback": "0/12", + "correct": false, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": null, + "match": false + }, + { + "field": "fname", + "want": "Marcus", + "got": null, + "match": false + }, + { + "field": "lname", + "want": "Feld", + "got": null, + "match": false + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": null, + "match": false + }, + { + "field": "sex", + "want": "Male", + "got": null, + "match": false + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": null, + "match": false + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": null, + "match": false + }, + { + "field": "state", + "want": "MA", + "got": null, + "match": false + }, + { + "field": "postal_code", + "want": "02139", + "got": null, + "match": false + }, + { + "field": "phone_home", + "want": "555-0124", + "got": null, + "match": false + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": null, + "match": false + }, + { + "field": "country_code", + "want": "USA", + "got": null, + "match": false + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.048, + "engine_s": 14.588, + "wall_s": 17.402, + "checkpoints": 40 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-3-r3-ines-carvalho.json new file mode 100644 index 0000000000..df48c52c94 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.05, + "engine_s": 16.038, + "wall_s": 18.852, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-4-r4-theo-lindqvist.json new file mode 100644 index 0000000000..43307b2f25 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.049, + "engine_s": 15.285, + "wall_s": 18.056, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-1-r5-grace-adeyemi.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-1-r5-grace-adeyemi.json new file mode 100644 index 0000000000..d6f3067dc9 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-1-r5-grace-adeyemi.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r5-grace-adeyemi", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Grace", + "got": "Grace", + "match": true + }, + { + "field": "lname", + "want": "Adeyemi", + "got": "Adeyemi", + "match": true + }, + { + "field": "DOB", + "want": "1990-02-08", + "got": "1990-02-08", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "402 CEDAR ROW", + "got": "402 CEDAR ROW", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0145", + "got": "555-0145", + "match": true + }, + { + "field": "email", + "want": "grace.adeyemi@example.invalid", + "got": "grace.adeyemi@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.025, + "engine_s": 1.188, + "wall_s": 4.307, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-2-r6-rosa-delgado.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-2-r6-rosa-delgado.json new file mode 100644 index 0000000000..fa684296b2 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-2-r6-rosa-delgado.json @@ -0,0 +1,99 @@ +{ + "arm": "api", + "ref": "r6-rosa-delgado", + "mismatch": true, + "error": null, + "engine_success": false, + "transaction_outcome": "RECONCILIATION_REQUIRED", + "actuation": "dom", + "api_fallback": "the check before saving (step_039) did not pass: missing ['No matches were found.']; the save was not sent", + "step_errors": [ + [ + "step_039", + "Postconditions failed for step 'step_039' (click recorded visual target): expected screen state not reached (semantic drift) \u2014 failed: text_present 'No matches were found.' \u2014 run aborted" + ] + ], + "new_charts": 0, + "readback": "0/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": null, + "match": false + }, + { + "field": "fname", + "want": "Rosa", + "got": null, + "match": false + }, + { + "field": "lname", + "want": "Delgado", + "got": null, + "match": false + }, + { + "field": "DOB", + "want": "1958-03-22", + "got": null, + "match": false + }, + { + "field": "sex", + "want": "Female", + "got": null, + "match": false + }, + { + "field": "street", + "want": "7 MILL ROAD", + "got": null, + "match": false + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": null, + "match": false + }, + { + "field": "state", + "want": "MA", + "got": null, + "match": false + }, + { + "field": "postal_code", + "want": "02139", + "got": null, + "match": false + }, + { + "field": "phone_home", + "want": "555-0188", + "got": null, + "match": false + }, + { + "field": "email", + "want": "rosa.delgado@example.invalid", + "got": null, + "match": false + }, + { + "field": "country_code", + "want": "USA", + "got": null, + "match": false + } + ], + "approvals": 1, + "approval_kinds": "dom", + "approval_wait_s": 0.026, + "engine_s": 22.666, + "wall_s": 25.59, + "checkpoints": 39 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-3-r7-arlo-whitcombe.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-3-r7-arlo-whitcombe.json new file mode 100644 index 0000000000..8c2bd8ca13 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-3-r7-arlo-whitcombe.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r7-arlo-whitcombe", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Arlo", + "got": "Arlo", + "match": true + }, + { + "field": "lname", + "want": "Whitcombe", + "got": "Whitcombe", + "match": true + }, + { + "field": "DOB", + "want": "1966-05-30", + "got": "1966-05-30", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "88 BEACON TERRACE", + "got": "88 BEACON TERRACE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0152", + "got": "555-0152", + "match": true + }, + { + "field": "email", + "want": "arlo.whitcombe@example.invalid", + "got": "arlo.whitcombe@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.03, + "engine_s": 1.125, + "wall_s": 3.803, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-4-r8-priya-raman.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-4-r8-priya-raman.json new file mode 100644 index 0000000000..5d47989f47 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-4-r8-priya-raman.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r8-priya-raman", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Priya", + "got": "Priya", + "match": true + }, + { + "field": "lname", + "want": "Raman", + "got": "Raman", + "match": true + }, + { + "field": "DOB", + "want": "1975-10-14", + "got": "1975-10-14", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "15 STATION AVENUE", + "got": "15 STATION AVENUE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0167", + "got": "555-0167", + "match": true + }, + { + "field": "email", + "want": "priya.raman@example.invalid", + "got": "priya.raman@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.025, + "engine_s": 1.152, + "wall_s": 3.842, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-1-r5-grace-adeyemi.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-1-r5-grace-adeyemi.json new file mode 100644 index 0000000000..b13c98f80a --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-1-r5-grace-adeyemi.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r5-grace-adeyemi", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Grace", + "got": "Grace", + "match": true + }, + { + "field": "lname", + "want": "Adeyemi", + "got": "Adeyemi", + "match": true + }, + { + "field": "DOB", + "want": "1990-02-08", + "got": "1990-02-08", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "402 CEDAR ROW", + "got": "402 CEDAR ROW", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0145", + "got": "555-0145", + "match": true + }, + { + "field": "email", + "want": "grace.adeyemi@example.invalid", + "got": "grace.adeyemi@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.055, + "engine_s": 18.26, + "wall_s": 21.718, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-2-r6-rosa-delgado.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-2-r6-rosa-delgado.json new file mode 100644 index 0000000000..2c2218b941 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-2-r6-rosa-delgado.json @@ -0,0 +1,99 @@ +{ + "arm": "browser", + "ref": "r6-rosa-delgado", + "mismatch": true, + "error": null, + "engine_success": false, + "transaction_outcome": "RECONCILIATION_REQUIRED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [ + [ + "step_039", + "Postconditions failed for step 'step_039' (click recorded visual target): expected screen state not reached (semantic drift) \u2014 failed: text_present 'No matches were found.' \u2014 run aborted" + ] + ], + "new_charts": 0, + "readback": "0/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": null, + "match": false + }, + { + "field": "fname", + "want": "Rosa", + "got": null, + "match": false + }, + { + "field": "lname", + "want": "Delgado", + "got": null, + "match": false + }, + { + "field": "DOB", + "want": "1958-03-22", + "got": null, + "match": false + }, + { + "field": "sex", + "want": "Female", + "got": null, + "match": false + }, + { + "field": "street", + "want": "7 MILL ROAD", + "got": null, + "match": false + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": null, + "match": false + }, + { + "field": "state", + "want": "MA", + "got": null, + "match": false + }, + { + "field": "postal_code", + "want": "02139", + "got": null, + "match": false + }, + { + "field": "phone_home", + "want": "555-0188", + "got": null, + "match": false + }, + { + "field": "email", + "want": "rosa.delgado@example.invalid", + "got": null, + "match": false + }, + { + "field": "country_code", + "want": "USA", + "got": null, + "match": false + } + ], + "approvals": 1, + "approval_kinds": "dom", + "approval_wait_s": 0.031, + "engine_s": 24.871, + "wall_s": 28.107, + "checkpoints": 39 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-3-r7-arlo-whitcombe.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-3-r7-arlo-whitcombe.json new file mode 100644 index 0000000000..ee05a2c9ef --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-3-r7-arlo-whitcombe.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r7-arlo-whitcombe", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Arlo", + "got": "Arlo", + "match": true + }, + { + "field": "lname", + "want": "Whitcombe", + "got": "Whitcombe", + "match": true + }, + { + "field": "DOB", + "want": "1966-05-30", + "got": "1966-05-30", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "88 BEACON TERRACE", + "got": "88 BEACON TERRACE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0152", + "got": "555-0152", + "match": true + }, + { + "field": "email", + "want": "arlo.whitcombe@example.invalid", + "got": "arlo.whitcombe@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.047, + "engine_s": 16.847, + "wall_s": 19.804, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-4-r8-priya-raman.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-4-r8-priya-raman.json new file mode 100644 index 0000000000..5cfcac1c3a --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-4-r8-priya-raman.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r8-priya-raman", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Priya", + "got": "Priya", + "match": true + }, + { + "field": "lname", + "want": "Raman", + "got": "Raman", + "match": true + }, + { + "field": "DOB", + "want": "1975-10-14", + "got": "1975-10-14", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "15 STATION AVENUE", + "got": "15 STATION AVENUE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0167", + "got": "555-0167", + "match": true + }, + { + "field": "email", + "want": "priya.raman@example.invalid", + "got": "priya.raman@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.048, + "engine_s": 15.714, + "wall_s": 18.419, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-1-r1-nadia-brennan.json new file mode 100644 index 0000000000..da326917ac --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.027, + "engine_s": 1.822, + "wall_s": 5.114, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-2-r2-marcus-feld.json new file mode 100644 index 0000000000..8dc00a019e --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-2-r2-marcus-feld.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Marcus", + "got": "Marcus", + "match": true + }, + { + "field": "lname", + "want": "Feld", + "got": "Feld", + "match": true + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": "1958-09-02", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": "114 QUARRY HILL DRIVE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0124", + "got": "555-0124", + "match": true + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": "marcus.feld@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.028, + "engine_s": 1.102, + "wall_s": 3.921, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-3-r3-ines-carvalho.json new file mode 100644 index 0000000000..e06609ff5e --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.031, + "engine_s": 1.081, + "wall_s": 3.904, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-4-r4-theo-lindqvist.json new file mode 100644 index 0000000000..86214e809a --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.025, + "engine_s": 1.106, + "wall_s": 3.773, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-1-r1-nadia-brennan.json new file mode 100644 index 0000000000..fe0d8e75c4 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.047, + "engine_s": 15.583, + "wall_s": 18.693, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-2-r2-marcus-feld.json new file mode 100644 index 0000000000..59874bcad9 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-2-r2-marcus-feld.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Marcus", + "got": "Marcus", + "match": true + }, + { + "field": "lname", + "want": "Feld", + "got": "Feld", + "match": true + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": "1958-09-02", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": "114 QUARRY HILL DRIVE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0124", + "got": "555-0124", + "match": true + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": "marcus.feld@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.052, + "engine_s": 15.871, + "wall_s": 18.792, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-3-r3-ines-carvalho.json new file mode 100644 index 0000000000..6ee11a79c3 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.047, + "engine_s": 22.049, + "wall_s": 25.165, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-4-r4-theo-lindqvist.json new file mode 100644 index 0000000000..d485d931df --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.053, + "engine_s": 15.59, + "wall_s": 18.753, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-1-r5-grace-adeyemi.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-1-r5-grace-adeyemi.json new file mode 100644 index 0000000000..e0ac70c6c6 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-1-r5-grace-adeyemi.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r5-grace-adeyemi", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Grace", + "got": "Grace", + "match": true + }, + { + "field": "lname", + "want": "Adeyemi", + "got": "Adeyemi", + "match": true + }, + { + "field": "DOB", + "want": "1990-02-08", + "got": "1990-02-08", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "402 CEDAR ROW", + "got": "402 CEDAR ROW", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0145", + "got": "555-0145", + "match": true + }, + { + "field": "email", + "want": "grace.adeyemi@example.invalid", + "got": "grace.adeyemi@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.028, + "engine_s": 1.082, + "wall_s": 4.212, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-2-r6-rosa-delgado.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-2-r6-rosa-delgado.json new file mode 100644 index 0000000000..d7d80d8e3c --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-2-r6-rosa-delgado.json @@ -0,0 +1,99 @@ +{ + "arm": "api", + "ref": "r6-rosa-delgado", + "mismatch": true, + "error": null, + "engine_success": false, + "transaction_outcome": "RECONCILIATION_REQUIRED", + "actuation": "dom", + "api_fallback": "the check before saving (step_039) did not pass: missing ['No matches were found.']; the save was not sent", + "step_errors": [ + [ + "step_039", + "Postconditions failed for step 'step_039' (click recorded visual target): expected screen state not reached (semantic drift) \u2014 failed: text_present 'No matches were found.' \u2014 run aborted" + ] + ], + "new_charts": 0, + "readback": "0/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": null, + "match": false + }, + { + "field": "fname", + "want": "Rosa", + "got": null, + "match": false + }, + { + "field": "lname", + "want": "Delgado", + "got": null, + "match": false + }, + { + "field": "DOB", + "want": "1958-03-22", + "got": null, + "match": false + }, + { + "field": "sex", + "want": "Female", + "got": null, + "match": false + }, + { + "field": "street", + "want": "7 MILL ROAD", + "got": null, + "match": false + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": null, + "match": false + }, + { + "field": "state", + "want": "MA", + "got": null, + "match": false + }, + { + "field": "postal_code", + "want": "02139", + "got": null, + "match": false + }, + { + "field": "phone_home", + "want": "555-0188", + "got": null, + "match": false + }, + { + "field": "email", + "want": "rosa.delgado@example.invalid", + "got": null, + "match": false + }, + { + "field": "country_code", + "want": "USA", + "got": null, + "match": false + } + ], + "approvals": 1, + "approval_kinds": "dom", + "approval_wait_s": 0.03, + "engine_s": 22.625, + "wall_s": 25.706, + "checkpoints": 39 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-3-r7-arlo-whitcombe.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-3-r7-arlo-whitcombe.json new file mode 100644 index 0000000000..a7173128c7 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-3-r7-arlo-whitcombe.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r7-arlo-whitcombe", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Arlo", + "got": "Arlo", + "match": true + }, + { + "field": "lname", + "want": "Whitcombe", + "got": "Whitcombe", + "match": true + }, + { + "field": "DOB", + "want": "1966-05-30", + "got": "1966-05-30", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "88 BEACON TERRACE", + "got": "88 BEACON TERRACE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0152", + "got": "555-0152", + "match": true + }, + { + "field": "email", + "want": "arlo.whitcombe@example.invalid", + "got": "arlo.whitcombe@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.03, + "engine_s": 1.123, + "wall_s": 3.806, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-4-r8-priya-raman.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-4-r8-priya-raman.json new file mode 100644 index 0000000000..5d8d09a2f1 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-4-r8-priya-raman.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r8-priya-raman", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Priya", + "got": "Priya", + "match": true + }, + { + "field": "lname", + "want": "Raman", + "got": "Raman", + "match": true + }, + { + "field": "DOB", + "want": "1975-10-14", + "got": "1975-10-14", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "15 STATION AVENUE", + "got": "15 STATION AVENUE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0167", + "got": "555-0167", + "match": true + }, + { + "field": "email", + "want": "priya.raman@example.invalid", + "got": "priya.raman@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.027, + "engine_s": 1.057, + "wall_s": 3.865, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-1-r5-grace-adeyemi.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-1-r5-grace-adeyemi.json new file mode 100644 index 0000000000..986266edf7 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-1-r5-grace-adeyemi.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r5-grace-adeyemi", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Grace", + "got": "Grace", + "match": true + }, + { + "field": "lname", + "want": "Adeyemi", + "got": "Adeyemi", + "match": true + }, + { + "field": "DOB", + "want": "1990-02-08", + "got": "1990-02-08", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "402 CEDAR ROW", + "got": "402 CEDAR ROW", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0145", + "got": "555-0145", + "match": true + }, + { + "field": "email", + "want": "grace.adeyemi@example.invalid", + "got": "grace.adeyemi@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.048, + "engine_s": 15.514, + "wall_s": 18.949, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-2-r6-rosa-delgado.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-2-r6-rosa-delgado.json new file mode 100644 index 0000000000..9018603155 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-2-r6-rosa-delgado.json @@ -0,0 +1,99 @@ +{ + "arm": "browser", + "ref": "r6-rosa-delgado", + "mismatch": true, + "error": null, + "engine_success": false, + "transaction_outcome": "RECONCILIATION_REQUIRED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [ + [ + "step_039", + "Postconditions failed for step 'step_039' (click recorded visual target): expected screen state not reached (semantic drift) \u2014 failed: text_present 'No matches were found.' \u2014 run aborted" + ] + ], + "new_charts": 0, + "readback": "0/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": null, + "match": false + }, + { + "field": "fname", + "want": "Rosa", + "got": null, + "match": false + }, + { + "field": "lname", + "want": "Delgado", + "got": null, + "match": false + }, + { + "field": "DOB", + "want": "1958-03-22", + "got": null, + "match": false + }, + { + "field": "sex", + "want": "Female", + "got": null, + "match": false + }, + { + "field": "street", + "want": "7 MILL ROAD", + "got": null, + "match": false + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": null, + "match": false + }, + { + "field": "state", + "want": "MA", + "got": null, + "match": false + }, + { + "field": "postal_code", + "want": "02139", + "got": null, + "match": false + }, + { + "field": "phone_home", + "want": "555-0188", + "got": null, + "match": false + }, + { + "field": "email", + "want": "rosa.delgado@example.invalid", + "got": null, + "match": false + }, + { + "field": "country_code", + "want": "USA", + "got": null, + "match": false + } + ], + "approvals": 1, + "approval_kinds": "dom", + "approval_wait_s": 0.022, + "engine_s": 22.555, + "wall_s": 25.298, + "checkpoints": 39 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-3-r7-arlo-whitcombe.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-3-r7-arlo-whitcombe.json new file mode 100644 index 0000000000..858ce9b659 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-3-r7-arlo-whitcombe.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r7-arlo-whitcombe", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Arlo", + "got": "Arlo", + "match": true + }, + { + "field": "lname", + "want": "Whitcombe", + "got": "Whitcombe", + "match": true + }, + { + "field": "DOB", + "want": "1966-05-30", + "got": "1966-05-30", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "88 BEACON TERRACE", + "got": "88 BEACON TERRACE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0152", + "got": "555-0152", + "match": true + }, + { + "field": "email", + "want": "arlo.whitcombe@example.invalid", + "got": "arlo.whitcombe@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.053, + "engine_s": 15.174, + "wall_s": 17.925, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-4-r8-priya-raman.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-4-r8-priya-raman.json new file mode 100644 index 0000000000..0ead314989 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-4-r8-priya-raman.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r8-priya-raman", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Priya", + "got": "Priya", + "match": true + }, + { + "field": "lname", + "want": "Raman", + "got": "Raman", + "match": true + }, + { + "field": "DOB", + "want": "1975-10-14", + "got": "1975-10-14", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "15 STATION AVENUE", + "got": "15 STATION AVENUE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0167", + "got": "555-0167", + "match": true + }, + { + "field": "email", + "want": "priya.raman@example.invalid", + "got": "priya.raman@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.043, + "engine_s": 15.596, + "wall_s": 18.571, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-1-r1-nadia-brennan.json new file mode 100644 index 0000000000..2974ad6af7 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.028, + "engine_s": 1.3, + "wall_s": 4.671, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-2-r2-marcus-feld.json new file mode 100644 index 0000000000..d824da6223 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-2-r2-marcus-feld.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Marcus", + "got": "Marcus", + "match": true + }, + { + "field": "lname", + "want": "Feld", + "got": "Feld", + "match": true + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": "1958-09-02", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": "114 QUARRY HILL DRIVE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0124", + "got": "555-0124", + "match": true + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": "marcus.feld@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.029, + "engine_s": 1.363, + "wall_s": 4.065, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-3-r3-ines-carvalho.json new file mode 100644 index 0000000000..060a97af06 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.026, + "engine_s": 1.495, + "wall_s": 4.167, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-4-r4-theo-lindqvist.json new file mode 100644 index 0000000000..e6bc091930 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.026, + "engine_s": 1.533, + "wall_s": 4.193, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-1-r1-nadia-brennan.json new file mode 100644 index 0000000000..f00af8a918 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.049, + "engine_s": 20.218, + "wall_s": 23.385, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-2-r2-marcus-feld.json new file mode 100644 index 0000000000..672a014682 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-2-r2-marcus-feld.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Marcus", + "got": "Marcus", + "match": true + }, + { + "field": "lname", + "want": "Feld", + "got": "Feld", + "match": true + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": "1958-09-02", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": "114 QUARRY HILL DRIVE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0124", + "got": "555-0124", + "match": true + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": "marcus.feld@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.053, + "engine_s": 18.634, + "wall_s": 21.533, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-3-r3-ines-carvalho.json new file mode 100644 index 0000000000..213defb27b --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.06, + "engine_s": 16.615, + "wall_s": 19.53, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-4-r4-theo-lindqvist.json new file mode 100644 index 0000000000..dbfeff64ec --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.052, + "engine_s": 15.536, + "wall_s": 18.458, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/runs.csv b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/runs.csv new file mode 100644 index 0000000000..0835c9741f --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/runs.csv @@ -0,0 +1,41 @@ +arm,block,pos,ref,mismatch,correct,readback,new_charts,engine_success,transaction_outcome,actuation,api_fallback,approvals,approval_kinds,approval_wait_s,engine_s,wall_s,checkpoints,error,load_start,load_end,gate_waited_s,engine_sha +api,1,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.023,1.114,4.369,41,,,,,352af585b +api,1,2,r2-marcus-feld,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.023,1.092,3.806,41,,,,,352af585b +api,1,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.021,1.053,3.758,41,,,,,352af585b +api,1,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.026,1.1,3.783,41,,,,,352af585b +browser,1,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.043,18.324,21.29,41,,,,,352af585b +browser,1,2,r2-marcus-feld,False,False,0/12,0,False,RECONCILIATION_REQUIRED,,API steps are off for this run,2,dom+dom,0.048,14.588,17.402,40,,,,,352af585b +browser,1,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.05,16.038,18.852,41,,,,,352af585b +browser,1,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.049,15.285,18.056,41,,,,,352af585b +api,2,1,r5-grace-adeyemi,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.025,1.188,4.307,41,,,,,352af585b +api,2,2,r6-rosa-delgado,True,True,0/12,0,False,RECONCILIATION_REQUIRED,dom,the check before saving (step_039) did not pass: missing ['No matches were found.']; the save was not sent,1,dom,0.026,22.666,25.59,39,,,,,352af585b +api,2,3,r7-arlo-whitcombe,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.03,1.125,3.803,41,,,,,352af585b +api,2,4,r8-priya-raman,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.025,1.152,3.842,41,,,,,352af585b +browser,2,1,r5-grace-adeyemi,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.055,18.26,21.718,41,,,,,352af585b +browser,2,2,r6-rosa-delgado,True,True,0/12,0,False,RECONCILIATION_REQUIRED,dom,API steps are off for this run,1,dom,0.031,24.871,28.107,39,,,,,352af585b +browser,2,3,r7-arlo-whitcombe,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.047,16.847,19.804,41,,,,,352af585b +browser,2,4,r8-priya-raman,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.048,15.714,18.419,41,,,,,352af585b +browser,3,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.047,15.583,18.693,41,,,,,352af585b +browser,3,2,r2-marcus-feld,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.052,15.871,18.792,41,,,,,352af585b +browser,3,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.047,22.049,25.165,41,,,,,352af585b +api,3,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.027,1.822,5.114,41,,,,,352af585b +api,3,2,r2-marcus-feld,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.028,1.102,3.921,41,,,,,352af585b +api,3,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.031,1.081,3.904,41,,,,,352af585b +api,3,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.025,1.106,3.773,41,,,,,352af585b +browser,3,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.053,15.59,18.753,41,,,,,352af585b +api,4,1,r5-grace-adeyemi,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.028,1.082,4.212,41,,,,,352af585b +api,4,2,r6-rosa-delgado,True,True,0/12,0,False,RECONCILIATION_REQUIRED,dom,the check before saving (step_039) did not pass: missing ['No matches were found.']; the save was not sent,1,dom,0.03,22.625,25.706,39,,,,,352af585b +api,4,3,r7-arlo-whitcombe,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.03,1.123,3.806,41,,,,,352af585b +api,4,4,r8-priya-raman,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.027,1.057,3.865,41,,,,,352af585b +browser,4,1,r5-grace-adeyemi,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.048,15.514,18.949,41,,,,,352af585b +browser,4,2,r6-rosa-delgado,True,True,0/12,0,False,RECONCILIATION_REQUIRED,dom,API steps are off for this run,1,dom,0.022,22.555,25.298,39,,,,,352af585b +browser,4,3,r7-arlo-whitcombe,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.053,15.174,17.925,41,,,,,352af585b +browser,4,4,r8-priya-raman,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.043,15.596,18.571,41,,,,,352af585b +api,5,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.028,1.3,4.671,41,,,,,352af585b +api,5,2,r2-marcus-feld,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.029,1.363,4.065,41,,,,,352af585b +api,5,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.026,1.495,4.167,41,,,,,352af585b +api,5,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.026,1.533,4.193,41,,,,,352af585b +browser,5,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.049,20.218,23.385,41,,,,,352af585b +browser,5,2,r2-marcus-feld,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.053,18.634,21.533,41,,,,,352af585b +browser,5,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.06,16.615,19.53,41,,,,,352af585b +browser,5,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.052,15.536,18.458,41,,,,,352af585b diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/summary.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/summary.json new file mode 100644 index 0000000000..ce31870351 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/summary.json @@ -0,0 +1,43 @@ +{ + "label": "API step under each_item approval + durable, no run gate; harness answers approvals; approval wait excluded", + "engine_sha": "352af585b", + "arms": { + "browser": { + "runs": 20, + "correct": 19, + "errors": 0, + "api_saves": 0, + "approvals_per_item": [ + 1, + 2 + ], + "approval_kinds": [ + "dom", + "dom+dom" + ], + "median_engine_s": 15.95, + "median_approval_wait_s": 0.049, + "max_load_start": null, + "safe_stops": 1, + "wrong_saves": 0 + }, + "api": { + "runs": 20, + "correct": 20, + "errors": 0, + "api_saves": 18, + "approvals_per_item": [ + 1 + ], + "approval_kinds": [ + "api_request", + "dom" + ], + "median_engine_s": 1.12, + "median_approval_wait_s": 0.026, + "max_load_start": null, + "safe_stops": 0, + "wrong_saves": 0 + } + } +} diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/README.md b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/README.md new file mode 100644 index 0000000000..8061b73ccc --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/README.md @@ -0,0 +1,55 @@ +# OpenEMR referral: API step under per-item approval and a durable run + +**Label: API step under each_item approval + durable, no run gate; harness +answers approvals; approval wait excluded.** These are not numbers for the +governed production profile. The referral bundle doesn't pass the run gate +yet: it is refused on its GUI path, for reasons unrelated to the API step +(see the admission list in the session notes, and the local-edit proposal, +PR #1366). + +Harness: [`approval_durable.py`](../../approval_durable.py). Data: +[`2026-10-19-combined/`](2026-10-19-combined/) (`runs.csv`, `summary.json`, +and each item's `result.json` under `items/`). + +## What ran + +- The OpenEMR new-patient referral, compiled from a recording with network + capture and admitted with `--identity subject_name=lname`, on a local + OpenEMR test copy with synthetic patients only. +- Two arms with the same bundle and record check: API steps off (`browser`) + and on (`api`). Both use `ApprovalGate("each_item")` and `durable=True`. +- 5 blocks of 4 referrals per arm, interleaved, with the arm order alternating + by block. Each block resets the fixture and seeds the existing chart that + referral r6 must not duplicate. +- The harness answers each approval request as soon as it is written, + standing in for the person. The time spent waiting for that answer is + measured (`approval_wait_s`) and left out of `engine_s`. +- Correctness is checked by the test setup's own direct SQL read-back. A new + referral is correct when exactly one new chart exists and it matches all 12 + fields. r6 is correct when no chart is created. +- Each item waited until the machine's 1-minute load was under 12. The run + was paused for a shared quiet window after 19 of 40 runs and resumed into a + second directory. The interrupted item was run again from the start. Load + per item was not kept for the rebuilt rows. +- Engine: the API-step stack (PRs #1358, #1361, #1362, #1364) at `352af585b`. + It does not include the review fixes #1367–#1372. With #1370 and #1371 this + referral's API step is refused: OpenEMR sends a background write while the + form is filled, and its form runs script on submit and on the name fields. + +## Results + +| Arm | Runs | Correct by read-back | Saved through | Median engine time | Approvals per item | +| --- | --- | --- | --- | --- | --- | +| API steps off | 20 | 19, plus 1 safe stop (nothing saved) | GUI | 15.95 s | 2 (1 for r6) | +| API step | 20 | 20 | API 18, GUI 2 (r6) | 1.12 s | 1 | + +- **API step.** Each new referral sent one request the person approved, with + the operation digest bound to the request bytes, and read back 12 of 12. + For r6 the API step's duplicate pre-check found the existing chart, so the + API step was not used. The GUI steps ran instead and created nothing, as + they should. +- **API steps off.** The GUI asks twice per new referral: once at "Create New + Patient", and once at the confirming save. One item (b01, r2) stopped itself + at the save's identity check (coverage 0.81) and saved nothing. The + earlier spike saw the same fail-closed stop once in 20 runs. +- No run saved a wrong or duplicate chart. diff --git a/packages/flow/docs/API_TIER.md b/packages/flow/docs/API_TIER.md index c47357d772..6060bfbdd0 100644 --- a/packages/flow/docs/API_TIER.md +++ b/packages/flow/docs/API_TIER.md @@ -1,9 +1,14 @@ # API steps from a recording's network traffic -A web form save that OpenAdapt can repeat as one HTTP request runs as one -request instead of as the clicks and keystrokes that filled the form. The -right-record check before the save and the record check after it stay the -same. If anything the API step depends on has changed, the recorded GUI +**Scope: a single HTML form submission.** An API step repeats one +`application/x-www-form-urlencoded` POST of one HTML form, read again from +the live page, in place of the clicks and keystrokes that filled it. It is +not a general API client: JSON or script-built requests, more than one +write, multi-page server state, and forms whose script validates or rewrites +values all stay on the GUI. + +The right-record check before the save and the record check after it stay +the same. If anything the API step depends on has changed, the recorded GUI steps run instead. Its speed is measured on the local OpenEMR referral as arm `program_api` of @@ -11,6 +16,12 @@ the benchmark scoreboard, against the same program on the GUI (`program`), with the same record check. The harness and the earlier spike are in [benchmark/openemr_api_tier](../benchmark/openemr_api_tier/README.md). +Under per-item approval and a durable run (no run gate; the harness answers +the approvals and their wait is left out), the API step saved each new +referral correctly with one approval at a median 1.12 s of engine time, +against 15.95 s and two approvals with API steps off (20 runs each, +2026-10-19): [results](../benchmark/openemr_api_tier/results/approval_durable/README.md). + ## How to use it ```bash @@ -40,9 +51,11 @@ openadapt-flow run bundle/ ... first of those steps carries `api_fallback` with the reason, and the GUI steps run as recorded. -If you change a check on one of the covered steps after compiling (for -example, you add a required on-screen text), the API step stops running -because it would skip that check. Run `openadapt-flow api propose bundle/ +If you change any of the covered steps after compiling (an action, a typed +value or parameter, a target or its identity evidence, a guard, an on-screen +check, or the save's record check), the API step stops running: its +admission is bound to a digest of the complete definition of every step it +replaces, the save included. Run `openadapt-flow api propose bundle/ --recording rec/` to propose it again, then admit it. ## What the compiler checks @@ -80,6 +93,10 @@ doesn't propose an API step in these cases: - the save sends a secret field; - the form page wasn't captured; - the save isn't the program's last step; +- the steps the API step would replace also sent another write (a POST, + PUT, PATCH or DELETE other than the save, such as a reservation, an + autosave or a background call). Admission and runs refuse such a binding + too; - the program has branches; - a URL value equals two inputs' values (ambiguous), or the save or a pre-check is on another origin than the form; @@ -108,9 +125,16 @@ Before it sends the save, the API step: 1. reads the form page again in this session; 2. refuses if the form's fields or its client-side validation rules (for example, OpenEMR's `constraints` object) differ from the recording; -3. builds the body, and refuses if a field the form requires would be - empty; -4. runs the pre-checks. +3. refuses if the form runs script it can't run: an `on*` handler on the + form (`onsubmit`) or on a field it sends (`onchange`, `oninput`, + `onblur`, script that can validate or rewrite a value), a page script + listening for the submit, or a validate.js rule kind other than + `presence`, `length`, `format`, `numericality` and `email`; +4. builds the body and checks it the way the browser would before it + submits: native HTML constraints (`required`, `minlength`, `maxlength`, + `pattern`, `min`, `max`, and the `email`, `url`, `number` and `date` + input types) and the page's validate.js rules; +5. runs the pre-checks. Each of these refusals means the save wasn't sent, so the GUI steps run. After the save request starts, the API tier's no-double-write rule @@ -239,6 +263,9 @@ Not yet covered, so the GUI steps run with that reason: probes and resumed run use the GUI steps, because the repair and the person's approval are about the GUI step ([SELF_REPAIR.md](SELF_REPAIR.md)). +Every feature under every run profile, and the API step with each other +feature, is in [COMPATIBILITY.md](COMPATIBILITY.md). + ## When it doesn't run The API step runs only in a plain linear run. It doesn't run, and the GUI @@ -291,9 +318,16 @@ recording. over a websocket aren't proposed. - The server may enforce less than the page. On OpenEMR, the duplicate check and the required fields run only in the browser. The API step - carries over the checks it can see (pre-checks and `constraints` - presence rules). Logic in the page's script that it can't see is the main - risk of this tier, and it's why a person admits each API step. + carries over the checks it can evaluate (pre-checks, native HTML + constraints and validate.js rules of the kinds above), and leaves a form + whose script it can't run to the GUI. A `pattern` is evaluated with + Python's regular expressions, which differ from the browser's in rare + syntax; one that doesn't compile keeps the save on the GUI. Letting + qualification show that the server enforces a script-only rule, so the + API step can run anyway, isn't built yet. +- OpenEMR's new-patient form runs script on submit (`submitme(...)`) and on + each name field (`capitalizeMe`, which rewrites the typed value), and its + `constraints` use a custom `pastDate` rule. So its save stays on the GUI. - Multi-step server state (wizards, uploads, locks held between pages) needs the whole request sequence, and isn't proposed. diff --git a/packages/flow/docs/COMPATIBILITY.md b/packages/flow/docs/COMPATIBILITY.md new file mode 100644 index 0000000000..9b4b5c395d --- /dev/null +++ b/packages/flow/docs/COMPATIBILITY.md @@ -0,0 +1,180 @@ +# Compatibility: features and run profiles + + + +Each feature either works under a run profile or is refused with a +named reason. Nothing falls back silently. When a run takes the slower or +narrower path, it records why: an API step's `api_fallback`, a +pipelined read-back's `inline_reason`, or the command's refusal before +anything runs. `tests/test_compatibility_matrix.py` checks every cell +against the runtime, and fails when a cell that should work falls back +or a refusal gives another reason. + +## The widest combination today + +A Regulated, durable, governed run with per-item approval, on parallel +workers, using the API step for an admitted save, is the combination +where the most work together: each worker's item sends one request a +person approved, checks it with the system-of-record read-back, and +reads `VERIFIED`. In that run the read-back runs inline (durable), and +self-repair runs through the Python API under the Standard contract. +Under a signed qualification admission (what `openadapt-flow run` +requires for Standard and Regulated actuation) the API step and +self-repair are still refused, each with its reason below. + +## Profiles + +- **Default (gated).** `openadapt-flow run` with no `--profile`: a run authorization from the run gate, and per-item approval (`each_item`), the default approval mode. +- **Durable.** `--durable`: a checkpoint after every verified step, and pause and resume from the last one. +- **Governed.** The Standard or Regulated profile contract with a run authorization from the run gate: durable, an identity check on every save, and a record check that reads the system of record. This is how the Python API and the tutorial run it. +- **Pinned.** `--pin-digest`: the run refuses any bundle but the one with that sealed content digest. +- **Qualified.** A signed production qualification admission (`--qualification-authority-file`), which `openadapt-flow run` requires for Standard and Regulated actuation. +- **Managed.** A Cloud-dispatched run (`--managed-dispatch-file`), whose run authorization comes from Cloud. + +## Features + +- **API step** ([API_TIER.md](API_TIER.md)). An admitted save compiled from network capture, run as one request in place of its GUI steps. +- **Pipelined read-back** ([THROUGHPUT.md](THROUGHPUT.md)). `--readback-window`: an item's record check runs while the next item starts. +- **Self-repair** ([SELF_REPAIR.md](SELF_REPAIR.md)). `--self-repair`: an agent proposes a fix for a halt on a changed screen; probes and a person's approval decide it. +- **Standard changes** ([SELF_REPAIR.md](SELF_REPAIR.md#standard-changes-repairs-without-a-person)). Self-repair fixes of a class a person approved in advance, in a signed repair policy. +- **Parallel workers** ([THROUGHPUT.md](THROUGHPUT.md)). `--workers N`: worklist items on several signed-in sessions at once. +- **Multi-surface** ([MULTI_SURFACE.md](MULTI_SURFACE.md)). One program across several windows, tabs or apps. + +## At a glance + +| Feature | Default (gated) | Durable | Governed | Pinned | Qualified | Managed | +| --- | --- | --- | --- | --- | --- | --- | +| API step | Works | Works | Works | Works | Refused | Refused | +| Pipelined read-back | Works | Refused | Refused | Works | Refused | Refused | +| Self-repair | Works | Works | Works | Refused | Refused | Refused | +| Standard changes | Works | Works | Works | Refused | Refused | Refused | +| Parallel workers | Works | Works | Works | Works | Works | Refused | +| Multi-surface | Works | Works | Works | Works | Works | Works | + +Every refusal names its reason (below), and a run or command that +refuses says it in those words. + +## Features by profile + +### API step + +| Profile | Result | +| --- | --- | +| Default (gated) | Works: the person approves the exact request | +| Durable | Works: a save that may have landed is never resent | +| Governed | Works: needs `api admit --identity` | +| Pinned | Works: the pin is checked before the run | +| Qualified | Refused: a production qualification admission doesn't cover API steps yet, so the qualified GUI path runs | +| Managed | Refused: a managed dispatch doesn't bind API steps yet | + +### Pipelined read-back + +| Profile | Result | +| --- | --- | +| Default (gated) | Works: GUI and API saves | +| Durable | Refused: durable runs verify inline (the record check runs before the next item) | +| Governed | Refused: durable runs verify inline (Standard and Regulated are durable) | +| Pinned | Works: when the run isn't durable | +| Qualified | Refused: durable runs verify inline (Standard and Regulated are durable) | +| Managed | Refused: --workers and --readback-window apply to local runs, not a managed dispatch or a qualification case | + +### Self-repair + +| Profile | Result | +| --- | --- | +| Default (gated) | Works | +| Durable | Works: self-repair always runs durable | +| Governed | Works: Standard contract through the Python API | +| Pinned | Refused: --pin-digest pins the original bundle | +| Qualified | Refused: a signed qualification admission binds the original bundle; the approved bundle needs its own admission | +| Managed | Refused: a managed dispatch binds the original bundle | + +### Standard changes + +| Profile | Result | +| --- | --- | +| Default (gated) | Works: needs a signed repair policy | +| Durable | Works: needs a signed repair policy | +| Governed | Works: needs a signed repair policy | +| Pinned | Refused: --pin-digest pins the original bundle | +| Qualified | Refused: a signed qualification admission binds the original bundle; the approved bundle needs its own admission | +| Managed | Refused: a managed dispatch binds the original bundle | + +### Parallel workers + +| Profile | Result | +| --- | --- | +| Default (gated) | Works | +| Durable | Works: each item has its own durable run | +| Governed | Works: each item gets its own single-use authorization | +| Pinned | Works | +| Qualified | Works: the admission is checked for every item | +| Managed | Refused: --workers and --readback-window apply to local runs, not a managed dispatch or a qualification case | + +### Multi-surface + +| Profile | Result | +| --- | --- | +| Default (gated) | Works: an API step inside it runs on the GUI (below) | +| Durable | Works: resume re-binds each surface by identity | +| Governed | Works | +| Pinned | Works | +| Qualified | Works: one environment boundary per surface | +| Managed | Works: every backend surface must be bound | + +## The API step with the other features + +| With | Result | +| --- | --- | +| Parallel workers | Works: each item runs its own API step | +| Pipelined read-back | Works: when the run isn't durable | +| Self-repair | Refused: self-repair's probes and resumed run exercise the repaired GUI step, and the person approved that step, so its API step doesn't run (the first, ordinary run can use the API step) | +| Standard changes | Refused: self-repair's probes and resumed run exercise the repaired GUI step, and the person approved that step, so its API step doesn't run (the repaired bundle's test-record run is GUI-only too) | +| Multi-surface | Refused: a run across several windows or tabs uses the GUI steps | + +## Evidence + +Each cell names the tests that show it. They run in CI. + +- API step / Default (gated): `tests/test_api_step_approval.py::test_the_person_approves_the_exact_request_and_it_is_sent_once`, `tests/test_compatibility_matrix.py::test_api_step_runs_where_the_matrix_says` +- API step / Durable: `tests/test_api_step_durable.py::test_a_durable_run_uses_the_api_step_and_checkpoints_after_it`, `tests/test_api_step_durable.py::test_an_uncertain_send_is_never_resent_by_resume` +- API step / Governed: `tests/test_api_step_governed.py::test_the_default_profile_runs_the_admitted_api_step` +- API step / Pinned: `tests/test_compatibility_matrix.py::test_api_step_runs_where_the_matrix_says` +- API step / Qualified: `tests/test_compatibility_matrix.py::test_api_step_refuses_where_the_matrix_says` +- API step / Managed: `tests/test_compatibility_matrix.py::test_api_step_refuses_where_the_matrix_says` +- Pipelined read-back / Default (gated): `tests/test_readback_pipeline.py::test_confirmed_readback_settles_the_saved_report`, `tests/test_api_step_readback_repair.py::test_an_api_save_reads_back_while_the_next_item_runs`, `tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says` +- Pipelined read-back / Durable: `tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says` +- Pipelined read-back / Governed: `tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says` +- Pipelined read-back / Pinned: `tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says` +- Pipelined read-back / Qualified: `tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says` +- Pipelined read-back / Managed: `tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says` +- Self-repair / Default (gated): `tests/test_self_repair_cli.py::test_run_self_repair_cli_resumes_after_one_phone_tap`, `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Self-repair / Durable: `tests/test_self_repair.py::test_renamed_button_is_repaired_approved_on_the_phone_and_verified` +- Self-repair / Governed: `tests/test_self_repair.py::test_renamed_button_is_repaired_approved_on_the_phone_and_verified` +- Self-repair / Pinned: `tests/test_self_repair_cli.py::test_run_self_repair_refuses_a_pinned_bundle`, `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Self-repair / Qualified: `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Self-repair / Managed: `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Standard changes / Default (gated): `tests/test_standard_change.py::test_relabelled_link_is_repaired_without_a_person` +- Standard changes / Durable: `tests/test_standard_change.py::test_relabelled_link_is_repaired_without_a_person` +- Standard changes / Governed: `tests/test_standard_change.py::test_relabelled_link_is_repaired_without_a_person` +- Standard changes / Pinned: `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Standard changes / Qualified: `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Standard changes / Managed: `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Parallel workers / Default (gated): `tests/test_cli_worklist_pool.py::test_two_workers_split_the_items_one_row_each`, `tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says` +- Parallel workers / Durable: `tests/test_compatibility_matrix.py::test_workers_run_governed_durable_api_items` +- Parallel workers / Governed: `tests/test_compatibility_matrix.py::test_workers_run_governed_durable_api_items` +- Parallel workers / Pinned: `tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says` +- Parallel workers / Qualified: `tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says` +- Parallel workers / Managed: `tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says` +- Multi-surface / Default (gated): `tests/test_multi_surface_governed.py::test_governed_durable_multi_surface_run_reaches_verified` +- Multi-surface / Durable: `tests/test_multi_surface_identity.py::test_resume_rebinds_surfaces_by_identity` +- Multi-surface / Governed: `tests/test_multi_surface_governed.py::test_governed_durable_multi_surface_run_reaches_verified` +- Multi-surface / Pinned: `tests/test_multi_surface_governed.py::test_governed_durable_multi_surface_run_reaches_verified` +- Multi-surface / Qualified: `tests/test_multi_surface_governed.py::test_case_evidence_needs_a_matching_observation_per_used_surface` +- Multi-surface / Managed: `tests/test_multi_surface_contract.py::test_managed_dispatch_needs_every_backend_surface_bound` +- API step / Parallel workers: `tests/test_compatibility_matrix.py::test_workers_run_governed_durable_api_items` +- API step / Pipelined read-back: `tests/test_api_step_readback_repair.py::test_an_api_save_reads_back_while_the_next_item_runs` +- API step / Self-repair: `tests/test_api_step_readback_repair.py::test_a_probe_or_resumed_run_never_sends_the_api_step` +- API step / Standard changes: `tests/test_api_step_readback_repair.py::test_a_standard_change_test_record_run_is_gui_only` +- API step / Multi-surface: `tests/test_compatibility_matrix.py::test_api_step_refuses_in_a_multi_surface_run` diff --git a/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md b/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md new file mode 100644 index 0000000000..4b94b33fca --- /dev/null +++ b/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md @@ -0,0 +1,149 @@ +# Classifying local edits without a person + +Status: approved by the founder on 2026-10-19, with a strict rule: the +compiler classifies a step only when the recording shows no mutating request +sent for it **and** a runtime check confirms only the expected local change. +Anything else goes to a person. + +## The problem + +A compiled program flags every special key and every unlabelled or +navigation-shaped click for review (`risk_review_required`), because its +risk can't be proven from the step alone. The run gate then refuses the +bundle until a person classifies each one. On the OpenEMR referral that is 12 +steps (blocker section A in the referral admission list): + +- 8 `Control+a` presses, each in a text field right before typing into it; +- 4 `Enter` presses, each right after typing an option's label into a + dropdown; + +plus, under `clinical-write`, the same keys fail `prohibit_vacuous_postconditions` +because a key has no outcome check (section B), and the Contact tab click is +unarmed (section C). + +These are the same few patterns in every form-filling program, on every app. +A person approving each one, per bundle, is busywork that trains people to +approve without reading. + +## The rule + +The compiler classifies five local-edit patterns itself. Every pattern needs +all of this evidence from the recording: + +- network capture (`record --capture-network`), with no request other than + `GET`, `HEAD` or `OPTIONS` between the previous event and the next one; +- URL, title and page count recorded on both sides of the action, and + unchanged; +- when the recorder snapshotted the record store around the step, the two + snapshots equal. + +A recording without network capture classifies nothing. + +| Pattern | Step evidence | Outcome check added to the step | +| --- | --- | --- | +| `select_all`: `Control+a` / `Meta+a` | The previous step clicked a text field (role `textbox`, `searchbox` or `spinbutton`, with a selector); the next step types into it | `local_edit`: focus is still on that field; URL, title and page count unchanged. The next step's typed-input read-back then checks the field's whole value. | +| `confirm_option`: `Enter` in a dropdown | The steps before clicked a dropdown (role `combobox` or `listbox`) and typed a parameter's value | `local_edit` on the dropdown, plus `field_value`: the focused dropdown shows this run's value | +| `confirm_entry`: `Enter` in a text field | The steps before clicked a text field and typed a parameter's value | `local_edit` on the field, plus `field_value`: the focused field's value equals this run's value exactly | +| `in_form_tab`: a tab click | Role `tab`, or a `button` whose only mined check is a region change; the next acting step is in the same frame | `local_edit` (URL, title and page count unchanged; focus may move) | +| `dismiss_dialog`: an informational dialog's dismiss | A `button` or `link` labelled only "Continue", "Continue to ...", "Next", "Close", "OK", "Got it", "Dismiss", "Not now", "Maybe later" or "Skip for now" | `local_edit`, plus `text_absent` of the label: the dialog is gone | + +`continue` and `next` are write words for the risk inference, so a dismiss +labelled with them starts irreversible. It is reconsidered only when those +are its only write words and its risk is still the per-step keyword +inference. A click that a later rule made irreversible (the click that +confirms a dialog, an icon-only click presumed to save) is never +reconsidered. Because a classified dismiss is reversible, effect mining no +longer attaches a placeholder `record_written` to it (#1377). + +The runtime checks fail closed: a backend that can't read focus, the field +value, the URL, the title or the page count fails the step. + +What stays the same: + +- **The risk rule.** A step whose evidence is missing or mixed (a URL change, + a write request in the window, no network capture for a submission key, a + key that isn't one of these, a field role the compiler can't read) keeps + `risk_review_required` and goes to a person, exactly as today. +- **Write-shaped context still wins.** A key or click whose retained text + names a consequential write (`submit`, `save`, `delete`, ...) classifies + irreversible before any of this runs. The one exception is a dismiss whose + only write words are `continue` or `next`, with the full evidence above. +- **Explicit `risk_overrides` still win.** +- **D, the duplicate-prevention (idempotency) key on the save, is untouched.** + That is a business decision. + +What the run gate sees: the classified step is reversible, reviewed by the +compiler with recorded evidence (`risk_explanation` names the pattern and the +evidence), and has an outcome check, so `require_reviewed_action_risk`, +`prohibit_vacuous_postconditions` and `max_unverified_steps` pass for it. A +tab click classified this way is in-form navigation that writes nothing, so it +isn't consequential and isn't entity navigation; `prohibit_unarmed_clicks` +under `clinical-write` still applies to it, so it still needs an identity +check (decision 2 below). + +## Why this is safe enough + +- The classification is backed by what the demonstration did, not by the + label. A step that sent a write request in the recording, changed the URL, + title or page count, or changed the record store is never classified. +- The outcome check runs every time. If the same key does something else at + run time (the app changed, a different field had focus), the step fails its + postcondition and the run halts at that step. The halt comes after the key + was pressed, which is the residual risk: a key that, at run time only, + commits something. For these patterns that requires the app to have + changed the field's behavior since the recording, and the first write it + makes is still caught by the save's record check or by the next step's + checks. +- Evidence is sealed in the bundle (the step's explanation and postconditions + are part of the content digest), so a later edit to a classification + changes the digest and needs a new admission. + +## Decisions + +1. The residual risk above is accepted for these patterns (founder, + 2026-10-19). +2. `clinical-write`'s `prohibit_unarmed_clicks` still applies to a classified + tab click. On the OpenEMR referral, the Contact tab (`step_015`) still needs + an identity check or a person. +3. Network capture is required for every pattern, `Control+a` included. + +## Implementation + +- `openadapt_flow.compiler.local_edit.local_edits(steps, events, network)` + returns the classified steps with their explanation and checks. +- `compile_recording` applies them after the per-step risk inference and the + confirmation-dialog pairing, and before `risk_overrides`, which still win. + Effect mining runs after both. +- `openadapt_flow.risk.is_local_edit` and `is_sealed_dismiss` let the run + gate's risk re-inference read a classified step from what the bundle seals: + the explanation and its runtime checks. +- Postcondition kinds `local_edit` and `field_value`, evaluated by the + replayer (`_local_edit_held`, `_field_value_shown`). +- Tests: `tests/test_local_edit_classification.py`. Each pattern has a + positive case and a case where a write request was sent, which must + refuse. + +## OpenEMR referral, re-admitted on this branch + +Run gate on the recorded referral with the API step admitted +(`--identity subject_name=lname`), compared with `main`: + +| Policy | Violations on `main` | With local edits | +| --- | --- | --- | +| `permissive` | 12 | 1 | +| `clinical-write` | 32 | 9 | + +What remains: + +- `step_007` (`Control+a`): OpenEMR's background + `POST /library/ajax/execute_background_services.php` landed in its window, + so the strict rule leaves it for a person. Under `clinical-write` it also + has no outcome check. +- `step_015` (the Contact tab): classified, but unarmed under + `clinical-write`'s `prohibit_unarmed_clicks`. +- `step_039` (Create New Patient): unarmed, no identity contract and no + record check on its GUI path. +- `step_039` and `step_040`: no duplicate-prevention (idempotency) key. That + stays a person's choice. +- Encryption at rest, under the default and regulated profiles: seal the + bundle. diff --git a/packages/flow/openadapt_flow/__main__.py b/packages/flow/openadapt_flow/__main__.py index 532b910e83..c4c9b72692 100644 --- a/packages/flow/openadapt_flow/__main__.py +++ b/packages/flow/openadapt_flow/__main__.py @@ -3603,12 +3603,14 @@ def _cmd_run(args: argparse.Namespace) -> int: # Parallel worklist (docs/THROUGHPUT.md): every item gets its own # single-use authorization, bound to that item's one-row worklist and # otherwise identical to the run's (same gate, policy, approvals). - if dispatch_file or qualification_case is not None: - print( - "run REFUSED: --workers and --readback-window apply to local " - "runs, not a managed dispatch or a qualification case. Nothing " - "was executed." - ) + from openadapt_flow.cli_worklist_pool import pool_profile_refusal + + pool_refused = pool_profile_refusal( + managed=bool(dispatch_file), + qualification_case=qualification_case is not None, + ) + if pool_refused is not None: + print(f"run REFUSED: {pool_refused}. Nothing was executed.") return 2 try: relation = loop_relation(workflow) diff --git a/packages/flow/openadapt_flow/backends/playwright_backend.py b/packages/flow/openadapt_flow/backends/playwright_backend.py index d30d4598f5..d68b05d853 100644 --- a/packages/flow/openadapt_flow/backends/playwright_backend.py +++ b/packages/flow/openadapt_flow/backends/playwright_backend.py @@ -1665,6 +1665,28 @@ def focused_text_value(self) -> Optional[str]: return self._focused_element_eval(_EDITABLE_VALUE_JS) + def focused_element_matches(self, selector: str) -> Optional[bool]: + """Whether the focused element (through frames) matches ``selector``. + + None when it can't be read. Used by the ``local_edit`` postcondition. + """ + + result = self._focused_element_call( + "(el, sel) => { try { return el.matches(sel); } catch (e) { return null; } }", + arg=selector, + with_arg=True, + ) + return result if isinstance(result, bool) else None + + def focused_select_label(self) -> Optional[str]: + """The focused `` + + + + +""" + + +def _controls(): + return form_controls(NATIVE, "/save.php") + + +@pytest.mark.parametrize( + "name, value, problem", + [ + ("code", "", "required"), + ("code", "abc-12", "pattern"), + ("code", "ABC-123", "maxlength"), + ("email", "not-an-address", "email"), + ("dose", "11", "max"), + ("dose", "0", "min"), + ("dose", "ten", "number"), + ("seen", "2026-10-20", "max"), + ("seen", "10/19/2026", "date"), + ], +) +def test_native_constraints_refuse_what_the_browser_would(name, value, problem): + found = native_problem(name, _controls()[name], value) + assert found is not None and problem in found + + +@pytest.mark.parametrize( + "name, value", + [ + ("code", "ABC-12"), + ("email", "a@b.example"), + ("dose", "5"), + ("seen", "2026-10-01"), + ("h", ""), + ], +) +def test_native_constraints_accept_valid_values(name, value): + assert native_problem(name, _controls()[name], value) is None + + +def test_validate_js_length_format_and_numericality_are_enforced(): + rule = {"length": {"maximum": 3}, "format": {"pattern": "[a-z]+"}} + assert "length" in client_rule_problem("f", rule, "abcd") + assert "format" in client_rule_problem("f", rule, "AB") + assert client_rule_problem("f", {"numericality": True}, "x") is not None + assert client_rule_problem("f", rule, "ab") is None + + +def test_the_actuator_refuses_a_value_the_browser_would_refuse(tmp_path): + spec = WebFormSpec( + form_path="/new.php", + fields=[WebFormField(name="code", source="param", param="code")], + controls=["code", "dose", "email", "h", "seen"], + replaces_steps=["s0"], + folded_checks_digest="sha256:" + "0" * 64, + ) + body, problem = WebFormActuator.build_body(spec, {"code": "abc"}, _controls()) + assert body == [] and "pattern" in problem + + +def _rec_with_form(tmp_path, html): + rec = _recording(tmp_path) + rows = [json.loads(x) for x in (rec / nc.NETWORK_FILE).read_text().splitlines()] + for row in rows[1:]: + if "document.forms[0].addEventListener('submit', v)" + ), + "listening for submit", + ), + ( + lambda h: h.replace( + '{"form_lname":{"presence":true}}', + '{"form_lname":{"presence":true,"pastDate":{}}}', + ), + "form_lname pastDate", + ), + ], +) +def test_script_the_api_step_cant_run_keeps_the_save_on_the_gui(tmp_path, edit, named): + html = edit(FORM_HTML) + _wf, step, reasons = propose_api_step(_workflow(), _rec_with_form(tmp_path, html)) + assert step is None and named in " ".join(reasons) + # The same script on the live form makes an admitted API step unavailable. + live = edit(FORM_HTML.replace("", "live-session-value")) + (tmp_path / "admitted").mkdir() + out = WebFormActuator(FakeTransport(form=live)).actuate( + _admitted_binding(tmp_path / "admitted"), + {"lname": "Marsh", "state_label": "California", "note": "Referred"}, + ) + assert out.status is ActuationStatus.UNAVAILABLE + # A rule kind the recording didn't have is first a changed rule set. + assert named in out.reason or "rules changed" in out.reason diff --git a/packages/flow/tests/test_api_step_replaced_steps_digest.py b/packages/flow/tests/test_api_step_replaced_steps_digest.py new file mode 100644 index 0000000000..b80ac338e7 --- /dev/null +++ b/packages/flow/tests/test_api_step_replaced_steps_digest.py @@ -0,0 +1,70 @@ +"""An API step stands for the exact steps it was admitted for. + +The reviewer's case for #1269: folded_checks_digest hashed only the folded +steps' on-screen checks, so changing what a replaced step does (its action, +its value, its parameter, its target or identity, a guard, the save's record +check) left the admitted API step running in its place. +""" + +from __future__ import annotations + +import pytest + +from openadapt_flow.ir import ActionKind, Anchor, Guard, Predicate, PredicateKind +from openadapt_flow.runtime.replayer import Replayer +from tests.test_api_tier_web_form import _span_workflow +from tests.test_replayer import FakeBackend, FakeVision + + +def _edit(wf, index, **update): + steps = list(wf.steps) + steps[index] = steps[index].model_copy(update=update) + return wf.model_copy(update={"steps": steps}) + + +def _why(wf): + replayer = Replayer(FakeBackend(), vision=FakeVision(), web_form_actuator=object()) + _end, _save, why = replayer._web_form_span_at(wf, 0, durable=False) + return why + + +def test_the_admitted_steps_run_as_the_api_step(): + assert _why(_span_workflow()) is None + + +@pytest.mark.parametrize( + "index, update", + [ + (0, {"action": ActionKind.CLICK, "key": None}), + (0, {"key": "Shift+Tab"}), + (0, {"text": "{lname}", "action": ActionKind.TYPE, "key": None}), + (0, {"param": "lname"}), + ( + 0, + { + "anchor": Anchor( + template="templates/x.png", + region=(0, 0, 10, 10), + click_point=(5, 5), + context_text="Another record", + ) + }, + ), + ( + 0, + { + "guard": Guard( + predicate=Predicate( + kind=PredicateKind.PARAM_EQUALS, param="lname", value="x" + ), + on_unmet="halt", + ) + }, + ), + (1, {"key": "Ctrl+Enter"}), + (1, {"risk": "irreversible"}), + ], +) +def test_any_change_to_a_replaced_step_keeps_it_on_the_gui(index, update): + why = _why(_edit(_span_workflow(), index, **update)) + assert why is not None and "replaces changed" in why diff --git a/packages/flow/tests/test_api_tier_web_form.py b/packages/flow/tests/test_api_tier_web_form.py index c632a4c5b3..c45f0f6643 100644 --- a/packages/flow/tests/test_api_tier_web_form.py +++ b/packages/flow/tests/test_api_tier_web_form.py @@ -561,7 +561,7 @@ def _span_workflow(*, admitted=True) -> Workflow: WebFormPrecheck(path="/dup.php", require_text=["No matches were found."]) ], replaces_steps=["step_000", "step_001"], - folded_checks_digest=folded_checks_digest(steps[:1]), + folded_checks_digest=folded_checks_digest(steps), ) binding = ApiBinding( kind="web_form", url_template="/save.php", effects=[effect], web_form=spec @@ -666,7 +666,7 @@ def test_a_rejected_save_halts_without_gui_retry(tmp_path): } ), {}, - "on-screen check", + "replaces changed", ), ], ) diff --git a/packages/flow/tests/test_compatibility_matrix.py b/packages/flow/tests/test_compatibility_matrix.py new file mode 100644 index 0000000000..491ae261b9 --- /dev/null +++ b/packages/flow/tests/test_compatibility_matrix.py @@ -0,0 +1,386 @@ +"""The compatibility matrix (openadapt_flow/compatibility.py, docs/COMPATIBILITY.md). + +Every feature against every run profile either works or is refused with a +named reason. These tests check each cell against the runtime: + +- a cell that works must engage the feature (the API step sends its request, + the read-back runs on the side thread); a fallback fails the test; +- a refused cell must give exactly the reason the matrix names; +- every cell names tests that exist, and the published page is current. + +The record system is a scripted verifier or the MockMed fault server; the +values are made up. +""" + +from __future__ import annotations + +import argparse +import ast +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Optional + +import pytest + +from openadapt_flow import compatibility as matrix +from openadapt_flow.approval_mode.gate import ApprovalGate +from openadapt_flow.approval_mode.item_approval import ItemApprovalStore +from openadapt_flow.deployment import DeploymentConfig, PolicySection +from openadapt_flow.execution_profiles import ExecutionProfile +from openadapt_flow.run_gate import build_runtime_authorization, evaluate_run_gate +from openadapt_flow.runtime.actuators.web_form import WebFormActuator +from openadapt_flow.runtime.effects.rest import RestRecordVerifier +from openadapt_flow.runtime.readback_pipeline import ReadbackPipeline +from tests.test_api_step_approval import APPROVE, Person +from tests.test_api_step_governed import PARAMS, Transport, _workflow +from tests.test_approval_mode_replayer import RecordSystem +from tests.test_execution_profiles import ( + _KEY, + Replayer, + _gate, + _ReadyVision, + _sealed, +) +from tests.test_replayer import FakeBackend + +FLOW = Path(__file__).resolve().parents[1] +REPO = FLOW.parents[1] + + +class RestRecordSystem(RecordSystem, RestRecordVerifier): + """The scripted record system, read the way a REST read is. + + The read-back pipeline defers only network reads of the system of record + (REST, FHIR, GraphQL, SQL); this is the scripted system as one of them. + """ + + +class Run: + """One item of the API-step program under one run profile.""" + + def __init__( + self, + tmp_path: Path, + profile: str, + *, + pipeline: Optional[ReadbackPipeline] = None, + ) -> None: + self.profile = profile + self.durable = profile in {"durable", "governed", "qualified"} + regulated = profile in {"governed", "qualified"} + self.workflow, self.bundle = _sealed(tmp_path, _workflow(), encrypted=True) + self.verifier = RestRecordSystem() + if regulated: + report = _gate( + self.workflow, + self.bundle, + ExecutionProfile.REGULATED, + verifier=self.verifier, + durable=True, + ) + else: + report = evaluate_run_gate( + self.workflow, + bundle_dir=self.bundle, + deployment=DeploymentConfig(policy=PolicySection(policy="permissive")), + effect_verifier=self.verifier, + effective_durable=self.durable, + ) + assert report.passed, report.render() + self.authorization = build_runtime_authorization( + self.workflow, report, params=PARAMS + ) + if profile == "qualified": + # The fields a signed production admission binds into the run. + self.authorization = self.authorization.model_copy( + update={ + "production_qualification_admission_id": "admission-1", + "production_qualification_admission_sha256": "1" * 64, + "production_qualification_evidence_identity_sha256": "2" * 64, + "production_qualification_runtime_validation_id": "validation-1", + "production_qualification_signer_registry_sha256": "3" * 64, + "production_qualification_signer_registry_revision": 1, + "production_qualification_signer_registry_expires_at": ( + "2099-01-01T00:00:00+00:00" + ), + "production_qualification_authority_sha256": "4" * 64, + } + ) + self.run_dir = tmp_path / "run" + self.person = Person([APPROVE]) + self.store = ItemApprovalStore(self.run_dir) + self.person.store = self.store + self.transport = Transport() + self.pipeline = pipeline + + def replayer(self) -> Any: + gate = ApprovalGate( + "each_item", + approval_store=self.store, + approval_wait_s=30.0, + approval_poll_s=1.0, + sleep=self.person.sleep, + monotonic=self.person.monotonic, + ) + return Replayer( + FakeBackend(), + vision=_ReadyVision(), + effect_verifier=self.verifier, + governed_authorization=self.authorization, + durable=self.durable, + checkpoint_key=_KEY, + require_settled=True, + approval_gate=gate, + web_form_actuator=WebFormActuator(self.transport), + readback_pipeline=self.pipeline, + ) + + def run(self) -> Any: + return self.replayer().run( + self.workflow, + params=PARAMS, + bundle_dir=self.bundle, + run_dir=self.run_dir, + ) + + +def _cells(feature: str, status: str) -> list[str]: + return [ + c.profile for c in matrix.CELLS if c.feature == feature and c.status == status + ] + + +# -- the matrix is complete and backed by tests ------------------------------------- + + +def test_every_feature_has_exactly_one_cell_per_profile(): + pairs = [(c.feature, c.profile) for c in matrix.CELLS] + assert len(pairs) == len(set(pairs)) + assert set(pairs) == { + (f.key, p.key) for f in matrix.FEATURES for p in matrix.PROFILES + } + features = {f.key for f in matrix.FEATURES} + for item in matrix.COMBINATIONS: + assert item.feature in features and item.profile in features + + +@pytest.mark.parametrize( + "item", + [*matrix.CELLS, *matrix.COMBINATIONS], + ids=lambda c: f"{c.feature}-{c.profile}", +) +def test_every_cell_names_a_reason_or_works_and_has_evidence(item): + if item.status == "refused": + assert item.reason, "a refused cell names its reason" + else: + assert item.reason is None + assert item.evidence, "every cell names the tests that show it" + for node in item.evidence: + path, _, name = node.partition("::") + source = FLOW / path + assert source.is_file(), node + names = { + n.name + for n in ast.walk(ast.parse(source.read_text())) + if isinstance(n, ast.FunctionDef) + } + assert name in names, node + + +def test_the_published_page_is_current_and_linked(): + page = FLOW / "docs" / "COMPATIBILITY.md" + assert page.read_text(encoding="utf-8") == matrix.render_markdown(), ( + "run python scripts/build_compatibility_doc.py" + ) + assert ( + "packages/flow/docs/COMPATIBILITY.md" in (REPO / "CAPABILITIES.md").read_text() + ) + + +# -- the API step --------------------------------------------------------------------- + + +@pytest.mark.parametrize("profile", _cells("api_step", "works")) +def test_api_step_runs_where_the_matrix_says(tmp_path, profile): + run = Run(tmp_path, profile) + report = run.run() + + assert [r.actuation for r in report.results] == ["api_folded", "api"], [ + r.api_fallback for r in report.results + ] + assert all(r.api_fallback is None for r in report.results) + assert [path for path, _ in run.transport.posts] == ["/save.php"] + assert report.success is True + (request,) = run.store.requests() + assert request.form_state_kind == "api_request" + if profile == "governed": + assert report.execution_outcome == "VERIFIED" + + +@pytest.mark.parametrize("profile", _cells("api_step", "refused")) +def test_api_step_refuses_where_the_matrix_says(tmp_path, profile): + reason = matrix.cell("api_step", profile).reason + run = Run(tmp_path, "qualified" if profile == "qualified" else "default") + if profile == "managed": + replayer = run.replayer() + replayer.managed_dispatch_binding = object() + _end, _save, why = replayer._web_form_span_at(run.workflow, 0, durable=True) + assert why == reason + return + report = run.run() + assert run.transport.posts == [] + assert report.results[0].api_fallback == reason + + +def test_api_step_refuses_in_a_multi_surface_run(tmp_path): + run = Run(tmp_path, "default") + replayer = run.replayer() + replayer._surface_router = object() + _end, _save, why = replayer._web_form_span_at(run.workflow, 0, durable=False) + assert why == matrix.API_MULTI_SURFACE + + +def test_the_repair_reason_is_the_runtimes(): + from openadapt_flow.repair.self_repair import REPAIR_GUI_ONLY + + assert matrix.API_REPAIR_PROBE == REPAIR_GUI_ONLY + + +# -- pipelined read-back --------------------------------------------------------------- + + +@pytest.mark.parametrize( + "profile", [p.key for p in matrix.PROFILES if p.key != "managed"] +) +def test_pipelined_readback_where_the_matrix_says(tmp_path, profile): + item = matrix.cell("pipelined_readback", profile) + pipeline = ReadbackPipeline() + try: + run = Run(tmp_path, profile, pipeline=pipeline) + report = run.run() + handoff = pipeline.take() + settled = handoff.settled.result(timeout=30) if handoff is not None else None + finally: + pipeline.close() + + assert report.results[-1].effect_verified is True or settled is not None + if item.status == "works": + assert settled is report, pipeline.inline_reason + assert report.success is True + assert report.results[-1].effect_verified is True + else: + assert settled is None + assert pipeline.inline_reason == item.reason + + +# -- the worklist pool ------------------------------------------------------------------- + + +@pytest.mark.parametrize("profile", [p.key for p in matrix.PROFILES]) +def test_the_worklist_pool_where_the_matrix_says(tmp_path, profile): + from openadapt_flow.cli_worklist_pool import pool_profile_refusal + from openadapt_flow.runtime.worklist_pool import ( + FunctionWorker, + PoolPolicy, + run_worklist_pool, + ) + + refusal = pool_profile_refusal( + managed=profile == "managed", qualification_case=False + ) + workers = matrix.cell("parallel_workers", profile) + if workers.status == "refused": + assert refusal == workers.reason + assert matrix.cell("pipelined_readback", profile).reason == refusal + return + assert refusal is None + ran: list[int] = [] + pool = [ + FunctionWorker(f"w{i}", lambda idx, row, d: ran.append(idx) or _Done()) + for i in range(2) + ] + report = run_worklist_pool( + [{"n": str(i)} for i in range(4)], + pool, + run_dir=tmp_path, + policy=PoolPolicy(workers=2, items_independent=True), + ) + assert sorted(ran) == [0, 1, 2, 3] and report.success + + +class _Done: + success = True + execution_outcome = "VERIFIED" + transaction_outcome = "VERIFIED" + halt = None + total_ms = 1.0 + + +def test_workers_run_governed_durable_api_items(tmp_path): + """Two workers, each item a Regulated, durable, gated API-step run.""" + + from openadapt_flow.runtime.worklist_pool import ( + VERIFIED, + FunctionWorker, + PoolPolicy, + run_worklist_pool, + ) + + runs: dict[int, Run] = {} + + def item(idx: int, row: dict[str, Any], item_dir: Path) -> Any: + runs[idx] = Run(item_dir, "governed") + return runs[idx].run() + + report = run_worklist_pool( + [{"n": str(i)} for i in range(4)], + [FunctionWorker(f"w{i}", item) for i in range(2)], + run_dir=tmp_path, + policy=PoolPolicy(workers=2, items_independent=True), + ) + assert {i["state"] for i in report.items} == {VERIFIED}, [ + i.get("detail") or i for i in report.items + ] + for run in runs.values(): + assert [path for path, _ in run.transport.posts] == ["/save.php"] + + +# -- self-repair and standard changes ---------------------------------------------------- + + +def _repair_args(profile: str) -> argparse.Namespace: + return argparse.Namespace( + managed_dispatch_file="dispatch.json" if profile == "managed" else None, + qualification_authority_file=( + "authority.json" if profile == "qualified" else None + ), + _qualification_case_execution=None, + pin_digest="a" * 64 if profile == "pinned" else None, + hold_on_halt=False, + save_healed_to=None, + record_video=False, + repair_agent="mod:factory", + repair_screen_access_by="reviewer", + durable=profile == "durable", + ) + + +@pytest.mark.parametrize("feature", ["self_repair", "standard_changes"]) +@pytest.mark.parametrize("profile", [p.key for p in matrix.PROFILES]) +def test_self_repair_where_the_matrix_says(feature, profile): + from openadapt_flow.cli_self_repair import _refusal + + backend = SimpleNamespace( + kind="web", browser_cdp_endpoint=None, browser_page_url=None + ) + refusal = _refusal(_repair_args(profile), backend, "http://app") + item = matrix.cell(feature, profile) + if item.status == "refused": + assert refusal == item.reason + elif profile == "governed": + # The CLI's Standard profile needs a signed admission (the + # "qualified" column); the Standard contract runs through the + # Python API, which these cells' evidence tests exercise. + assert item.evidence + else: + assert refusal is None diff --git a/packages/flow/tests/test_local_edit_classification.py b/packages/flow/tests/test_local_edit_classification.py new file mode 100644 index 0000000000..755e63e2cd --- /dev/null +++ b/packages/flow/tests/test_local_edit_classification.py @@ -0,0 +1,546 @@ +"""Local edits classified from the recording (docs/design/LOCAL_EDIT_CLASSIFICATION.md). + +Each pattern classifies only with all of its evidence; each missing piece +leaves the step for a person; each outcome check fails when the action did +anything else at run time. Synthetic events only. +""" + +from __future__ import annotations + +from typing import Any, Optional + +import pytest + +from openadapt_flow.compiler.local_edit import local_edits +from openadapt_flow.ir import ( + ActionKind, + Anchor, + Postcondition, + PostconditionKind, + Step, + StructuralLocator, + Workflow, +) +from openadapt_flow.policy import step_tags +from openadapt_flow.risk import infer_step_risk, is_local_edit + +FRAME = ["div > iframe"] +URL = "http://app/form" + + +def _click(i: int, selector: Optional[str], role: str, name: str = "Field") -> Step: + return Step( + id=f"step_{i:03d}", + intent=f"click {name}", + action=ActionKind.CLICK, + anchor=Anchor( + template=f"templates/{i}.png", + region=(0, 0, 10, 10), + click_point=(5, 5), + structural=StructuralLocator( + selector=selector, frame_path=list(FRAME), role=role, name=name + ), + ), + risk_review_required=False, + ) + + +def _key(i: int, key: str) -> Step: + return Step( + id=f"step_{i:03d}", + intent=f"press {key}", + action=ActionKind.KEY, + key=key, + risk_review_required=True, + risk_explanation="special key may navigate, edit, or invoke an application shortcut", + ) + + +def _type(i: int, param: str) -> Step: + return Step( + id=f"step_{i:03d}", + intent="type a value", + action=ActionKind.TYPE, + text="{" + param + "}", + ) + + +def _event(i: int, **changes: Any) -> dict[str, Any]: + event = { + "i": i, + "t": float(i), + "url_before": URL, + "url_after": URL, + "title_before": "Form", + "title_after": "Form", + "pages_before": 1, + "pages_after": 1, + } + event.update(changes) + return event + + +def _events(n: int, **at: dict[str, Any]) -> list[dict[str, Any]]: + return [_event(i, **at.get(f"e{i}", {})) for i in range(n)] + + +def _ids(edits) -> dict[str, str]: + return {e.step_id: e.pattern for e in edits} + + +# -- select-all before typing --------------------------------------------------------- + + +def test_select_all_in_a_text_field_before_typing_is_a_local_edit(): + steps = [_click(0, "#lname", "textbox"), _key(1, "Control+a"), _type(2, "lname")] + (edit,) = local_edits(steps, _events(3), network=[]) + assert edit.step_id == "step_001" and edit.pattern == "select_all" + (check,) = edit.postconditions + assert check.kind is PostconditionKind.LOCAL_EDIT + assert check.field_selector == "#lname" + + +@pytest.mark.parametrize( + "steps, events, network, why", + [ + ( + [_click(0, "#lname", "button"), _key(1, "Control+a"), _type(2, "lname")], + _events(3), + [], + "not a text field", + ), + ( + [ + _click(0, "#lname", "textbox"), + _key(1, "Control+a"), + _click(2, "#x", "textbox"), + ], + _events(3), + [], + "nothing is typed next", + ), + ( + [_click(0, "#lname", "textbox"), _key(1, "Control+a"), _type(2, "lname")], + _events(3, e1={"url_after": URL + "?next"}), + [], + "the URL changed", + ), + ( + [_click(0, "#lname", "textbox"), _key(1, "Control+a"), _type(2, "lname")], + _events(3), + [{"method": "POST", "t": 1.2, "url": URL}], + "a write request in the window", + ), + ( + [_click(0, None, "textbox"), _key(1, "Control+a"), _type(2, "lname")], + _events(3), + [], + "no field selector to keep focus in", + ), + ], +) +def test_select_all_without_its_evidence_stays_with_a_person( + steps, events, network, why +): + assert local_edits(steps, events, network=network) == [], why + + +def test_select_all_needs_network_capture(): + steps = [_click(0, "#lname", "textbox"), _key(1, "Control+a"), _type(2, "lname")] + assert local_edits(steps, _events(3), network=None) == [] + + +def test_select_all_with_a_write_request_sent_stays_with_a_person(): + steps = [_click(0, "#lname", "textbox"), _key(1, "Control+a"), _type(2, "lname")] + sent = [{"method": "PUT", "t": 1.5, "url": URL + "/draft"}] + assert local_edits(steps, _events(3), network=sent) == [] + + +def test_a_changed_record_store_stays_with_a_person(): + steps = [_click(0, "#lname", "textbox"), _key(1, "Control+a"), _type(2, "lname")] + events = _events(3, e1={"sor_before": {"n": 1}, "sor_after": {"n": 2}}) + assert local_edits(steps, events, network=[]) == [] + same = _events(3, e1={"sor_before": {"n": 1}, "sor_after": {"n": 1}}) + assert _ids(local_edits(steps, same, network=[])) == {"step_001": "select_all"} + + +# -- Enter on a typed dropdown option --------------------------------------------------- + + +def _dropdown(network: Optional[list] = None, **changes): + steps = [_click(0, "#state", "combobox"), _type(1, "state"), _key(2, "Enter")] + events = _events(3, e1={"param": "state"}, **changes) + return local_edits(steps, events, network=[] if network is None else network) + + +def test_enter_after_typing_a_dropdown_option_is_a_local_edit(): + (edit,) = _dropdown() + assert edit.step_id == "step_002" and edit.pattern == "confirm_option" + kinds = [p.kind for p in edit.postconditions] + assert kinds == [PostconditionKind.LOCAL_EDIT, PostconditionKind.FIELD_VALUE] + assert edit.postconditions[1].param == "state" + + +def test_enter_needs_network_capture_and_no_write(): + steps = [_click(0, "#state", "combobox"), _type(1, "state"), _key(2, "Enter")] + events = _events(3, e1={"param": "state"}) + assert local_edits(steps, events, network=None) == [] + assert _dropdown(network=[{"method": "POST", "t": 2.5, "url": URL}]) == [] + assert _dropdown(e2={"title_after": "Saved"}) == [] + + +def _entry(network: Optional[list] = None, **changes): + steps = [_click(0, "#q", "textbox"), _type(1, "q"), _key(2, "Enter")] + events = _events(3, e1={"param": "q"}, **changes) + return local_edits(steps, events, network=[] if network is None else network) + + +def test_enter_after_typing_into_a_text_field_is_a_local_edit(): + (edit,) = _entry() + assert edit.step_id == "step_002" and edit.pattern == "confirm_entry" + kinds = [p.kind for p in edit.postconditions] + assert kinds == [PostconditionKind.LOCAL_EDIT, PostconditionKind.FIELD_VALUE] + assert edit.postconditions[0].field_selector == "#q" + assert edit.postconditions[1].param == "q" + + +def test_enter_in_a_text_field_with_a_write_request_sent_stays_with_a_person(): + assert _entry(network=[{"method": "POST", "t": 2.4, "url": URL}]) == [] + steps = [_click(0, "#q", "textbox"), _type(1, "q"), _key(2, "Enter")] + assert local_edits(steps, _events(3, e1={"param": "q"}), network=None) == [] + + +def test_enter_after_a_literal_or_nothing_typed_stays_with_a_person(): + steps = [_click(0, "#q", "textbox"), _key(1, "Enter")] + assert local_edits(steps, _events(2), network=[]) == [] + + +# -- in-form tabs ------------------------------------------------------------------------- + + +def test_a_tab_of_the_same_form_is_a_local_edit(): + steps = [_click(0, None, "button", "Contact"), _click(1, "#street", "textbox")] + (edit,) = local_edits(steps, _events(2), network=[]) + assert edit.pattern == "in_form_tab" + assert [p.kind for p in edit.postconditions] == [PostconditionKind.LOCAL_EDIT] + + +def test_a_tab_click_needs_capture_the_same_frame_and_no_write(): + steps = [_click(0, None, "tab", "Contact"), _click(1, "#street", "textbox")] + assert local_edits(steps, _events(2), network=None) == [] + assert local_edits(steps, _events(2), network=[{"method": "POST", "t": 0.5}]) == [] + other = _click(1, "#street", "textbox") + other.anchor.structural.frame_path = ["div > iframe:nth-of-type(2)"] + assert local_edits([steps[0], other], _events(2), network=[]) == [] + + +def test_a_tab_role_click_in_the_same_frame_is_a_local_edit(): + steps = [ + _click(0, "#tab-contact", "tab", "Contact"), + _click(1, "#street", "textbox"), + ] + (edit,) = local_edits(steps, _events(2), network=[]) + assert edit.pattern == "in_form_tab" + + +def test_a_tab_click_with_a_write_request_sent_stays_with_a_person(): + steps = [ + _click(0, "#tab-contact", "tab", "Contact"), + _click(1, "#street", "textbox"), + ] + sent = [{"method": "PATCH", "t": 0.8, "url": URL + "/autosave"}] + assert local_edits(steps, _events(2), network=sent) == [] + + +def test_a_write_shaped_click_is_never_a_local_edit(): + save = _click(0, None, "button", "Save") + save.risk = "irreversible" + steps = [save, _click(1, "#street", "textbox")] + assert local_edits(steps, _events(2), network=[]) == [] + + +# -- the dismiss of an informational dialog (#1377) ------------------------------------------ + + +def _infer(step: Step) -> Step: + inference = infer_step_risk(step) + step.risk = inference.risk + step.risk_explanation = inference.explanation + step.risk_review_required = inference.requires_review + return step + + +def _whats_new() -> list[Step]: + # MockMed's What's New notice: "Continue to tasks" closes it, the page stays. + dismiss = _infer(_click(0, "#notice-continue", "button", "Continue to tasks")) + assert dismiss.risk == "irreversible" # "continue" is a write word + return [dismiss, _click(1, "#task-search", "textbox", "Search tasks")] + + +def test_dismissing_an_informational_dialog_is_a_local_edit(): + (edit,) = local_edits(_whats_new(), _events(2), network=[]) + assert edit.step_id == "step_000" and edit.pattern == "dismiss_dialog" + local, gone = edit.postconditions + assert local.kind is PostconditionKind.LOCAL_EDIT and local.field_selector is None + assert gone.kind is PostconditionKind.TEXT_ABSENT + assert gone.text == "Continue to tasks" + + +def test_a_dismiss_with_a_write_request_sent_stays_with_a_person(): + sent = [{"method": "POST", "t": 0.4, "url": URL + "/notices/seen"}] + assert local_edits(_whats_new(), _events(2), network=sent) == [] + assert local_edits(_whats_new(), _events(2), network=None) == [] + moved = _events(2, e0={"url_after": URL + "/tasks"}) + assert local_edits(_whats_new(), moved, network=[]) == [] + + +@pytest.mark.parametrize( + "name", ["Continue to checkout", "Continue and save", "Next: submit", "Save"] +) +def test_a_dismiss_shaped_label_with_another_write_word_stays_irreversible(name): + steps = [_infer(_click(0, "#b", "button", name)), _click(1, "#x", "textbox")] + assert steps[0].risk == "irreversible" + assert local_edits(steps, _events(2), network=[]) == [] + + +def test_the_click_that_confirms_a_dialog_is_never_reconsidered(): + ok = _infer(_click(0, "#ok", "button", "OK")) + ok.risk = "irreversible" + ok.risk_explanation = "confirms the dialog opened by step_000; this click saves" + steps = [ok, _click(1, "#x", "textbox")] + assert local_edits(steps, _events(2), network=[]) == [] + + +def test_the_risk_reinference_reads_a_sealed_dismiss(): + steps = _whats_new() + (edit,) = local_edits(steps, _events(2), network=[]) + dismiss = steps[0] + dismiss.risk_explanation = edit.explanation + dismiss.expect = list(edit.postconditions) + inference = infer_step_risk(dismiss) + assert inference.risk == "reversible" and not inference.requires_review + # Without the label-gone check the write word wins again. + dismiss.expect = [edit.postconditions[0]] + assert infer_step_risk(dismiss).risk == "irreversible" + + +# -- risk, tags and the outcome checks ------------------------------------------------------ + + +def _classified_key() -> Step: + step = _key(1, "Control+a") + step.risk_review_required = False + step.risk_explanation = "local edit: selects the text of the field" + step.expect = [ + Postcondition(kind=PostconditionKind.LOCAL_EDIT, field_selector="#a") + ] + return step + + +def test_the_risk_reinference_reads_a_sealed_local_edit(): + step = _classified_key() + assert is_local_edit(step) + inference = infer_step_risk(step) + assert inference.risk == "reversible" and not inference.requires_review + # Without its runtime check the explanation alone proves nothing. + step.expect = [] + assert infer_step_risk(step).requires_review + + +def test_a_local_edit_tab_is_not_entity_navigation(): + tab = _click(0, None, "button", "Contact") + tab.anchor.ocr_text = "Contact" + wf = Workflow(name="w", steps=[tab]) + assert "entity_navigation" in step_tags( + tab, wf, require_current_certification=False + ) + tab.risk_explanation = "local edit: switches the panel of the same form" + tab.expect = [Postcondition(kind=PostconditionKind.LOCAL_EDIT)] + assert "entity_navigation" not in step_tags( + tab, wf, require_current_certification=False + ) + + +def test_unset_local_edit_fields_serialize_as_before(): + pc = Postcondition(kind=PostconditionKind.TEXT_PRESENT, text="Saved") + assert set(pc.model_dump()) == { + "kind", + "text", + "region", + "phash", + "phash_tolerance", + "timeout_s", + "template", + } + + +class _Backend: + def __init__( + self, *, url=URL, focused=True, label="Massachusetts", text=None + ) -> None: + self.url, self.page_title, self.page_count = url, "Form", 1 + self._focused, self._label, self._text = focused, label, text + + def focused_element_matches(self, selector): + return self._focused + + def focused_select_label(self): + return self._label + + def focused_text_value(self): + return self._text + + +def _replayer(backend): + from openadapt_flow.runtime.replayer import Replayer + from tests.test_replayer import FakeVision + + replayer = Replayer(backend, vision=FakeVision()) + replayer._current_params = {"state": "Massachusetts"} + return replayer + + +START = {"url": URL, "page_title": "Form", "page_count": 1} + + +@pytest.mark.parametrize( + "backend, passes", + [ + (_Backend(), True), + (_Backend(url=URL + "?saved"), False), + (_Backend(focused=False), False), + (_Backend(focused=None), False), + ], +) +def test_the_local_edit_check_fails_when_the_action_did_anything_else(backend, passes): + pc = Postcondition(kind=PostconditionKind.LOCAL_EDIT, field_selector="#a") + assert _replayer(backend)._postcondition_passes(pc, b"", None, START) is passes + + +def test_the_local_edit_check_fails_closed_when_it_cant_observe(): + pc = Postcondition(kind=PostconditionKind.LOCAL_EDIT) + assert _replayer(_Backend())._postcondition_passes(pc, b"", None, {}) is False + + +@pytest.mark.parametrize( + "label, passes", + [ + ("Massachusetts", True), + (" massachusetts ", True), + ("Maine", False), + (None, False), + ], +) +def test_the_field_value_check_reads_the_dropdown(label, passes): + pc = Postcondition(kind=PostconditionKind.FIELD_VALUE, param="state") + replayer = _replayer(_Backend(label=label)) + assert replayer._postcondition_passes(pc, b"", None, START) is passes + + +@pytest.mark.parametrize( + "text, passes", + [ + ("Massachusetts", True), + ("massachusetts", False), + ("Massachusetts ", False), + (None, False), + ], +) +def test_the_field_value_check_reads_a_text_field_exactly(text, passes): + pc = Postcondition(kind=PostconditionKind.FIELD_VALUE, param="state") + replayer = _replayer(_Backend(label=None, text=text)) + assert replayer._postcondition_passes(pc, b"", None, START) is passes + + +# -- compiled end to end: the What's New dismiss gets no record_written (#1377) --------------- + + +def _whats_new_recording(tmp_path, *, dismiss_post_t: Optional[float]): + import json + + from openadapt_flow.compiler.effect_mining import SOR_AFTER_KEY, SOR_BEFORE_KEY + + page = { + "url_before": URL, + "url_after": URL, + "title_before": "MockMed", + "title_after": "MockMed", + "pages_before": 1, + "pages_after": 1, + } + chart = [{"id": "e1", "type": "Encounter", "patient_id": "p1"}] + events = [ + { + "i": 0, + "kind": "click", + "x": 129, + "y": 236, + "t": 1.0, + "structural": { + "selector": "#notice-continue", + "role": "button", + "name": "Continue to tasks", + }, + SOR_BEFORE_KEY: [], + SOR_AFTER_KEY: [], + **page, + }, + { + "i": 1, + "kind": "click", + "x": 300, + "y": 400, + "t": 3.0, + "structural": { + "selector": "#save", + "role": "button", + "name": "Save encounter", + }, + SOR_BEFORE_KEY: [], + SOR_AFTER_KEY: chart, + **page, + }, + ] + rec = tmp_path / "recording" + (rec / "frames").mkdir(parents=True) + from PIL import Image + + for i in range(len(events)): + for side in ("before", "after"): + Image.new("RGB", (1280, 800), "white").save( + rec / "frames" / f"{i:04d}_{side}.png" + ) + (rec / "events.jsonl").write_text("".join(json.dumps(e) + "\n" for e in events)) + (rec / "meta.json").write_text(json.dumps({"params": {}})) + lines = [{"method": "POST", "url": URL + "/api/encounters", "t": 3.2, "seq": 1}] + if dismiss_post_t is not None: + lines.insert( + 0, + {"method": "POST", "url": URL + "/api/seen", "t": dismiss_post_t, "seq": 0}, + ) + header = {"schema": "openadapt.network-capture/1", "exchanges": len(lines)} + (rec / "network.jsonl").write_text( + "".join(json.dumps(x) + "\n" for x in [header, *lines]) + ) + return rec + + +def test_a_compiled_whats_new_dismiss_gets_no_record_written(tmp_path): + from openadapt_flow.compiler.compile import compile_recording + from openadapt_flow.runtime.effects import EffectKind + + rec = _whats_new_recording(tmp_path, dismiss_post_t=None) + wf = compile_recording(rec, tmp_path / "bundle", name="notice", mine_effects=True) + dismiss, save = wf.steps + assert dismiss.risk == "reversible" and not dismiss.risk_review_required + assert is_local_edit(dismiss) and dismiss.effects == [] + assert EffectKind.RECORD_WRITTEN in [e.kind for e in save.effects] + + +def test_a_compiled_dismiss_that_sent_a_write_keeps_its_risk(tmp_path): + from openadapt_flow.compiler.compile import compile_recording + + rec = _whats_new_recording(tmp_path, dismiss_post_t=1.1) + wf = compile_recording(rec, tmp_path / "bundle", name="notice", mine_effects=True) + dismiss = wf.steps[0] + assert dismiss.risk == "irreversible" and not is_local_edit(dismiss)