From fb27548a1da35c1b603850f18b5d27e8c5571577 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 13:08:15 -0400 Subject: [PATCH 01/13] flow: the compatibility matrix, checked against the runtime in CI openadapt_flow/compatibility.py holds every feature (API step, pipelined read-back, self-repair, standard changes, parallel workers, multi-surface) against every run profile (default gated, durable, governed, pinned, qualified, managed), plus the API step with each other feature. Each cell works or is refused with its reason verbatim, and names the tests behind it. docs/COMPATIBILITY.md is generated from it (scripts/build_compatibility_doc.py) and linked from CAPABILITIES.md. tests/test_compatibility_matrix.py runs the API-step and read-back cells through the real replayer under each profile and fails when a cell that should work falls back, or a refusal gives another reason. The pool and self-repair cells call the refusals the CLI uses (the pool's is now cli_worklist_pool.pool_profile_refusal). Two workers each running a Regulated, durable, gated API-step item read VERIFIED. Building that probe surfaced a race in the durable authority database's first creation (fixed separately in #1363 and carried here). Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 1 + llms-full.txt | 1 + packages/flow/docs/API_TIER.md | 3 + packages/flow/docs/COMPATIBILITY.md | 180 +++++ packages/flow/openadapt_flow/__main__.py | 14 +- .../flow/openadapt_flow/cli_worklist_pool.py | 15 + packages/flow/openadapt_flow/compatibility.py | 655 ++++++++++++++++++ .../runtime/durable/authority.py | 19 + .../flow/scripts/build_compatibility_doc.py | 36 + .../flow/tests/test_compatibility_matrix.py | 386 +++++++++++ .../tests/test_durable_authority_create.py | 69 ++ 11 files changed, 1373 insertions(+), 6 deletions(-) create mode 100644 packages/flow/docs/COMPATIBILITY.md create mode 100644 packages/flow/openadapt_flow/compatibility.py create mode 100644 packages/flow/scripts/build_compatibility_doc.py create mode 100644 packages/flow/tests/test_compatibility_matrix.py create mode 100644 packages/flow/tests/test_durable_authority_create.py diff --git a/CAPABILITIES.md b/CAPABILITIES.md index b29347101..183e27445 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -185,6 +185,7 @@ improvising or retrying when something is off. | Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#in-a-durable-run) | | Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | In review ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-governance) | | Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | In review ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | +| Say, for every feature under every run profile, whether it works or is refused and why, checked against the runtime in CI | In review | Synthetic | [COMPATIBILITY.md](packages/flow/docs/COMPATIBILITY.md) | | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | diff --git a/llms-full.txt b/llms-full.txt index 2fcd25df9..f03cf2fbd 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -541,6 +541,7 @@ improvising or retrying when something is off. | Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#in-a-durable-run) | | Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | In review ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-governance) | | Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | In review ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | +| Say, for every feature under every run profile, whether it works or is refused and why, checked against the runtime in CI | In review | Synthetic | [COMPATIBILITY.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/COMPATIBILITY.md) | | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | diff --git a/packages/flow/docs/API_TIER.md b/packages/flow/docs/API_TIER.md index 9a6c19751..0c195ff66 100644 --- a/packages/flow/docs/API_TIER.md +++ b/packages/flow/docs/API_TIER.md @@ -220,6 +220,9 @@ Not yet covered, so the GUI steps run with that reason: probes and resumed run use the GUI steps, because the repair and the person's approval are about the GUI step ([SELF_REPAIR.md](SELF_REPAIR.md)). +Every feature under every run profile, and the API step with each other +feature, is in [COMPATIBILITY.md](COMPATIBILITY.md). + ## When it doesn't run The API step runs only in a plain linear run. It doesn't run, and the GUI diff --git a/packages/flow/docs/COMPATIBILITY.md b/packages/flow/docs/COMPATIBILITY.md new file mode 100644 index 000000000..9b4b5c395 --- /dev/null +++ b/packages/flow/docs/COMPATIBILITY.md @@ -0,0 +1,180 @@ +# Compatibility: features and run profiles + + + +Each feature either works under a run profile or is refused with a +named reason. Nothing falls back silently. When a run takes the slower or +narrower path, it records why: an API step's `api_fallback`, a +pipelined read-back's `inline_reason`, or the command's refusal before +anything runs. `tests/test_compatibility_matrix.py` checks every cell +against the runtime, and fails when a cell that should work falls back +or a refusal gives another reason. + +## The widest combination today + +A Regulated, durable, governed run with per-item approval, on parallel +workers, using the API step for an admitted save, is the combination +where the most work together: each worker's item sends one request a +person approved, checks it with the system-of-record read-back, and +reads `VERIFIED`. In that run the read-back runs inline (durable), and +self-repair runs through the Python API under the Standard contract. +Under a signed qualification admission (what `openadapt-flow run` +requires for Standard and Regulated actuation) the API step and +self-repair are still refused, each with its reason below. + +## Profiles + +- **Default (gated).** `openadapt-flow run` with no `--profile`: a run authorization from the run gate, and per-item approval (`each_item`), the default approval mode. +- **Durable.** `--durable`: a checkpoint after every verified step, and pause and resume from the last one. +- **Governed.** The Standard or Regulated profile contract with a run authorization from the run gate: durable, an identity check on every save, and a record check that reads the system of record. This is how the Python API and the tutorial run it. +- **Pinned.** `--pin-digest`: the run refuses any bundle but the one with that sealed content digest. +- **Qualified.** A signed production qualification admission (`--qualification-authority-file`), which `openadapt-flow run` requires for Standard and Regulated actuation. +- **Managed.** A Cloud-dispatched run (`--managed-dispatch-file`), whose run authorization comes from Cloud. + +## Features + +- **API step** ([API_TIER.md](API_TIER.md)). An admitted save compiled from network capture, run as one request in place of its GUI steps. +- **Pipelined read-back** ([THROUGHPUT.md](THROUGHPUT.md)). `--readback-window`: an item's record check runs while the next item starts. +- **Self-repair** ([SELF_REPAIR.md](SELF_REPAIR.md)). `--self-repair`: an agent proposes a fix for a halt on a changed screen; probes and a person's approval decide it. +- **Standard changes** ([SELF_REPAIR.md](SELF_REPAIR.md#standard-changes-repairs-without-a-person)). Self-repair fixes of a class a person approved in advance, in a signed repair policy. +- **Parallel workers** ([THROUGHPUT.md](THROUGHPUT.md)). `--workers N`: worklist items on several signed-in sessions at once. +- **Multi-surface** ([MULTI_SURFACE.md](MULTI_SURFACE.md)). One program across several windows, tabs or apps. + +## At a glance + +| Feature | Default (gated) | Durable | Governed | Pinned | Qualified | Managed | +| --- | --- | --- | --- | --- | --- | --- | +| API step | Works | Works | Works | Works | Refused | Refused | +| Pipelined read-back | Works | Refused | Refused | Works | Refused | Refused | +| Self-repair | Works | Works | Works | Refused | Refused | Refused | +| Standard changes | Works | Works | Works | Refused | Refused | Refused | +| Parallel workers | Works | Works | Works | Works | Works | Refused | +| Multi-surface | Works | Works | Works | Works | Works | Works | + +Every refusal names its reason (below), and a run or command that +refuses says it in those words. + +## Features by profile + +### API step + +| Profile | Result | +| --- | --- | +| Default (gated) | Works: the person approves the exact request | +| Durable | Works: a save that may have landed is never resent | +| Governed | Works: needs `api admit --identity` | +| Pinned | Works: the pin is checked before the run | +| Qualified | Refused: a production qualification admission doesn't cover API steps yet, so the qualified GUI path runs | +| Managed | Refused: a managed dispatch doesn't bind API steps yet | + +### Pipelined read-back + +| Profile | Result | +| --- | --- | +| Default (gated) | Works: GUI and API saves | +| Durable | Refused: durable runs verify inline (the record check runs before the next item) | +| Governed | Refused: durable runs verify inline (Standard and Regulated are durable) | +| Pinned | Works: when the run isn't durable | +| Qualified | Refused: durable runs verify inline (Standard and Regulated are durable) | +| Managed | Refused: --workers and --readback-window apply to local runs, not a managed dispatch or a qualification case | + +### Self-repair + +| Profile | Result | +| --- | --- | +| Default (gated) | Works | +| Durable | Works: self-repair always runs durable | +| Governed | Works: Standard contract through the Python API | +| Pinned | Refused: --pin-digest pins the original bundle | +| Qualified | Refused: a signed qualification admission binds the original bundle; the approved bundle needs its own admission | +| Managed | Refused: a managed dispatch binds the original bundle | + +### Standard changes + +| Profile | Result | +| --- | --- | +| Default (gated) | Works: needs a signed repair policy | +| Durable | Works: needs a signed repair policy | +| Governed | Works: needs a signed repair policy | +| Pinned | Refused: --pin-digest pins the original bundle | +| Qualified | Refused: a signed qualification admission binds the original bundle; the approved bundle needs its own admission | +| Managed | Refused: a managed dispatch binds the original bundle | + +### Parallel workers + +| Profile | Result | +| --- | --- | +| Default (gated) | Works | +| Durable | Works: each item has its own durable run | +| Governed | Works: each item gets its own single-use authorization | +| Pinned | Works | +| Qualified | Works: the admission is checked for every item | +| Managed | Refused: --workers and --readback-window apply to local runs, not a managed dispatch or a qualification case | + +### Multi-surface + +| Profile | Result | +| --- | --- | +| Default (gated) | Works: an API step inside it runs on the GUI (below) | +| Durable | Works: resume re-binds each surface by identity | +| Governed | Works | +| Pinned | Works | +| Qualified | Works: one environment boundary per surface | +| Managed | Works: every backend surface must be bound | + +## The API step with the other features + +| With | Result | +| --- | --- | +| Parallel workers | Works: each item runs its own API step | +| Pipelined read-back | Works: when the run isn't durable | +| Self-repair | Refused: self-repair's probes and resumed run exercise the repaired GUI step, and the person approved that step, so its API step doesn't run (the first, ordinary run can use the API step) | +| Standard changes | Refused: self-repair's probes and resumed run exercise the repaired GUI step, and the person approved that step, so its API step doesn't run (the repaired bundle's test-record run is GUI-only too) | +| Multi-surface | Refused: a run across several windows or tabs uses the GUI steps | + +## Evidence + +Each cell names the tests that show it. They run in CI. + +- API step / Default (gated): `tests/test_api_step_approval.py::test_the_person_approves_the_exact_request_and_it_is_sent_once`, `tests/test_compatibility_matrix.py::test_api_step_runs_where_the_matrix_says` +- API step / Durable: `tests/test_api_step_durable.py::test_a_durable_run_uses_the_api_step_and_checkpoints_after_it`, `tests/test_api_step_durable.py::test_an_uncertain_send_is_never_resent_by_resume` +- API step / Governed: `tests/test_api_step_governed.py::test_the_default_profile_runs_the_admitted_api_step` +- API step / Pinned: `tests/test_compatibility_matrix.py::test_api_step_runs_where_the_matrix_says` +- API step / Qualified: `tests/test_compatibility_matrix.py::test_api_step_refuses_where_the_matrix_says` +- API step / Managed: `tests/test_compatibility_matrix.py::test_api_step_refuses_where_the_matrix_says` +- Pipelined read-back / Default (gated): `tests/test_readback_pipeline.py::test_confirmed_readback_settles_the_saved_report`, `tests/test_api_step_readback_repair.py::test_an_api_save_reads_back_while_the_next_item_runs`, `tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says` +- Pipelined read-back / Durable: `tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says` +- Pipelined read-back / Governed: `tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says` +- Pipelined read-back / Pinned: `tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says` +- Pipelined read-back / Qualified: `tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says` +- Pipelined read-back / Managed: `tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says` +- Self-repair / Default (gated): `tests/test_self_repair_cli.py::test_run_self_repair_cli_resumes_after_one_phone_tap`, `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Self-repair / Durable: `tests/test_self_repair.py::test_renamed_button_is_repaired_approved_on_the_phone_and_verified` +- Self-repair / Governed: `tests/test_self_repair.py::test_renamed_button_is_repaired_approved_on_the_phone_and_verified` +- Self-repair / Pinned: `tests/test_self_repair_cli.py::test_run_self_repair_refuses_a_pinned_bundle`, `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Self-repair / Qualified: `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Self-repair / Managed: `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Standard changes / Default (gated): `tests/test_standard_change.py::test_relabelled_link_is_repaired_without_a_person` +- Standard changes / Durable: `tests/test_standard_change.py::test_relabelled_link_is_repaired_without_a_person` +- Standard changes / Governed: `tests/test_standard_change.py::test_relabelled_link_is_repaired_without_a_person` +- Standard changes / Pinned: `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Standard changes / Qualified: `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Standard changes / Managed: `tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says` +- Parallel workers / Default (gated): `tests/test_cli_worklist_pool.py::test_two_workers_split_the_items_one_row_each`, `tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says` +- Parallel workers / Durable: `tests/test_compatibility_matrix.py::test_workers_run_governed_durable_api_items` +- Parallel workers / Governed: `tests/test_compatibility_matrix.py::test_workers_run_governed_durable_api_items` +- Parallel workers / Pinned: `tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says` +- Parallel workers / Qualified: `tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says` +- Parallel workers / Managed: `tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says` +- Multi-surface / Default (gated): `tests/test_multi_surface_governed.py::test_governed_durable_multi_surface_run_reaches_verified` +- Multi-surface / Durable: `tests/test_multi_surface_identity.py::test_resume_rebinds_surfaces_by_identity` +- Multi-surface / Governed: `tests/test_multi_surface_governed.py::test_governed_durable_multi_surface_run_reaches_verified` +- Multi-surface / Pinned: `tests/test_multi_surface_governed.py::test_governed_durable_multi_surface_run_reaches_verified` +- Multi-surface / Qualified: `tests/test_multi_surface_governed.py::test_case_evidence_needs_a_matching_observation_per_used_surface` +- Multi-surface / Managed: `tests/test_multi_surface_contract.py::test_managed_dispatch_needs_every_backend_surface_bound` +- API step / Parallel workers: `tests/test_compatibility_matrix.py::test_workers_run_governed_durable_api_items` +- API step / Pipelined read-back: `tests/test_api_step_readback_repair.py::test_an_api_save_reads_back_while_the_next_item_runs` +- API step / Self-repair: `tests/test_api_step_readback_repair.py::test_a_probe_or_resumed_run_never_sends_the_api_step` +- API step / Standard changes: `tests/test_api_step_readback_repair.py::test_a_standard_change_test_record_run_is_gui_only` +- API step / Multi-surface: `tests/test_compatibility_matrix.py::test_api_step_refuses_in_a_multi_surface_run` diff --git a/packages/flow/openadapt_flow/__main__.py b/packages/flow/openadapt_flow/__main__.py index 46023e918..3fdb5bf40 100644 --- a/packages/flow/openadapt_flow/__main__.py +++ b/packages/flow/openadapt_flow/__main__.py @@ -3603,12 +3603,14 @@ def _cmd_run(args: argparse.Namespace) -> int: # Parallel worklist (docs/THROUGHPUT.md): every item gets its own # single-use authorization, bound to that item's one-row worklist and # otherwise identical to the run's (same gate, policy, approvals). - if dispatch_file or qualification_case is not None: - print( - "run REFUSED: --workers and --readback-window apply to local " - "runs, not a managed dispatch or a qualification case. Nothing " - "was executed." - ) + from openadapt_flow.cli_worklist_pool import pool_profile_refusal + + pool_refused = pool_profile_refusal( + managed=bool(dispatch_file), + qualification_case=qualification_case is not None, + ) + if pool_refused is not None: + print(f"run REFUSED: {pool_refused}. Nothing was executed.") return 2 try: relation = loop_relation(workflow) diff --git a/packages/flow/openadapt_flow/cli_worklist_pool.py b/packages/flow/openadapt_flow/cli_worklist_pool.py index 43d6029a3..a09f17bf7 100644 --- a/packages/flow/openadapt_flow/cli_worklist_pool.py +++ b/packages/flow/openadapt_flow/cli_worklist_pool.py @@ -60,6 +60,21 @@ def requested_pool(args: Any, cfg: Any) -> Optional[PoolPolicy]: ) +def pool_profile_refusal(*, managed: bool, qualification_case: bool) -> Optional[str]: + """Why ``--workers`` / ``--readback-window`` can't run here, or None. + + Workers and a read-back window are for local runs. A managed dispatch and + a qualification case each bind one exact run. + """ + + if managed or qualification_case: + return ( + "--workers and --readback-window apply to local runs, not a managed " + "dispatch or a qualification case" + ) + return None + + def loop_relation(workflow: Any) -> str: """The relation of a single-LOOP program (the ``for-each`` shape). diff --git a/packages/flow/openadapt_flow/compatibility.py b/packages/flow/openadapt_flow/compatibility.py new file mode 100644 index 000000000..f5a5cbc2b --- /dev/null +++ b/packages/flow/openadapt_flow/compatibility.py @@ -0,0 +1,655 @@ +"""Which features work under which run profiles: the compatibility matrix. + +Each cell either works or is refused with a named reason. A refused feature +never falls back silently: the run either says why it used the slower or +narrower path (an API step's ``api_fallback``, a pipelined read-back's +``inline_reason``), or the command refuses before anything runs. + +``tests/test_compatibility_matrix.py`` checks every cell against the runtime: +a cell that works must engage the feature, and a refused cell must produce +exactly the reason written here. ``docs/COMPATIBILITY.md`` is generated from +this module by ``scripts/build_compatibility_doc.py``. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Literal, Optional + +Status = Literal["works", "refused"] + + +@dataclass(frozen=True) +class Profile: + key: str + title: str + description: str + + +@dataclass(frozen=True) +class Feature: + key: str + title: str + description: str + doc: str + + +@dataclass(frozen=True) +class Cell: + feature: str + profile: str + status: Status + #: For a refused cell: the reason the run or the command gives, verbatim. + reason: Optional[str] = None + #: A short note for the table, such as a condition the cell needs. + note: Optional[str] = None + #: The tests that show the cell's behavior (``file::test``). + evidence: tuple[str, ...] = () + + +PROFILES: tuple[Profile, ...] = ( + Profile( + "default", + "Default (gated)", + "`openadapt-flow run` with no `--profile`: a run authorization from the " + "run gate, and per-item approval (`each_item`), the default approval mode.", + ), + Profile( + "durable", + "Durable", + "`--durable`: a checkpoint after every verified step, and pause and " + "resume from the last one.", + ), + Profile( + "governed", + "Governed", + "The Standard or Regulated profile contract with a run authorization " + "from the run gate: durable, an identity check on every save, and a " + "record check that reads the system of record. This is how the Python " + "API and the tutorial run it.", + ), + Profile( + "pinned", + "Pinned", + "`--pin-digest`: the run refuses any bundle but the one with that " + "sealed content digest.", + ), + Profile( + "qualified", + "Qualified", + "A signed production qualification admission " + "(`--qualification-authority-file`), which `openadapt-flow run` " + "requires for Standard and Regulated actuation.", + ), + Profile( + "managed", + "Managed", + "A Cloud-dispatched run (`--managed-dispatch-file`), whose run " + "authorization comes from Cloud.", + ), +) + +FEATURES: tuple[Feature, ...] = ( + Feature( + "api_step", + "API step", + "An admitted save compiled from network capture, run as one request in " + "place of its GUI steps.", + "API_TIER.md", + ), + Feature( + "pipelined_readback", + "Pipelined read-back", + "`--readback-window`: an item's record check runs while the next item starts.", + "THROUGHPUT.md", + ), + Feature( + "self_repair", + "Self-repair", + "`--self-repair`: an agent proposes a fix for a halt on a changed " + "screen; probes and a person's approval decide it.", + "SELF_REPAIR.md", + ), + Feature( + "standard_changes", + "Standard changes", + "Self-repair fixes of a class a person approved in advance, in a " + "signed repair policy.", + "SELF_REPAIR.md#standard-changes-repairs-without-a-person", + ), + Feature( + "parallel_workers", + "Parallel workers", + "`--workers N`: worklist items on several signed-in sessions at once.", + "THROUGHPUT.md", + ), + Feature( + "multi_surface", + "Multi-surface", + "One program across several windows, tabs or apps.", + "MULTI_SURFACE.md", + ), +) + +API_QUALIFIED = ( + "a production qualification admission doesn't cover API steps yet, so the " + "qualified GUI path runs" +) +API_MANAGED = "a managed dispatch doesn't bind API steps yet" +DURABLE_INLINE = "durable runs verify inline" +POOL_LOCAL_ONLY = ( + "--workers and --readback-window apply to local runs, not a managed dispatch " + "or a qualification case" +) +REPAIR_PINNED = "--pin-digest pins the original bundle" +REPAIR_QUALIFIED = ( + "a signed qualification admission binds the original bundle; the approved " + "bundle needs its own admission" +) +REPAIR_MANAGED = "a managed dispatch binds the original bundle" + +_REPAIR_EVIDENCE = ( + "tests/test_self_repair.py::test_renamed_button_is_repaired_approved_on_the_phone_and_verified", +) +_STANDARD_EVIDENCE = ( + "tests/test_standard_change.py::test_relabelled_link_is_repaired_without_a_person", +) + +CELLS: tuple[Cell, ...] = ( + # -- API step --------------------------------------------------------------- + Cell( + "api_step", + "default", + "works", + note="the person approves the exact request", + evidence=( + "tests/test_api_step_approval.py::test_the_person_approves_the_exact_request_and_it_is_sent_once", + "tests/test_compatibility_matrix.py::test_api_step_runs_where_the_matrix_says", + ), + ), + Cell( + "api_step", + "durable", + "works", + note="a save that may have landed is never resent", + evidence=( + "tests/test_api_step_durable.py::test_a_durable_run_uses_the_api_step_and_checkpoints_after_it", + "tests/test_api_step_durable.py::test_an_uncertain_send_is_never_resent_by_resume", + ), + ), + Cell( + "api_step", + "governed", + "works", + note="needs `api admit --identity`", + evidence=( + "tests/test_api_step_governed.py::test_the_default_profile_runs_the_admitted_api_step", + ), + ), + Cell( + "api_step", + "pinned", + "works", + note="the pin is checked before the run", + evidence=( + "tests/test_compatibility_matrix.py::test_api_step_runs_where_the_matrix_says", + ), + ), + Cell( + "api_step", + "qualified", + "refused", + reason=API_QUALIFIED, + evidence=( + "tests/test_compatibility_matrix.py::test_api_step_refuses_where_the_matrix_says", + ), + ), + Cell( + "api_step", + "managed", + "refused", + reason=API_MANAGED, + evidence=( + "tests/test_compatibility_matrix.py::test_api_step_refuses_where_the_matrix_says", + ), + ), + # -- pipelined read-back ------------------------------------------------------ + Cell( + "pipelined_readback", + "default", + "works", + note="GUI and API saves", + evidence=( + "tests/test_readback_pipeline.py::test_confirmed_readback_settles_the_saved_report", + "tests/test_api_step_readback_repair.py::test_an_api_save_reads_back_while_the_next_item_runs", + "tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says", + ), + ), + Cell( + "pipelined_readback", + "durable", + "refused", + reason=DURABLE_INLINE, + note="the record check runs before the next item", + evidence=( + "tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says", + ), + ), + Cell( + "pipelined_readback", + "governed", + "refused", + reason=DURABLE_INLINE, + note="Standard and Regulated are durable", + evidence=( + "tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says", + ), + ), + Cell( + "pipelined_readback", + "pinned", + "works", + note="when the run isn't durable", + evidence=( + "tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says", + ), + ), + Cell( + "pipelined_readback", + "qualified", + "refused", + reason=DURABLE_INLINE, + note="Standard and Regulated are durable", + evidence=( + "tests/test_compatibility_matrix.py::test_pipelined_readback_where_the_matrix_says", + ), + ), + Cell( + "pipelined_readback", + "managed", + "refused", + reason=POOL_LOCAL_ONLY, + evidence=( + "tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says", + ), + ), + # -- self-repair --------------------------------------------------------------- + Cell( + "self_repair", + "default", + "works", + evidence=( + "tests/test_self_repair_cli.py::test_run_self_repair_cli_resumes_after_one_phone_tap", + "tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says", + ), + ), + Cell( + "self_repair", + "durable", + "works", + note="self-repair always runs durable", + evidence=_REPAIR_EVIDENCE, + ), + Cell( + "self_repair", + "governed", + "works", + note="Standard contract through the Python API", + evidence=_REPAIR_EVIDENCE, + ), + Cell( + "self_repair", + "pinned", + "refused", + reason=REPAIR_PINNED, + evidence=( + "tests/test_self_repair_cli.py::test_run_self_repair_refuses_a_pinned_bundle", + "tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says", + ), + ), + Cell( + "self_repair", + "qualified", + "refused", + reason=REPAIR_QUALIFIED, + evidence=( + "tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says", + ), + ), + Cell( + "self_repair", + "managed", + "refused", + reason=REPAIR_MANAGED, + evidence=( + "tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says", + ), + ), + # -- standard changes ----------------------------------------------------------- + Cell( + "standard_changes", + "default", + "works", + note="needs a signed repair policy", + evidence=_STANDARD_EVIDENCE, + ), + Cell( + "standard_changes", + "durable", + "works", + note="needs a signed repair policy", + evidence=_STANDARD_EVIDENCE, + ), + Cell( + "standard_changes", + "governed", + "works", + note="needs a signed repair policy", + evidence=_STANDARD_EVIDENCE, + ), + Cell( + "standard_changes", + "pinned", + "refused", + reason=REPAIR_PINNED, + evidence=( + "tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says", + ), + ), + Cell( + "standard_changes", + "qualified", + "refused", + reason=REPAIR_QUALIFIED, + evidence=( + "tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says", + ), + ), + Cell( + "standard_changes", + "managed", + "refused", + reason=REPAIR_MANAGED, + evidence=( + "tests/test_compatibility_matrix.py::test_self_repair_where_the_matrix_says", + ), + ), + # -- parallel workers ----------------------------------------------------------- + Cell( + "parallel_workers", + "default", + "works", + evidence=( + "tests/test_cli_worklist_pool.py::test_two_workers_split_the_items_one_row_each", + "tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says", + ), + ), + Cell( + "parallel_workers", + "durable", + "works", + note="each item has its own durable run", + evidence=( + "tests/test_compatibility_matrix.py::test_workers_run_governed_durable_api_items", + ), + ), + Cell( + "parallel_workers", + "governed", + "works", + note="each item gets its own single-use authorization", + evidence=( + "tests/test_compatibility_matrix.py::test_workers_run_governed_durable_api_items", + ), + ), + Cell( + "parallel_workers", + "pinned", + "works", + evidence=( + "tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says", + ), + ), + Cell( + "parallel_workers", + "qualified", + "works", + note="the admission is checked for every item", + evidence=( + "tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says", + ), + ), + Cell( + "parallel_workers", + "managed", + "refused", + reason=POOL_LOCAL_ONLY, + evidence=( + "tests/test_compatibility_matrix.py::test_the_worklist_pool_where_the_matrix_says", + ), + ), + # -- multi-surface ---------------------------------------------------------------- + Cell( + "multi_surface", + "default", + "works", + note="an API step inside it runs on the GUI (below)", + evidence=( + "tests/test_multi_surface_governed.py::test_governed_durable_multi_surface_run_reaches_verified", + ), + ), + Cell( + "multi_surface", + "durable", + "works", + note="resume re-binds each surface by identity", + evidence=( + "tests/test_multi_surface_identity.py::test_resume_rebinds_surfaces_by_identity", + ), + ), + Cell( + "multi_surface", + "governed", + "works", + evidence=( + "tests/test_multi_surface_governed.py::test_governed_durable_multi_surface_run_reaches_verified", + ), + ), + Cell( + "multi_surface", + "pinned", + "works", + evidence=( + "tests/test_multi_surface_governed.py::test_governed_durable_multi_surface_run_reaches_verified", + ), + ), + Cell( + "multi_surface", + "qualified", + "works", + note="one environment boundary per surface", + evidence=( + "tests/test_multi_surface_governed.py::test_case_evidence_needs_a_matching_observation_per_used_surface", + ), + ), + Cell( + "multi_surface", + "managed", + "works", + note="every backend surface must be bound", + evidence=( + "tests/test_multi_surface_contract.py::test_managed_dispatch_needs_every_backend_surface_bound", + ), + ), +) + +#: The API step combined with the other features, in one run. +API_MULTI_SURFACE = "a run across several windows or tabs uses the GUI steps" +API_REPAIR_PROBE = ( + "self-repair's probes and resumed run exercise the repaired GUI step, and " + "the person approved that step, so its API step doesn't run" +) + +COMBINATIONS: tuple[Cell, ...] = ( + Cell( + "api_step", + "parallel_workers", + "works", + note="each item runs its own API step", + evidence=( + "tests/test_compatibility_matrix.py::test_workers_run_governed_durable_api_items", + ), + ), + Cell( + "api_step", + "pipelined_readback", + "works", + note="when the run isn't durable", + evidence=( + "tests/test_api_step_readback_repair.py::test_an_api_save_reads_back_while_the_next_item_runs", + ), + ), + Cell( + "api_step", + "self_repair", + "refused", + reason=API_REPAIR_PROBE, + note="the first, ordinary run can use the API step", + evidence=( + "tests/test_api_step_readback_repair.py::test_a_probe_or_resumed_run_never_sends_the_api_step", + ), + ), + Cell( + "api_step", + "standard_changes", + "refused", + reason=API_REPAIR_PROBE, + note="the repaired bundle's test-record run is GUI-only too", + evidence=( + "tests/test_api_step_readback_repair.py::test_a_standard_change_test_record_run_is_gui_only", + ), + ), + Cell( + "api_step", + "multi_surface", + "refused", + reason=API_MULTI_SURFACE, + evidence=( + "tests/test_compatibility_matrix.py::test_api_step_refuses_in_a_multi_surface_run", + ), + ), +) + + +def cell(feature: str, profile: str) -> Cell: + """The cell for ``feature`` under ``profile``. Raises ``KeyError``.""" + + for item in CELLS: + if item.feature == feature and item.profile == profile: + return item + raise KeyError((feature, profile)) + + +def _mark(item: Cell) -> str: + if item.status == "works": + text = "Works" + if item.note: + text += f": {item.note}" + return text + text = f"Refused: {item.reason}" + if item.note: + text += f" ({item.note})" + return text + + +def render_markdown() -> str: + """The body of ``docs/COMPATIBILITY.md``, generated from this module.""" + + feature_titles = {f.key: f.title for f in FEATURES} + lines = [ + "# Compatibility: features and run profiles", + "", + "", + "", + "Each feature either works under a run profile or is refused with a", + "named reason. Nothing falls back silently. When a run takes the slower or", + "narrower path, it records why: an API step's `api_fallback`, a", + "pipelined read-back's `inline_reason`, or the command's refusal before", + "anything runs. `tests/test_compatibility_matrix.py` checks every cell", + "against the runtime, and fails when a cell that should work falls back", + "or a refusal gives another reason.", + "", + "## The widest combination today", + "", + "A Regulated, durable, governed run with per-item approval, on parallel", + "workers, using the API step for an admitted save, is the combination", + "where the most work together: each worker's item sends one request a", + "person approved, checks it with the system-of-record read-back, and", + "reads `VERIFIED`. In that run the read-back runs inline (durable), and", + "self-repair runs through the Python API under the Standard contract.", + "Under a signed qualification admission (what `openadapt-flow run`", + "requires for Standard and Regulated actuation) the API step and", + "self-repair are still refused, each with its reason below.", + "", + "## Profiles", + "", + ] + for profile in PROFILES: + lines.append(f"- **{profile.title}.** {profile.description}") + lines += ["", "## Features", ""] + for feature in FEATURES: + lines.append( + f"- **{feature.title}** ([{feature.doc.split('#')[0]}]({feature.doc})). " + f"{feature.description}" + ) + lines += [ + "", + "## At a glance", + "", + "| Feature | " + " | ".join(p.title for p in PROFILES) + " |", + "| --- | " + " | ".join("---" for _ in PROFILES) + " |", + ] + for feature in FEATURES: + marks = [ + "Works" if cell(feature.key, p.key).status == "works" else "Refused" + for p in PROFILES + ] + lines.append(f"| {feature.title} | " + " | ".join(marks) + " |") + lines += [ + "", + "Every refusal names its reason (below), and a run or command that", + "refuses says it in those words.", + ] + lines += ["", "## Features by profile", ""] + for feature in FEATURES: + lines += [f"### {feature.title}", "", "| Profile | Result |", "| --- | --- |"] + for profile in PROFILES: + lines.append( + f"| {profile.title} | {_mark(cell(feature.key, profile.key))} |" + ) + lines.append("") + lines += [ + "## The API step with the other features", + "", + "| With | Result |", + "| --- | --- |", + ] + for item in COMBINATIONS: + lines.append(f"| {feature_titles[item.profile]} | {_mark(item)} |") + lines += [ + "", + "## Evidence", + "", + "Each cell names the tests that show it. They run in CI.", + "", + ] + for item in (*CELLS, *COMBINATIONS): + against = next( + (p.title for p in PROFILES if p.key == item.profile), + feature_titles.get(item.profile, item.profile), + ) + tests = ", ".join(f"`{t}`" for t in item.evidence) + lines.append(f"- {feature_titles[item.feature]} / {against}: {tests}") + lines.append("") + return "\n".join(lines) diff --git a/packages/flow/openadapt_flow/runtime/durable/authority.py b/packages/flow/openadapt_flow/runtime/durable/authority.py index 03e3fe649..c5183fb44 100644 --- a/packages/flow/openadapt_flow/runtime/durable/authority.py +++ b/packages/flow/openadapt_flow/runtime/durable/authority.py @@ -655,6 +655,25 @@ def _connect(self) -> sqlite3.Connection: raise DurableAuthorityBusy( "the external durable authority database permissions are too broad" ) + elif os.name != "nt": + # Create the database owner-only before SQLite opens it. SQLite + # would create it with the process umask and the chmod below comes + # later, so a second durable run starting at the same moment (two + # workers, two processes) saw a world-readable file and refused. + try: + descriptor = os.open( + self.db_path, + os.O_CREAT | os.O_EXCL | os.O_WRONLY | getattr(os, "O_NOFOLLOW", 0), + 0o600, + ) + except FileExistsError: + pass + except OSError as exc: + raise DurableAuthorityBusy( + "the external durable authority database can't be created" + ) from exc + else: + os.close(descriptor) connection: Optional[sqlite3.Connection] = None try: connection = sqlite3.connect( diff --git a/packages/flow/scripts/build_compatibility_doc.py b/packages/flow/scripts/build_compatibility_doc.py new file mode 100644 index 000000000..c25e9618d --- /dev/null +++ b/packages/flow/scripts/build_compatibility_doc.py @@ -0,0 +1,36 @@ +"""Write docs/COMPATIBILITY.md from openadapt_flow/compatibility.py. + +Usage (from packages/flow): python scripts/build_compatibility_doc.py +``--check`` exits 1 when the file is out of date instead of writing it. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from openadapt_flow.compatibility import render_markdown # noqa: E402 + +TARGET = ROOT / "docs" / "COMPATIBILITY.md" + + +def main(argv: list[str]) -> int: + text = render_markdown() + if "--check" in argv: + current = TARGET.read_text(encoding="utf-8") if TARGET.exists() else "" + if current != text: + print( + f"{TARGET} is out of date: run python scripts/build_compatibility_doc.py" + ) + return 1 + return 0 + TARGET.write_text(text, encoding="utf-8") + print(f"wrote {TARGET}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/packages/flow/tests/test_compatibility_matrix.py b/packages/flow/tests/test_compatibility_matrix.py new file mode 100644 index 000000000..491ae261b --- /dev/null +++ b/packages/flow/tests/test_compatibility_matrix.py @@ -0,0 +1,386 @@ +"""The compatibility matrix (openadapt_flow/compatibility.py, docs/COMPATIBILITY.md). + +Every feature against every run profile either works or is refused with a +named reason. These tests check each cell against the runtime: + +- a cell that works must engage the feature (the API step sends its request, + the read-back runs on the side thread); a fallback fails the test; +- a refused cell must give exactly the reason the matrix names; +- every cell names tests that exist, and the published page is current. + +The record system is a scripted verifier or the MockMed fault server; the +values are made up. +""" + +from __future__ import annotations + +import argparse +import ast +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Optional + +import pytest + +from openadapt_flow import compatibility as matrix +from openadapt_flow.approval_mode.gate import ApprovalGate +from openadapt_flow.approval_mode.item_approval import ItemApprovalStore +from openadapt_flow.deployment import DeploymentConfig, PolicySection +from openadapt_flow.execution_profiles import ExecutionProfile +from openadapt_flow.run_gate import build_runtime_authorization, evaluate_run_gate +from openadapt_flow.runtime.actuators.web_form import WebFormActuator +from openadapt_flow.runtime.effects.rest import RestRecordVerifier +from openadapt_flow.runtime.readback_pipeline import ReadbackPipeline +from tests.test_api_step_approval import APPROVE, Person +from tests.test_api_step_governed import PARAMS, Transport, _workflow +from tests.test_approval_mode_replayer import RecordSystem +from tests.test_execution_profiles import ( + _KEY, + Replayer, + _gate, + _ReadyVision, + _sealed, +) +from tests.test_replayer import FakeBackend + +FLOW = Path(__file__).resolve().parents[1] +REPO = FLOW.parents[1] + + +class RestRecordSystem(RecordSystem, RestRecordVerifier): + """The scripted record system, read the way a REST read is. + + The read-back pipeline defers only network reads of the system of record + (REST, FHIR, GraphQL, SQL); this is the scripted system as one of them. + """ + + +class Run: + """One item of the API-step program under one run profile.""" + + def __init__( + self, + tmp_path: Path, + profile: str, + *, + pipeline: Optional[ReadbackPipeline] = None, + ) -> None: + self.profile = profile + self.durable = profile in {"durable", "governed", "qualified"} + regulated = profile in {"governed", "qualified"} + self.workflow, self.bundle = _sealed(tmp_path, _workflow(), encrypted=True) + self.verifier = RestRecordSystem() + if regulated: + report = _gate( + self.workflow, + self.bundle, + ExecutionProfile.REGULATED, + verifier=self.verifier, + durable=True, + ) + else: + report = evaluate_run_gate( + self.workflow, + bundle_dir=self.bundle, + deployment=DeploymentConfig(policy=PolicySection(policy="permissive")), + effect_verifier=self.verifier, + effective_durable=self.durable, + ) + assert report.passed, report.render() + self.authorization = build_runtime_authorization( + self.workflow, report, params=PARAMS + ) + if profile == "qualified": + # The fields a signed production admission binds into the run. + self.authorization = self.authorization.model_copy( + update={ + "production_qualification_admission_id": "admission-1", + "production_qualification_admission_sha256": "1" * 64, + "production_qualification_evidence_identity_sha256": "2" * 64, + "production_qualification_runtime_validation_id": "validation-1", + "production_qualification_signer_registry_sha256": "3" * 64, + "production_qualification_signer_registry_revision": 1, + "production_qualification_signer_registry_expires_at": ( + "2099-01-01T00:00:00+00:00" + ), + "production_qualification_authority_sha256": "4" * 64, + } + ) + self.run_dir = tmp_path / "run" + self.person = Person([APPROVE]) + self.store = ItemApprovalStore(self.run_dir) + self.person.store = self.store + self.transport = Transport() + self.pipeline = pipeline + + def replayer(self) -> Any: + gate = ApprovalGate( + "each_item", + approval_store=self.store, + approval_wait_s=30.0, + approval_poll_s=1.0, + sleep=self.person.sleep, + monotonic=self.person.monotonic, + ) + return Replayer( + FakeBackend(), + vision=_ReadyVision(), + effect_verifier=self.verifier, + governed_authorization=self.authorization, + durable=self.durable, + checkpoint_key=_KEY, + require_settled=True, + approval_gate=gate, + web_form_actuator=WebFormActuator(self.transport), + readback_pipeline=self.pipeline, + ) + + def run(self) -> Any: + return self.replayer().run( + self.workflow, + params=PARAMS, + bundle_dir=self.bundle, + run_dir=self.run_dir, + ) + + +def _cells(feature: str, status: str) -> list[str]: + return [ + c.profile for c in matrix.CELLS if c.feature == feature and c.status == status + ] + + +# -- the matrix is complete and backed by tests ------------------------------------- + + +def test_every_feature_has_exactly_one_cell_per_profile(): + pairs = [(c.feature, c.profile) for c in matrix.CELLS] + assert len(pairs) == len(set(pairs)) + assert set(pairs) == { + (f.key, p.key) for f in matrix.FEATURES for p in matrix.PROFILES + } + features = {f.key for f in matrix.FEATURES} + for item in matrix.COMBINATIONS: + assert item.feature in features and item.profile in features + + +@pytest.mark.parametrize( + "item", + [*matrix.CELLS, *matrix.COMBINATIONS], + ids=lambda c: f"{c.feature}-{c.profile}", +) +def test_every_cell_names_a_reason_or_works_and_has_evidence(item): + if item.status == "refused": + assert item.reason, "a refused cell names its reason" + else: + assert item.reason is None + assert item.evidence, "every cell names the tests that show it" + for node in item.evidence: + path, _, name = node.partition("::") + source = FLOW / path + assert source.is_file(), node + names = { + n.name + for n in ast.walk(ast.parse(source.read_text())) + if isinstance(n, ast.FunctionDef) + } + assert name in names, node + + +def test_the_published_page_is_current_and_linked(): + page = FLOW / "docs" / "COMPATIBILITY.md" + assert page.read_text(encoding="utf-8") == matrix.render_markdown(), ( + "run python scripts/build_compatibility_doc.py" + ) + assert ( + "packages/flow/docs/COMPATIBILITY.md" in (REPO / "CAPABILITIES.md").read_text() + ) + + +# -- the API step --------------------------------------------------------------------- + + +@pytest.mark.parametrize("profile", _cells("api_step", "works")) +def test_api_step_runs_where_the_matrix_says(tmp_path, profile): + run = Run(tmp_path, profile) + report = run.run() + + assert [r.actuation for r in report.results] == ["api_folded", "api"], [ + r.api_fallback for r in report.results + ] + assert all(r.api_fallback is None for r in report.results) + assert [path for path, _ in run.transport.posts] == ["/save.php"] + assert report.success is True + (request,) = run.store.requests() + assert request.form_state_kind == "api_request" + if profile == "governed": + assert report.execution_outcome == "VERIFIED" + + +@pytest.mark.parametrize("profile", _cells("api_step", "refused")) +def test_api_step_refuses_where_the_matrix_says(tmp_path, profile): + reason = matrix.cell("api_step", profile).reason + run = Run(tmp_path, "qualified" if profile == "qualified" else "default") + if profile == "managed": + replayer = run.replayer() + replayer.managed_dispatch_binding = object() + _end, _save, why = replayer._web_form_span_at(run.workflow, 0, durable=True) + assert why == reason + return + report = run.run() + assert run.transport.posts == [] + assert report.results[0].api_fallback == reason + + +def test_api_step_refuses_in_a_multi_surface_run(tmp_path): + run = Run(tmp_path, "default") + replayer = run.replayer() + replayer._surface_router = object() + _end, _save, why = replayer._web_form_span_at(run.workflow, 0, durable=False) + assert why == matrix.API_MULTI_SURFACE + + +def test_the_repair_reason_is_the_runtimes(): + from openadapt_flow.repair.self_repair import REPAIR_GUI_ONLY + + assert matrix.API_REPAIR_PROBE == REPAIR_GUI_ONLY + + +# -- pipelined read-back --------------------------------------------------------------- + + +@pytest.mark.parametrize( + "profile", [p.key for p in matrix.PROFILES if p.key != "managed"] +) +def test_pipelined_readback_where_the_matrix_says(tmp_path, profile): + item = matrix.cell("pipelined_readback", profile) + pipeline = ReadbackPipeline() + try: + run = Run(tmp_path, profile, pipeline=pipeline) + report = run.run() + handoff = pipeline.take() + settled = handoff.settled.result(timeout=30) if handoff is not None else None + finally: + pipeline.close() + + assert report.results[-1].effect_verified is True or settled is not None + if item.status == "works": + assert settled is report, pipeline.inline_reason + assert report.success is True + assert report.results[-1].effect_verified is True + else: + assert settled is None + assert pipeline.inline_reason == item.reason + + +# -- the worklist pool ------------------------------------------------------------------- + + +@pytest.mark.parametrize("profile", [p.key for p in matrix.PROFILES]) +def test_the_worklist_pool_where_the_matrix_says(tmp_path, profile): + from openadapt_flow.cli_worklist_pool import pool_profile_refusal + from openadapt_flow.runtime.worklist_pool import ( + FunctionWorker, + PoolPolicy, + run_worklist_pool, + ) + + refusal = pool_profile_refusal( + managed=profile == "managed", qualification_case=False + ) + workers = matrix.cell("parallel_workers", profile) + if workers.status == "refused": + assert refusal == workers.reason + assert matrix.cell("pipelined_readback", profile).reason == refusal + return + assert refusal is None + ran: list[int] = [] + pool = [ + FunctionWorker(f"w{i}", lambda idx, row, d: ran.append(idx) or _Done()) + for i in range(2) + ] + report = run_worklist_pool( + [{"n": str(i)} for i in range(4)], + pool, + run_dir=tmp_path, + policy=PoolPolicy(workers=2, items_independent=True), + ) + assert sorted(ran) == [0, 1, 2, 3] and report.success + + +class _Done: + success = True + execution_outcome = "VERIFIED" + transaction_outcome = "VERIFIED" + halt = None + total_ms = 1.0 + + +def test_workers_run_governed_durable_api_items(tmp_path): + """Two workers, each item a Regulated, durable, gated API-step run.""" + + from openadapt_flow.runtime.worklist_pool import ( + VERIFIED, + FunctionWorker, + PoolPolicy, + run_worklist_pool, + ) + + runs: dict[int, Run] = {} + + def item(idx: int, row: dict[str, Any], item_dir: Path) -> Any: + runs[idx] = Run(item_dir, "governed") + return runs[idx].run() + + report = run_worklist_pool( + [{"n": str(i)} for i in range(4)], + [FunctionWorker(f"w{i}", item) for i in range(2)], + run_dir=tmp_path, + policy=PoolPolicy(workers=2, items_independent=True), + ) + assert {i["state"] for i in report.items} == {VERIFIED}, [ + i.get("detail") or i for i in report.items + ] + for run in runs.values(): + assert [path for path, _ in run.transport.posts] == ["/save.php"] + + +# -- self-repair and standard changes ---------------------------------------------------- + + +def _repair_args(profile: str) -> argparse.Namespace: + return argparse.Namespace( + managed_dispatch_file="dispatch.json" if profile == "managed" else None, + qualification_authority_file=( + "authority.json" if profile == "qualified" else None + ), + _qualification_case_execution=None, + pin_digest="a" * 64 if profile == "pinned" else None, + hold_on_halt=False, + save_healed_to=None, + record_video=False, + repair_agent="mod:factory", + repair_screen_access_by="reviewer", + durable=profile == "durable", + ) + + +@pytest.mark.parametrize("feature", ["self_repair", "standard_changes"]) +@pytest.mark.parametrize("profile", [p.key for p in matrix.PROFILES]) +def test_self_repair_where_the_matrix_says(feature, profile): + from openadapt_flow.cli_self_repair import _refusal + + backend = SimpleNamespace( + kind="web", browser_cdp_endpoint=None, browser_page_url=None + ) + refusal = _refusal(_repair_args(profile), backend, "http://app") + item = matrix.cell(feature, profile) + if item.status == "refused": + assert refusal == item.reason + elif profile == "governed": + # The CLI's Standard profile needs a signed admission (the + # "qualified" column); the Standard contract runs through the + # Python API, which these cells' evidence tests exercise. + assert item.evidence + else: + assert refusal is None diff --git a/packages/flow/tests/test_durable_authority_create.py b/packages/flow/tests/test_durable_authority_create.py new file mode 100644 index 000000000..70588a9ee --- /dev/null +++ b/packages/flow/tests/test_durable_authority_create.py @@ -0,0 +1,69 @@ +"""The external durable authority database is created owner-only. + +Two durable runs that start at the same moment (two workers, or two +processes) both open the authority database. SQLite creates a missing file +with the process umask; if the first run's later chmod hadn't happened yet, +the second run saw a world-readable file and refused. The file now exists, +owner-only, before SQLite opens it. +""" + +from __future__ import annotations + +import os +import sqlite3 +import stat +import threading + +import pytest + +from openadapt_flow.runtime.durable import authority as authority_module +from tests.test_durable_authority_v13 import _fresh_run + + +def _store(path): + """Start one fresh durable run (its own run id and namespace).""" + + import uuid + + return _fresh_run(path, run_id=str(uuid.uuid4()), namespace_id=uuid.uuid4().hex) + + +@pytest.mark.skipif(os.name == "nt", reason="POSIX permission bits") +def test_a_fresh_database_is_owner_only_before_sqlite_opens_it(tmp_path, monkeypatch): + db_path = tmp_path / "authority" / "authority.sqlite3" + monkeypatch.setenv(authority_module.AUTHORITY_DB_ENV, str(db_path)) + seen: list[int] = [] + real_connect = sqlite3.connect + + def connect(*args, **kwargs): + if not seen: + seen.append(stat.S_IMODE(os.lstat(db_path).st_mode)) + return real_connect(*args, **kwargs) + + monkeypatch.setattr(authority_module.sqlite3, "connect", connect) + os.umask(0o022) + _store(tmp_path / "a") + assert seen and seen[0] & 0o077 == 0, oct(seen[0]) + + +@pytest.mark.skipif(os.name == "nt", reason="POSIX permission bits") +def test_two_durable_runs_starting_together_both_start(tmp_path, monkeypatch): + db_path = tmp_path / "authority" / "authority.sqlite3" + monkeypatch.setenv(authority_module.AUTHORITY_DB_ENV, str(db_path)) + errors: list[BaseException] = [] + barrier = threading.Barrier(4) + + def start(name: str) -> None: + try: + barrier.wait() + _store(tmp_path / name) + except BaseException as exc: # noqa: BLE001 - collected for the assert + errors.append(exc) + + threads = [threading.Thread(target=start, args=(f"run-{i}",)) for i in range(4)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + assert errors == [] + assert stat.S_IMODE(os.lstat(db_path).st_mode) & 0o077 == 0 From 7e3bce60a314992053aa07c266adf13645fc53f6 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 13:08:30 -0400 Subject: [PATCH 02/13] docs: name #1364 in the capability row Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 2 +- llms-full.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 183e27445..0877806c6 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -185,7 +185,7 @@ improvising or retrying when something is off. | Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#in-a-durable-run) | | Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | In review ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-governance) | | Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | In review ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | -| Say, for every feature under every run profile, whether it works or is refused and why, checked against the runtime in CI | In review | Synthetic | [COMPATIBILITY.md](packages/flow/docs/COMPATIBILITY.md) | +| Say, for every feature under every run profile, whether it works or is refused and why, checked against the runtime in CI | In review ([#1364](https://github.com/OpenAdaptAI/OpenAdapt/pull/1364)) | Synthetic | [COMPATIBILITY.md](packages/flow/docs/COMPATIBILITY.md) | | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | diff --git a/llms-full.txt b/llms-full.txt index f03cf2fbd..9c07878c5 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -541,7 +541,7 @@ improvising or retrying when something is off. | Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#in-a-durable-run) | | Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | In review ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-governance) | | Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | In review ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | -| Say, for every feature under every run profile, whether it works or is refused and why, checked against the runtime in CI | In review | Synthetic | [COMPATIBILITY.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/COMPATIBILITY.md) | +| Say, for every feature under every run profile, whether it works or is refused and why, checked against the runtime in CI | In review ([#1364](https://github.com/OpenAdaptAI/OpenAdapt/pull/1364)) | Synthetic | [COMPATIBILITY.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/COMPATIBILITY.md) | | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | From 352af585b04648473e50ed86cd8983bd0da69a08 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 13:20:27 -0400 Subject: [PATCH 03/13] bench(flow): measure the referral's API step under each_item approval + durable benchmark/openemr_api_tier/approval_durable.py runs the OpenEMR referral with API steps off and on, both under ApprovalGate("each_item") and a durable run, interleaved, with the setup's own SQL read-back. The harness answers each approval as a stand-in for the person and its wait is measured and excluded. Every number it writes carries the label "API step under each_item approval + durable, no run gate; harness answers approvals; approval wait excluded": the referral bundle doesn't pass the run gate yet, so this is not the governed production profile. The approval view now lists each record-check selector once: the real bundle's 11 record checks select the same record, which repeated the rows 11 times and pushed the phone view past its row limit. Co-Authored-By: Claude Opus 5.5 --- .../openemr_api_tier/approval_durable.py | 366 ++++++++++++++++++ .../flow/openadapt_flow/runtime/replayer.py | 14 +- packages/flow/tests/test_api_step_approval.py | 27 ++ 3 files changed, 402 insertions(+), 5 deletions(-) create mode 100644 packages/flow/benchmark/openemr_api_tier/approval_durable.py diff --git a/packages/flow/benchmark/openemr_api_tier/approval_durable.py b/packages/flow/benchmark/openemr_api_tier/approval_durable.py new file mode 100644 index 000000000..9094af0d8 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/approval_durable.py @@ -0,0 +1,366 @@ +"""The OpenEMR referral with the API step under each_item approval and a durable run. + +Label every number from this harness exactly as: "API step under each_item +approval + durable, no run gate; harness answers approvals; approval wait +excluded". It is not the governed production profile. The referral bundle +doesn't pass the run gate yet (GUI-path admission; see docs/API_TIER.md and +the admission list in the session notes), so this measures the parts that +do run: the person approving the exact request (docs/APPROVAL_MODES.md) and +the durable checkpoints (docs/API_TIER.md "In a durable run"). + +- Arms: ``browser`` (API steps off) and ``api`` (API steps on), both under + ``ApprovalGate("each_item")`` with ``durable=True``. Interleaved blocks; each + block resets the fixture and seeds the chart r6 must not duplicate. +- Approvals: the harness stands in for the person and answers each request + as soon as it is written (``decided_by="human"``, route ``desktop_run_it``, + approver ``harness-approver``). Time spent waiting for the answer is + measured and subtracted: ``engine_s`` excludes it, ``approval_wait_s`` is it. +- Read-back: the test setup's own direct SQL check (``common.record_check``), + never the arm under test. r6 is right when no chart is created. +- Load: each item waits until the 1-minute load is under GATE_LOAD1 (12). + + PYTHONPATH=packages/flow python -m benchmark.openemr_api_tier.approval_durable \\ + compile --recording --out + PYTHONPATH=packages/flow python -m benchmark.openemr_api_tier.approval_durable \\ + run --bundle --out [--blocks 5 --per-block 4] +""" + +from __future__ import annotations + +import argparse +import csv +import json +import shutil +import statistics +import subprocess +import sys +import time +from pathlib import Path +from typing import Any + +from benchmark.openemr_api_tier.common import ( + REFERRALS, + load_demo, + record_check, + reset_and_seed, + spec_for, + sql_patients, +) +from benchmark.openemr_api_tier.feature_e2e import NO_MATCH_TEXT, _load1, wait_quiet +from benchmark.openemr_api_tier.spike_fixture import SpikeFixture + +LABEL = ( + "API step under each_item approval + durable, no run gate; harness answers " + "approvals; approval wait excluded" +) +APPROVER = "harness-approver" +COLS = [ + "arm", + "block", + "pos", + "ref", + "mismatch", + "correct", + "readback", + "new_charts", + "engine_success", + "transaction_outcome", + "actuation", + "api_fallback", + "approvals", + "approval_kinds", + "approval_wait_s", + "engine_s", + "wall_s", + "checkpoints", + "error", + "load_start", + "load_end", + "gate_waited_s", + "engine_sha", +] + + +def cmd_compile(a: argparse.Namespace) -> None: + """The harness compile (feature_e2e), admitted with a right-record parameter.""" + + from openadapt_flow.compiler.api_proposal import ( + admit_api_step, + describe_api_step, + propose_api_step, + ) + from openadapt_flow.ir import Postcondition, PostconditionKind, Workflow + + demo = load_demo() + rec, bundle = Path(a.recording), Path(a.out) + demo.compile_recording(rec, bundle) + wf = Workflow.load(bundle) + dialog = [ + s + for s in wf.steps + if s.anchor is not None + and s.anchor.structural is not None + and s.anchor.structural.selector == "#create" + ] + assert len(dialog) == 1 + dialog[0].expect = list(dialog[0].expect) + [ + Postcondition( + kind=PostconditionKind.TEXT_PRESENT, text=NO_MATCH_TEXT, timeout_s=10.0 + ) + ] + wf, step, reasons = propose_api_step(wf, rec, replace=True) + if step is None: + raise SystemExit(f"no API step: {reasons}") + # The harness admits under the fixture's name so the matrix runs + # unattended; in use a person admits with `openadapt-flow api admit`. + wf, _ = admit_api_step( + wf, admitted_by="benchmark-fixture", identity={"subject_name": "lname"} + ) + wf.save(bundle) + print(describe_api_step(next(s for s in wf.steps if s.id == step.id))) + + +class HarnessPerson: + """Answers every waiting approval request at once, and times the wait.""" + + def __init__(self, store: Any) -> None: + self.store = store + self.waited_s = 0.0 + self.answered: list[str] = [] + + def sleep(self, seconds: float) -> None: + from openadapt_flow.approval_mode.item_approval import make_answer + + t0 = time.monotonic() + for request in self.store.requests(): + if request.request_id in self.answered: + continue + if self.store.status(request) != "waiting": + continue + self.store.write_answer( + make_answer( + request, + decision="approve", + decided_by="human", + route="desktop_run_it", + approver_id=APPROVER, + auth_method="cloud_aal2", + ) + ) + self.answered.append(request.request_id) + time.sleep(min(seconds, 0.02)) + self.waited_s += time.monotonic() - t0 + + +def run_one( + env: dict[str, Any], arm: str, ref_idx: int, bundle: Path, out: Path +) -> dict[str, Any]: + from openadapt_flow.approval_mode.gate import ApprovalGate + from openadapt_flow.approval_mode.item_approval import ItemApprovalStore + from openadapt_flow.benchmark.openemr_local import OpenEMRPatientOracle + from openadapt_flow.ir import Workflow + from openadapt_flow.runtime import Replayer + from openadapt_flow.runtime.durable.checkpoint import CheckpointStore + + demo, f, browser, ocr = env["demo"], env["f"], env["browser"], env["ocr"] + ref = REFERRALS[ref_idx - 1] + spec = spec_for(ref) + out.mkdir(parents=True) + engine_dir = out / "engine" + before_ids = {r["id"] for r in sql_patients(f)} + oracle = OpenEMRPatientOracle(f.api_base_url, f.token_session("oracle"), spec) + res: dict[str, Any] = { + "arm": arm, + "ref": ref.key, + "mismatch": ref.mismatch, + "error": None, + } + store = ItemApprovalStore(engine_dir) + person = HarnessPerson(store) + gate = ApprovalGate( + "each_item", + approval_store=store, + approval_wait_s=120.0, + approval_poll_s=0.05, + sleep=person.sleep, + monotonic=time.monotonic, + ) + t0 = time.time() + ctx = browser.new_context( + viewport={"width": 1280, "height": 800}, device_scale_factor=1 + ) + try: + page = ctx.new_page() + page.goto(f.ui_base_url) + backend = demo._BasePlaywrightBackend(page) + demo._preauthenticate_browser(backend, f) + t_setup = time.time() + ocr.clear_ocr_memo() + rep = Replayer( + backend, + effect_verifier=oracle.verifier, + api_steps=(arm == "api"), + approval_gate=gate, + durable=True, + ).run( + Workflow.load(bundle), + params=spec.params(), + bundle_dir=bundle, + run_dir=engine_dir, + ) + t_task = time.time() + raw = rep.model_dump(mode="json") + last = raw["results"][-1] if raw["results"] else {} + res.update( + engine_success=raw.get("success"), + transaction_outcome=raw.get("transaction_outcome"), + actuation=last.get("actuation"), + api_fallback=next( + (r["api_fallback"] for r in raw["results"] if r.get("api_fallback")), + None, + ), + step_errors=[ + (r["step_id"], (r.get("error") or "")[:300]) + for r in raw["results"] + if r.get("error") + ], + ) + except Exception as exc: # noqa: BLE001 + t_setup = locals().get("t_setup", time.time()) + t_task = time.time() + res["error"] = f"{type(exc).__name__}: {exc}"[:500] + time.sleep(demo.ORACLE_SETTLE_S) + res.update(record_check(f, ref, before_ids)) + t_end = time.time() + ctx.close() + requests = store.requests() + res.update( + approvals=len(requests), + approval_kinds="+".join(r.form_state_kind for r in requests), + approval_wait_s=round(person.waited_s, 3), + engine_s=round(t_task - t_setup - person.waited_s, 3), + wall_s=round(t_end - t0, 3), + checkpoints=len(CheckpointStore(engine_dir).checkpoints()), + ) + (out / "result.json").write_text(json.dumps(res, indent=1, default=str)) + shutil.rmtree(engine_dir / "steps", ignore_errors=True) + return res + + +def cmd_run(a: argparse.Namespace) -> None: + from playwright.sync_api import sync_playwright + + import openadapt_flow.vision.ocr # noqa: F401 + + ocr = sys.modules["openadapt_flow.vision.ocr"] + out, bundle = Path(a.out), Path(a.bundle) + out.mkdir(parents=True, exist_ok=True) + sha = subprocess.run( + ["git", "rev-parse", "HEAD"], capture_output=True, text=True + ).stdout.strip()[:9] + demo, f = load_demo(), SpikeFixture() + pw = sync_playwright().start() + browser = pw.chromium.launch(headless=True) + ocr._get_engine() + env = {"demo": demo, "f": f, "browser": browser, "ocr": ocr} + csv_path = out / "runs.csv" + rows: list[dict[str, Any]] = [] + try: + for b in range(a.blocks): + for arm in ["browser", "api"] if b % 2 == 0 else ["api", "browser"]: + reset_and_seed(f) + for pos in range(a.per_block): + idx = ((b * a.per_block + pos) % len(REFERRALS)) + 1 + gate = wait_quiet() + load_start = _load1() + res = run_one( + env, + arm, + idx, + bundle, + out + / "items" + / f"b{b + 1:02d}-{arm}-{pos + 1}-{REFERRALS[idx - 1].key}", + ) + row = {k: res.get(k) for k in COLS} | { + "block": b + 1, + "pos": pos + 1, + "load_start": load_start, + "load_end": _load1(), + "gate_waited_s": gate["gate_waited_s"], + "engine_sha": sha, + } + rows.append(row) + print( + json.dumps( + { + k: row[k] + for k in ( + "arm", + "block", + "ref", + "correct", + "actuation", + "approvals", + "engine_s", + "approval_wait_s", + "error", + ) + } + ), + flush=True, + ) + finally: + browser.close() + pw.stop() + with open(csv_path, "w", newline="") as fh: + w = csv.DictWriter(fh, fieldnames=COLS) + w.writeheader() + w.writerows(rows) + (out / "summary.json").write_text( + json.dumps(summarize(rows, sha), indent=1) + "\n" + ) + + +def summarize(rows: list[dict[str, Any]], sha: str) -> dict[str, Any]: + """Per arm: correctness by read-back, engine time, approvals.""" + + out: dict[str, Any] = {"label": LABEL, "engine_sha": sha, "arms": {}} + for arm in ("browser", "api"): + mine = [r for r in rows if r["arm"] == arm] + if not mine: + continue + times = [r["engine_s"] for r in mine if r["engine_s"] is not None] + out["arms"][arm] = { + "runs": len(mine), + "correct": sum(bool(r["correct"]) for r in mine), + "errors": sum(bool(r["error"]) for r in mine), + "api_saves": sum(r["actuation"] == "api" for r in mine), + "approvals_per_item": sorted({r["approvals"] for r in mine}), + "approval_kinds": sorted({r["approval_kinds"] for r in mine}), + "median_engine_s": round(statistics.median(times), 2) if times else None, + "median_approval_wait_s": round( + statistics.median(r["approval_wait_s"] for r in mine), 3 + ), + "max_load_start": max(r["load_start"] for r in mine), + } + return out + + +def main() -> None: + ap = argparse.ArgumentParser() + sub = ap.add_subparsers(dest="cmd", required=True) + c = sub.add_parser("compile") + c.add_argument("--recording", required=True) + c.add_argument("--out", required=True) + r = sub.add_parser("run") + r.add_argument("--bundle", required=True) + r.add_argument("--out", required=True) + r.add_argument("--blocks", type=int, default=5) + r.add_argument("--per-block", type=int, default=4) + a = ap.parse_args() + {"compile": cmd_compile, "run": cmd_run}[a.cmd](a) + + +if __name__ == "__main__": + main() diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py index 4f3a94f57..0314f4a2b 100644 --- a/packages/flow/openadapt_flow/runtime/replayer.py +++ b/packages/flow/openadapt_flow/runtime/replayer.py @@ -11778,11 +11778,15 @@ def _web_form_approval_view( if isinstance(prepared, ApiActuationResult): return prepared.reason or "the API step is unavailable" effects = self._resolve_effects(list(binding.effects), params) - record = [ - (key, str(expr)) - for effect in effects - for key, expr in sorted(effect.match.items()) - ] + # Several record checks usually select the same record (one per + # field): the person reads, and the digest binds, each pair once. + record = list( + dict.fromkeys( + (key, str(expr)) + for effect in effects + for key, expr in sorted(effect.match.items()) + ) + ) digest = operation_digest( prepared, record_check=[effect.contract_hash() for effect in effects], diff --git a/packages/flow/tests/test_api_step_approval.py b/packages/flow/tests/test_api_step_approval.py index 98f69e5e7..61a4a1893 100644 --- a/packages/flow/tests/test_api_step_approval.py +++ b/packages/flow/tests/test_api_step_approval.py @@ -315,3 +315,30 @@ def test_operation_digest_binds_the_record_check(): ) assert base != operation_digest(changed, record_check=["h1"], record=[("id", "p1")]) assert up.parse_qs(prepared.encoded()) == {"lname": ["Marsh"]} + + +def test_the_record_rows_are_listed_once_per_field(tmp_path): + from openadapt_flow.runtime.effects import Effect, EffectKind, ValueExpr + + wf = _workflow() + save = wf.steps[-1] + match = {"patient_id": ValueExpr(param="lname")} + effects = [ + Effect(kind=EffectKind.RECORD_WRITTEN, match=match, expected_count=1), + Effect( + kind=EffectKind.FIELD_EQUALS, + match=match, + field="note", + value=ValueExpr(literal="Referred"), + ), + ] + binding = save.api_binding.model_copy(update={"effects": effects}) + save = save.model_copy(update={"effects": effects, "api_binding": binding}) + replayer = Replayer( + EnterWritesBackend("http://127.0.0.1:9"), + vision=FakeVision(), + web_form_actuator=WebFormActuator(SessionTransport("http://127.0.0.1:9")), + ) + view = replayer._web_form_approval_view(save, PARAMS) + record_rows = [row for row in view.rows if row[0].startswith("Record check:")] + assert record_rows == [("Record check: patient_id", "Marsh")] From 59c7d287fb4ee65deac6581a25d4097d282889b9 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 13:50:34 -0400 Subject: [PATCH 04/13] flow (proposal): classify common local edits from the recording Draft for the founder; not for merging until decided (docs/design/LOCAL_EDIT_CLASSIFICATION.md). Co-Authored-By: Claude Opus 5.5 --- .../docs/design/LOCAL_EDIT_CLASSIFICATION.md | 102 ++++++ .../backends/playwright_backend.py | 22 ++ .../flow/openadapt_flow/compiler/compile.py | 24 ++ .../openadapt_flow/compiler/local_edit.py | 252 ++++++++++++++ packages/flow/openadapt_flow/ir.py | 26 ++ packages/flow/openadapt_flow/policy.py | 7 +- packages/flow/openadapt_flow/risk.py | 30 ++ .../flow/openadapt_flow/runtime/replayer.py | 60 ++++ .../tests/test_local_edit_classification.py | 317 ++++++++++++++++++ 9 files changed, 838 insertions(+), 2 deletions(-) create mode 100644 packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md create mode 100644 packages/flow/openadapt_flow/compiler/local_edit.py create mode 100644 packages/flow/tests/test_local_edit_classification.py diff --git a/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md b/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md new file mode 100644 index 000000000..036956b04 --- /dev/null +++ b/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md @@ -0,0 +1,102 @@ +# Classifying local edits without a person (proposal) + +Status: proposal. Not merged; the founder decides. + +## The problem + +A compiled program flags every special key and every unlabelled or +navigation-shaped click for review (`risk_review_required`), because its +risk can't be proven from the step alone. The run gate then refuses the +bundle until a person classifies each one. On the OpenEMR referral that is 12 +steps (blocker section A in the referral admission list): + +- 8 `Control+a` presses, each in a text field right before typing into it; +- 4 `Enter` presses, each right after typing an option's label into a + dropdown; + +plus, under `clinical-write`, the same keys fail `prohibit_vacuous_postconditions` +because a key has no outcome check (section B), and the Contact tab click is +unarmed (section C). + +These are the same few patterns in every form-filling program, on every app. +A person approving each one, per bundle, is busywork that trains people to +approve without reading. + +## The proposal + +The compiler classifies three local-edit patterns itself, and only when the +recording shows the action stayed local. Each classified step gets an +automatic outcome check that halts the run if, at run time, the action did +anything else. + +| Pattern | Evidence the compiler needs (all of it) | Outcome check added to the step | +| --- | --- | --- | +| Select-all before typing: `Control+a` / `Meta+a` | The previous step clicked a single text field (structural role `textbox`, `searchbox` or `spinbutton`); the next step types into it; URL, title and page count unchanged across the key; no write request during the key (when the recording has network capture) | `local_edit`: after the key, the focused element is still that field, and URL, title and page count are unchanged. The next step's typed-input read-back then checks the field's whole value. | +| Confirm a typed option: `Enter` | The previous steps clicked a dropdown (role `combobox` or `listbox`) and typed an option's label (a parameter or a literal); URL, title and page count unchanged; no write request during the key (network capture **required**: `Enter` is a submission key) | `local_edit` as above, plus `field_value`: the focused dropdown shows the typed label (the run's parameter value). | +| In-form tab: a click on a tab | Role `tab`, or a `button` whose recorded click changed only the visible panel: URL, title and page count unchanged, no write request (network capture required), and the next step acts in the same frame | `local_edit` (URL, title and page count unchanged; focus need not stay). | + +What stays the same: + +- **The risk rule.** A step whose evidence is missing or mixed (a URL change, + a write request in the window, no network capture for a submission key, a + key that isn't one of these, a field role the compiler can't read) keeps + `risk_review_required` and goes to a person, exactly as today. +- **Write-shaped context still wins.** A key or click whose retained text + names a consequential write (`submit`, `save`, `delete`, ...) classifies + irreversible before any of this runs. +- **Explicit `risk_overrides` still win.** +- **D, the duplicate-prevention (idempotency) key on the save, is untouched.** + That is a business decision. + +What the run gate sees: the classified step is reversible, reviewed by the +compiler with recorded evidence (`risk_explanation` names the pattern and the +evidence), and has an outcome check, so `require_reviewed_action_risk`, +`prohibit_vacuous_postconditions` and `max_unverified_steps` pass for it. A +tab click classified this way is in-form navigation that writes nothing, so it +isn't consequential and isn't entity navigation; `prohibit_unarmed_clicks` +under `clinical-write` still applies to it, so it still needs an identity +check unless that policy exempts `local_edit` clicks (an open question below). + +## Why this is safe enough + +- The classification is backed by what the demonstration did, not by the + label. A key that submitted the form in the recording (a POST, a URL change) + is never classified. +- The outcome check runs every time. If the same key does something else at + run time (the app changed, a different field had focus), the step fails its + postcondition and the run halts at that step. The halt comes after the key + was pressed, which is the residual risk: a key that, at run time only, + commits something. For the three patterns that requires the app to have + changed the field's behavior since the recording, and the first write it + makes is still caught by the save's record check or by the next step's + checks. +- Evidence is sealed in the bundle (the step's explanation and postconditions + are part of the content digest), so a later edit to a classification + changes the digest and needs a new admission. + +## Open questions for the founder + +1. Accept the residual risk above for these three patterns? +2. Should `clinical-write`'s `prohibit_unarmed_clicks` exempt a `local_edit` + tab click (it writes nothing and stays in the form), or must a person or the + compiler still arm it with an identity check? Today it must be armed. +3. Without network capture, `Control+a` is classified on the structural and + URL evidence alone (select-all in a focused text field isn't a submission + key). `Enter` and tab clicks need network capture. Is that the right line? + +## Implementation sketch + +- `openadapt_flow.risk.local_edit_classifications(steps, events, network)`: + the three rules, returning `{step_id: (explanation, evidence)}`. +- `compile_recording` applies them after the per-step inference and before + `risk_overrides`, and adds the outcome checks. +- Two postcondition kinds: `local_edit` (structural: focus and URL, title, + page count) and `field_value` (the focused dropdown's label equals a run + parameter). Both are evaluated by the runtime like the existing structural + kinds, and both refuse rather than pass when the backend can't observe them + (unlike `url_changed`, which passes unverified). +- Qualification's draft classification reads the same evidence, so a + compiled `local_edit` step drafts as `reversible` instead of `unknown`. +- Tests: each pattern classifies with full evidence; each missing piece of + evidence leaves the step for a person; each outcome check halts when the + action did something else (focus moved, URL changed, wrong option). diff --git a/packages/flow/openadapt_flow/backends/playwright_backend.py b/packages/flow/openadapt_flow/backends/playwright_backend.py index a9bad239c..ed7589a1b 100644 --- a/packages/flow/openadapt_flow/backends/playwright_backend.py +++ b/packages/flow/openadapt_flow/backends/playwright_backend.py @@ -1665,6 +1665,28 @@ def focused_text_value(self) -> Optional[str]: return self._focused_element_eval(_EDITABLE_VALUE_JS) + def focused_element_matches(self, selector: str) -> Optional[bool]: + """Whether the focused element (through frames) matches ``selector``. + + None when it can't be read. Used by the ``local_edit`` postcondition. + """ + + result = self._focused_element_call( + "(el, sel) => { try { return el.matches(sel); } catch (e) { return null; } }", + arg=selector, + with_arg=True, + ) + return result if isinstance(result, bool) else None + + def focused_select_label(self) -> Optional[str]: + """The focused `` + + + + +""" + + +def _controls(): + return form_controls(NATIVE, "/save.php") + + +@pytest.mark.parametrize( + "name, value, problem", + [ + ("code", "", "required"), + ("code", "abc-12", "pattern"), + ("code", "ABC-123", "maxlength"), + ("email", "not-an-address", "email"), + ("dose", "11", "max"), + ("dose", "0", "min"), + ("dose", "ten", "number"), + ("seen", "2026-10-20", "max"), + ("seen", "10/19/2026", "date"), + ], +) +def test_native_constraints_refuse_what_the_browser_would(name, value, problem): + found = native_problem(name, _controls()[name], value) + assert found is not None and problem in found + + +@pytest.mark.parametrize( + "name, value", + [ + ("code", "ABC-12"), + ("email", "a@b.example"), + ("dose", "5"), + ("seen", "2026-10-01"), + ("h", ""), + ], +) +def test_native_constraints_accept_valid_values(name, value): + assert native_problem(name, _controls()[name], value) is None + + +def test_validate_js_length_format_and_numericality_are_enforced(): + rule = {"length": {"maximum": 3}, "format": {"pattern": "[a-z]+"}} + assert "length" in client_rule_problem("f", rule, "abcd") + assert "format" in client_rule_problem("f", rule, "AB") + assert client_rule_problem("f", {"numericality": True}, "x") is not None + assert client_rule_problem("f", rule, "ab") is None + + +def test_the_actuator_refuses_a_value_the_browser_would_refuse(tmp_path): + spec = WebFormSpec( + form_path="/new.php", + fields=[WebFormField(name="code", source="param", param="code")], + controls=["code", "dose", "email", "h", "seen"], + replaces_steps=["s0"], + folded_checks_digest="sha256:" + "0" * 64, + ) + body, problem = WebFormActuator.build_body(spec, {"code": "abc"}, _controls()) + assert body == [] and "pattern" in problem + + +def _rec_with_form(tmp_path, html): + rec = _recording(tmp_path) + rows = [json.loads(x) for x in (rec / nc.NETWORK_FILE).read_text().splitlines()] + for row in rows[1:]: + if "document.forms[0].addEventListener('submit', v)" + ), + "listening for submit", + ), + ( + lambda h: h.replace( + '{"form_lname":{"presence":true}}', + '{"form_lname":{"presence":true,"pastDate":{}}}', + ), + "form_lname pastDate", + ), + ], +) +def test_script_the_api_step_cant_run_keeps_the_save_on_the_gui(tmp_path, edit, named): + html = edit(FORM_HTML) + _wf, step, reasons = propose_api_step(_workflow(), _rec_with_form(tmp_path, html)) + assert step is None and named in " ".join(reasons) + # The same script on the live form makes an admitted API step unavailable. + live = edit(FORM_HTML.replace("", "live-session-value")) + (tmp_path / "admitted").mkdir() + out = WebFormActuator(FakeTransport(form=live)).actuate( + _admitted_binding(tmp_path / "admitted"), + {"lname": "Marsh", "state_label": "California", "note": "Referred"}, + ) + assert out.status is ActuationStatus.UNAVAILABLE + # A rule kind the recording didn't have is first a changed rule set. + assert named in out.reason or "rules changed" in out.reason From 1163d8dce226915d74f18e9bfe999a42e4802db3 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 14:23:58 -0400 Subject: [PATCH 07/13] fix(flow): bind an API step to the complete definition of the steps it replaces Review of #1269: folded_checks_digest hashed only the folded steps' on-screen checks, so changing what a replaced step does (its action, value, parameter, target or identity, a guard, the save itself) left the admitted API step running in its place. The digest now covers every replaced step's complete definition, the save included, minus only the save's own API binding. Any change keeps the API step on the GUI with the reason until it is proposed and admitted again. Bindings proposed before this need proposing again (they fail closed). Co-Authored-By: Claude Opus 5.5 --- packages/flow/docs/API_TIER.md | 8 ++- .../openadapt_flow/compiler/api_proposal.py | 2 +- packages/flow/openadapt_flow/ir.py | 9 ++- .../runtime/actuators/web_form.py | 12 +++- .../flow/openadapt_flow/runtime/replayer.py | 7 +- .../test_api_step_replaced_steps_digest.py | 70 +++++++++++++++++++ packages/flow/tests/test_api_tier_web_form.py | 4 +- 7 files changed, 98 insertions(+), 14 deletions(-) create mode 100644 packages/flow/tests/test_api_step_replaced_steps_digest.py diff --git a/packages/flow/docs/API_TIER.md b/packages/flow/docs/API_TIER.md index 38242de35..915831072 100644 --- a/packages/flow/docs/API_TIER.md +++ b/packages/flow/docs/API_TIER.md @@ -38,9 +38,11 @@ openadapt-flow run bundle/ --approval-mode not_gated ... first of those steps carries `api_fallback` with the reason, and the GUI steps run as recorded. -If you change a check on one of the covered steps after compiling (for -example, you add a required on-screen text), the API step stops running -because it would skip that check. Run `openadapt-flow api propose bundle/ +If you change any of the covered steps after compiling (an action, a typed +value or parameter, a target or its identity evidence, a guard, an on-screen +check, or the save's record check), the API step stops running: its +admission is bound to a digest of the complete definition of every step it +replaces, the save included. Run `openadapt-flow api propose bundle/ --recording rec/` to propose it again, then admit it. ## What the compiler checks diff --git a/packages/flow/openadapt_flow/compiler/api_proposal.py b/packages/flow/openadapt_flow/compiler/api_proposal.py index 97a89ce08..260d3166f 100644 --- a/packages/flow/openadapt_flow/compiler/api_proposal.py +++ b/packages/flow/openadapt_flow/compiler/api_proposal.py @@ -612,7 +612,7 @@ def carried(line: dict[str, Any]) -> int: replaces_steps=[s.id for s in folded], other_writes_seen=others, screen_only_checks=screen_only, - folded_checks_digest=folded_checks_digest(folded[:-1]), + folded_checks_digest=folded_checks_digest(folded), ) binding = ApiBinding( kind="web_form", diff --git a/packages/flow/openadapt_flow/ir.py b/packages/flow/openadapt_flow/ir.py index ddae6a1ca..d5621334f 100644 --- a/packages/flow/openadapt_flow/ir.py +++ b/packages/flow/openadapt_flow/ir.py @@ -612,9 +612,12 @@ class WebFormSpec(BaseModel): #: On-screen checks of folded steps that caused no request (screen-only, #: not repeated), listed for the person admitting the step. screen_only_checks: list[str] = Field(default_factory=list) - #: Digest of the folded steps' on-screen checks when this was proposed. - #: If a check is added or changed later, the API step does not run (it - #: would skip that check) until it is proposed again. + #: Digest of the complete definition of every replaced step, the save + #: included (minus its API binding), when this was proposed + #: (``folded_checks_digest`` in runtime.actuators.web_form). Any change to + #: those steps keeps the API step from running until it is proposed and + #: admitted again. Bindings proposed before it covered more than the + #: on-screen checks need proposing again. folded_checks_digest: str = Field(pattern=r"^sha256:[0-9a-f]{64}$") diff --git a/packages/flow/openadapt_flow/runtime/actuators/web_form.py b/packages/flow/openadapt_flow/runtime/actuators/web_form.py index 27915c01a..9fd991210 100644 --- a/packages/flow/openadapt_flow/runtime/actuators/web_form.py +++ b/packages/flow/openadapt_flow/runtime/actuators/web_form.py @@ -345,11 +345,19 @@ def option_for_label(control: dict[str, Any], label: str) -> Optional[str]: def folded_checks_digest(steps: Any) -> str: - """SHA-256 over the on-screen checks (``expect``) of the folded steps.""" + """SHA-256 over the complete definition of the steps an API step replaces. + + Every replaced step, the save included: its action and values, its + parameters, its target and identity evidence, its guards and waits, its + on-screen checks, its record check and its risk. Only the save's own API + binding is left out (it carries this digest). Any change to what the GUI + steps do or check changes the digest, and the API step doesn't run until + it is proposed and admitted again. + """ import hashlib - payload = [[s.id, [p.model_dump(mode="json") for p in s.expect]] for s in steps] + payload = [s.model_dump(mode="json", exclude={"api_binding"}) for s in steps] canonical = json.dumps(payload, sort_keys=True, separators=(",", ":")) return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest() diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py index 7f26df4fc..3d68fd123 100644 --- a/packages/flow/openadapt_flow/runtime/replayer.py +++ b/packages/flow/openadapt_flow/runtime/replayer.py @@ -2436,12 +2436,13 @@ def _web_form_span_at( elif ids != list(binding.web_form.replaces_steps): reason = "the program's steps no longer match the API step" elif ( - folded_checks_digest(workflow.steps[step_index:end_index]) + folded_checks_digest(workflow.steps[step_index : end_index + 1]) != binding.web_form.folded_checks_digest ): reason = ( - "an on-screen check of the GUI steps changed after the API " - "step was proposed, and the API step would skip it " + "the steps the API step replaces changed after it was " + "proposed (an action, value, target, check, guard or record " + "check), so the API step no longer stands for them " "(openadapt-flow api propose)" ) elif self._actuator_for(binding) is None: diff --git a/packages/flow/tests/test_api_step_replaced_steps_digest.py b/packages/flow/tests/test_api_step_replaced_steps_digest.py new file mode 100644 index 000000000..b80ac338e --- /dev/null +++ b/packages/flow/tests/test_api_step_replaced_steps_digest.py @@ -0,0 +1,70 @@ +"""An API step stands for the exact steps it was admitted for. + +The reviewer's case for #1269: folded_checks_digest hashed only the folded +steps' on-screen checks, so changing what a replaced step does (its action, +its value, its parameter, its target or identity, a guard, the save's record +check) left the admitted API step running in its place. +""" + +from __future__ import annotations + +import pytest + +from openadapt_flow.ir import ActionKind, Anchor, Guard, Predicate, PredicateKind +from openadapt_flow.runtime.replayer import Replayer +from tests.test_api_tier_web_form import _span_workflow +from tests.test_replayer import FakeBackend, FakeVision + + +def _edit(wf, index, **update): + steps = list(wf.steps) + steps[index] = steps[index].model_copy(update=update) + return wf.model_copy(update={"steps": steps}) + + +def _why(wf): + replayer = Replayer(FakeBackend(), vision=FakeVision(), web_form_actuator=object()) + _end, _save, why = replayer._web_form_span_at(wf, 0, durable=False) + return why + + +def test_the_admitted_steps_run_as_the_api_step(): + assert _why(_span_workflow()) is None + + +@pytest.mark.parametrize( + "index, update", + [ + (0, {"action": ActionKind.CLICK, "key": None}), + (0, {"key": "Shift+Tab"}), + (0, {"text": "{lname}", "action": ActionKind.TYPE, "key": None}), + (0, {"param": "lname"}), + ( + 0, + { + "anchor": Anchor( + template="templates/x.png", + region=(0, 0, 10, 10), + click_point=(5, 5), + context_text="Another record", + ) + }, + ), + ( + 0, + { + "guard": Guard( + predicate=Predicate( + kind=PredicateKind.PARAM_EQUALS, param="lname", value="x" + ), + on_unmet="halt", + ) + }, + ), + (1, {"key": "Ctrl+Enter"}), + (1, {"risk": "irreversible"}), + ], +) +def test_any_change_to_a_replaced_step_keeps_it_on_the_gui(index, update): + why = _why(_edit(_span_workflow(), index, **update)) + assert why is not None and "replaces changed" in why diff --git a/packages/flow/tests/test_api_tier_web_form.py b/packages/flow/tests/test_api_tier_web_form.py index 4e99f9099..75396777e 100644 --- a/packages/flow/tests/test_api_tier_web_form.py +++ b/packages/flow/tests/test_api_tier_web_form.py @@ -561,7 +561,7 @@ def _span_workflow(*, admitted=True) -> Workflow: WebFormPrecheck(path="/dup.php", require_text=["No matches were found."]) ], replaces_steps=["step_000", "step_001"], - folded_checks_digest=folded_checks_digest(steps[:1]), + folded_checks_digest=folded_checks_digest(steps), ) binding = ApiBinding( kind="web_form", url_template="/save.php", effects=[effect], web_form=spec @@ -666,7 +666,7 @@ def test_a_rejected_save_halts_without_gui_retry(tmp_path): } ), {}, - "on-screen check", + "replaces changed", ), ], ) From ac9a17d50b1320e636d6144b6c0bcb1c28086824 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 14:24:28 -0400 Subject: [PATCH 08/13] docs(flow): scope the API step to a single HTML form submission Co-Authored-By: Claude Opus 5.5 --- packages/flow/docs/API_TIER.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/packages/flow/docs/API_TIER.md b/packages/flow/docs/API_TIER.md index 915831072..fe991d5c9 100644 --- a/packages/flow/docs/API_TIER.md +++ b/packages/flow/docs/API_TIER.md @@ -1,9 +1,14 @@ # API steps from a recording's network traffic -A web form save that OpenAdapt can repeat as one HTTP request runs as one -request instead of as the clicks and keystrokes that filled the form. The -right-record check before the save and the record check after it stay the -same. If anything the API step depends on has changed, the recorded GUI +**Scope: a single HTML form submission.** An API step repeats one +`application/x-www-form-urlencoded` POST of one HTML form, read again from +the live page, in place of the clicks and keystrokes that filled it. It is +not a general API client: JSON or script-built requests, more than one +write, multi-page server state, and forms whose script validates or rewrites +values all stay on the GUI. + +The right-record check before the save and the record check after it stay +the same. If anything the API step depends on has changed, the recorded GUI steps run instead. Its speed is measured on the local OpenEMR referral as arm `program_api` of From b983d512c585d9e7588bb35599d1ff7669f0c94d Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 14:50:09 -0400 Subject: [PATCH 09/13] bench(flow): referral API step under each_item approval + durable, 20 runs per arm Label: API step under each_item approval + durable, no run gate; harness answers approvals; approval wait excluded. Not the governed production profile: the referral bundle doesn't pass the run gate yet on its GUI path. On the local OpenEMR test copy (synthetic patients), interleaved blocks: - API step: 20/20 correct by SQL read-back, 18 saved through the API (r6, the possible duplicate, fell back and created nothing, twice), median 1.12 s engine time, 1 approval per item. - API steps off: 19/20 correct plus 1 fail-closed safe stop (nothing saved), median 15.95 s, 2 approvals per new referral. - No wrong or duplicate chart. The run paused for a shared quiet window after 19 runs and resumed into a second directory; approval_durable.py gains --from-block, --skip-done and a summarize command that rebuilds runs.csv and summary.json from the items. Only the CSV, the summary and each item's result.json are committed (the 21 MB of screenshots and engine reports are kept outside the repository). Co-Authored-By: Claude Opus 5.5 --- .../openemr_api_tier/approval_durable.py | 85 ++++++++- .../items/b01-api-1-r1-nadia-brennan.json | 94 ++++++++++ .../items/b01-api-2-r2-marcus-feld.json | 94 ++++++++++ .../items/b01-api-3-r3-ines-carvalho.json | 94 ++++++++++ .../items/b01-api-4-r4-theo-lindqvist.json | 94 ++++++++++ .../items/b01-browser-1-r1-nadia-brennan.json | 94 ++++++++++ .../items/b01-browser-2-r2-marcus-feld.json | 99 +++++++++++ .../items/b01-browser-3-r3-ines-carvalho.json | 94 ++++++++++ .../b01-browser-4-r4-theo-lindqvist.json | 94 ++++++++++ .../items/b02-api-1-r5-grace-adeyemi.json | 94 ++++++++++ .../items/b02-api-2-r6-rosa-delgado.json | 99 +++++++++++ .../items/b02-api-3-r7-arlo-whitcombe.json | 94 ++++++++++ .../items/b02-api-4-r8-priya-raman.json | 94 ++++++++++ .../items/b02-browser-1-r5-grace-adeyemi.json | 94 ++++++++++ .../items/b02-browser-2-r6-rosa-delgado.json | 99 +++++++++++ .../b02-browser-3-r7-arlo-whitcombe.json | 94 ++++++++++ .../items/b02-browser-4-r8-priya-raman.json | 94 ++++++++++ .../items/b03-api-1-r1-nadia-brennan.json | 94 ++++++++++ .../items/b03-api-2-r2-marcus-feld.json | 94 ++++++++++ .../items/b03-api-3-r3-ines-carvalho.json | 94 ++++++++++ .../items/b03-api-4-r4-theo-lindqvist.json | 94 ++++++++++ .../items/b03-browser-1-r1-nadia-brennan.json | 94 ++++++++++ .../items/b03-browser-2-r2-marcus-feld.json | 94 ++++++++++ .../items/b03-browser-3-r3-ines-carvalho.json | 94 ++++++++++ .../b03-browser-4-r4-theo-lindqvist.json | 94 ++++++++++ .../items/b04-api-1-r5-grace-adeyemi.json | 94 ++++++++++ .../items/b04-api-2-r6-rosa-delgado.json | 99 +++++++++++ .../items/b04-api-3-r7-arlo-whitcombe.json | 94 ++++++++++ .../items/b04-api-4-r8-priya-raman.json | 94 ++++++++++ .../items/b04-browser-1-r5-grace-adeyemi.json | 94 ++++++++++ .../items/b04-browser-2-r6-rosa-delgado.json | 99 +++++++++++ .../b04-browser-3-r7-arlo-whitcombe.json | 94 ++++++++++ .../items/b04-browser-4-r8-priya-raman.json | 94 ++++++++++ .../items/b05-api-1-r1-nadia-brennan.json | 94 ++++++++++ .../items/b05-api-2-r2-marcus-feld.json | 94 ++++++++++ .../items/b05-api-3-r3-ines-carvalho.json | 94 ++++++++++ .../items/b05-api-4-r4-theo-lindqvist.json | 94 ++++++++++ .../items/b05-browser-1-r1-nadia-brennan.json | 94 ++++++++++ .../items/b05-browser-2-r2-marcus-feld.json | 94 ++++++++++ .../items/b05-browser-3-r3-ines-carvalho.json | 94 ++++++++++ .../b05-browser-4-r4-theo-lindqvist.json | 94 ++++++++++ .../2026-10-19-combined/runs.csv | 41 +++++ .../2026-10-19-combined/summary.json | 43 +++++ .../results/approval_durable/README.md | 55 ++++++ packages/flow/docs/API_TIER.md | 6 + packages/flow/public-artifacts.json | 168 ++++++++++++++++++ 46 files changed, 4180 insertions(+), 3 deletions(-) create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-1-r1-nadia-brennan.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-2-r2-marcus-feld.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-3-r3-ines-carvalho.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-4-r4-theo-lindqvist.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-1-r1-nadia-brennan.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-2-r2-marcus-feld.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-3-r3-ines-carvalho.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-4-r4-theo-lindqvist.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-1-r5-grace-adeyemi.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-2-r6-rosa-delgado.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-3-r7-arlo-whitcombe.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-4-r8-priya-raman.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-1-r5-grace-adeyemi.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-2-r6-rosa-delgado.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-3-r7-arlo-whitcombe.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-4-r8-priya-raman.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-1-r1-nadia-brennan.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-2-r2-marcus-feld.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-3-r3-ines-carvalho.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-4-r4-theo-lindqvist.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-1-r1-nadia-brennan.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-2-r2-marcus-feld.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-3-r3-ines-carvalho.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-4-r4-theo-lindqvist.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-1-r5-grace-adeyemi.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-2-r6-rosa-delgado.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-3-r7-arlo-whitcombe.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-4-r8-priya-raman.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-1-r5-grace-adeyemi.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-2-r6-rosa-delgado.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-3-r7-arlo-whitcombe.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-4-r8-priya-raman.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-1-r1-nadia-brennan.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-2-r2-marcus-feld.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-3-r3-ines-carvalho.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-4-r4-theo-lindqvist.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-1-r1-nadia-brennan.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-2-r2-marcus-feld.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-3-r3-ines-carvalho.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-4-r4-theo-lindqvist.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/runs.csv create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/summary.json create mode 100644 packages/flow/benchmark/openemr_api_tier/results/approval_durable/README.md diff --git a/packages/flow/benchmark/openemr_api_tier/approval_durable.py b/packages/flow/benchmark/openemr_api_tier/approval_durable.py index 9094af0d8..7b3e57f92 100644 --- a/packages/flow/benchmark/openemr_api_tier/approval_durable.py +++ b/packages/flow/benchmark/openemr_api_tier/approval_durable.py @@ -265,12 +265,30 @@ def cmd_run(a: argparse.Namespace) -> None: env = {"demo": demo, "f": f, "browser": browser, "ocr": ocr} csv_path = out / "runs.csv" rows: list[dict[str, Any]] = [] + done = ( + { + d.name + for d in (Path(a.skip_done) / "items").iterdir() + if (d / "result.json").is_file() + } + if a.skip_done + else set() + ) try: - for b in range(a.blocks): + for b in range(a.from_block - 1, a.blocks): for arm in ["browser", "api"] if b % 2 == 0 else ["api", "browser"]: + names = [ + f"b{b + 1:02d}-{arm}-{pos + 1}-" + f"{REFERRALS[(b * a.per_block + pos) % len(REFERRALS)].key}" + for pos in range(a.per_block) + ] + if all(n in done for n in names): + continue reset_and_seed(f) for pos in range(a.per_block): idx = ((b * a.per_block + pos) % len(REFERRALS)) + 1 + if names[pos] in done: + continue gate = wait_quiet() load_start = _load1() res = run_one( @@ -342,14 +360,71 @@ def summarize(rows: list[dict[str, Any]], sha: str) -> dict[str, Any]: "median_approval_wait_s": round( statistics.median(r["approval_wait_s"] for r in mine), 3 ), - "max_load_start": max(r["load_start"] for r in mine), + "max_load_start": max( + (r["load_start"] for r in mine if r["load_start"] is not None), + default=None, + ), + # Stopped before saving, nothing written (a fail-closed halt or a + # refused duplicate): not a wrong save. + "safe_stops": sum( + not r["correct"] and r.get("new_charts") == 0 for r in mine + ), + "wrong_saves": sum( + not r["correct"] and (r.get("new_charts") or 0) > 0 for r in mine + ), } return out +def rows_from_items(out: Path, sha: str) -> list[dict[str, Any]]: + """Rebuild the rows from each item's result.json (an interrupted run).""" + + rows = [] + for item in sorted((out / "items").iterdir()): + result = item / "result.json" + if not result.is_file(): + continue # interrupted before the item finished + res = json.loads(result.read_text()) + block, arm, pos = item.name.split("-")[:3] + rows.append( + {k: res.get(k) for k in COLS} + | { + "block": int(block[1:]), + "pos": int(pos), + "load_start": None, + "load_end": None, + "gate_waited_s": None, + "engine_sha": sha, + } + ) + return rows + + +def cmd_summarize(a: argparse.Namespace) -> None: + """Combine the finished items of one or more --out directories into --to.""" + + out = Path(a.to) + out.mkdir(parents=True, exist_ok=True) + rows = [row for d in a.out for row in rows_from_items(Path(d), a.engine_sha)] + with open(out / "runs.csv", "w", newline="") as fh: + w = csv.DictWriter(fh, fieldnames=COLS) + w.writeheader() + w.writerows(rows) + (out / "summary.json").write_text( + json.dumps(summarize(rows, a.engine_sha), indent=1) + "\n" + ) + print(json.dumps(summarize(rows, a.engine_sha), indent=1)) + + def main() -> None: ap = argparse.ArgumentParser() sub = ap.add_subparsers(dest="cmd", required=True) + sm = sub.add_parser( + "summarize", help="Rebuild runs.csv and summary.json from items/" + ) + sm.add_argument("--out", required=True, action="append") + sm.add_argument("--to", required=True) + sm.add_argument("--engine-sha", required=True) c = sub.add_parser("compile") c.add_argument("--recording", required=True) c.add_argument("--out", required=True) @@ -358,8 +433,12 @@ def main() -> None: r.add_argument("--out", required=True) r.add_argument("--blocks", type=int, default=5) r.add_argument("--per-block", type=int, default=4) + r.add_argument("--from-block", type=int, default=1) + r.add_argument( + "--skip-done", default=None, help="An earlier --out; skip its finished items" + ) a = ap.parse_args() - {"compile": cmd_compile, "run": cmd_run}[a.cmd](a) + {"compile": cmd_compile, "run": cmd_run, "summarize": cmd_summarize}[a.cmd](a) if __name__ == "__main__": diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-1-r1-nadia-brennan.json new file mode 100644 index 000000000..f716662ab --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.023, + "engine_s": 1.114, + "wall_s": 4.369, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-2-r2-marcus-feld.json new file mode 100644 index 000000000..21282d886 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-2-r2-marcus-feld.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Marcus", + "got": "Marcus", + "match": true + }, + { + "field": "lname", + "want": "Feld", + "got": "Feld", + "match": true + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": "1958-09-02", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": "114 QUARRY HILL DRIVE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0124", + "got": "555-0124", + "match": true + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": "marcus.feld@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.023, + "engine_s": 1.092, + "wall_s": 3.806, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-3-r3-ines-carvalho.json new file mode 100644 index 000000000..96a6aa09f --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.021, + "engine_s": 1.053, + "wall_s": 3.758, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-4-r4-theo-lindqvist.json new file mode 100644 index 000000000..b7d10d91c --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.026, + "engine_s": 1.1, + "wall_s": 3.783, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-1-r1-nadia-brennan.json new file mode 100644 index 000000000..78095f21c --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.043, + "engine_s": 18.324, + "wall_s": 21.29, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-2-r2-marcus-feld.json new file mode 100644 index 000000000..1f8d8bbe9 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-2-r2-marcus-feld.json @@ -0,0 +1,99 @@ +{ + "arm": "browser", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": false, + "transaction_outcome": "RECONCILIATION_REQUIRED", + "actuation": null, + "api_fallback": "API steps are off for this run", + "step_errors": [ + [ + "step_040", + "Identity check failed for step 'step_040' (click recorded visual target): a target was found positionally (rung 'structural', confidence 1.00) but its surrounding text does not match the recorded target's \u2014 expected '', observed 'Emergency Contact: Confirm Create New Patient Cance" + ] + ], + "new_charts": 0, + "readback": "0/12", + "correct": false, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": null, + "match": false + }, + { + "field": "fname", + "want": "Marcus", + "got": null, + "match": false + }, + { + "field": "lname", + "want": "Feld", + "got": null, + "match": false + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": null, + "match": false + }, + { + "field": "sex", + "want": "Male", + "got": null, + "match": false + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": null, + "match": false + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": null, + "match": false + }, + { + "field": "state", + "want": "MA", + "got": null, + "match": false + }, + { + "field": "postal_code", + "want": "02139", + "got": null, + "match": false + }, + { + "field": "phone_home", + "want": "555-0124", + "got": null, + "match": false + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": null, + "match": false + }, + { + "field": "country_code", + "want": "USA", + "got": null, + "match": false + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.048, + "engine_s": 14.588, + "wall_s": 17.402, + "checkpoints": 40 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-3-r3-ines-carvalho.json new file mode 100644 index 000000000..df48c52c9 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.05, + "engine_s": 16.038, + "wall_s": 18.852, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-4-r4-theo-lindqvist.json new file mode 100644 index 000000000..43307b2f2 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.049, + "engine_s": 15.285, + "wall_s": 18.056, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-1-r5-grace-adeyemi.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-1-r5-grace-adeyemi.json new file mode 100644 index 000000000..d6f3067dc --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-1-r5-grace-adeyemi.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r5-grace-adeyemi", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Grace", + "got": "Grace", + "match": true + }, + { + "field": "lname", + "want": "Adeyemi", + "got": "Adeyemi", + "match": true + }, + { + "field": "DOB", + "want": "1990-02-08", + "got": "1990-02-08", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "402 CEDAR ROW", + "got": "402 CEDAR ROW", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0145", + "got": "555-0145", + "match": true + }, + { + "field": "email", + "want": "grace.adeyemi@example.invalid", + "got": "grace.adeyemi@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.025, + "engine_s": 1.188, + "wall_s": 4.307, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-2-r6-rosa-delgado.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-2-r6-rosa-delgado.json new file mode 100644 index 000000000..fa684296b --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-2-r6-rosa-delgado.json @@ -0,0 +1,99 @@ +{ + "arm": "api", + "ref": "r6-rosa-delgado", + "mismatch": true, + "error": null, + "engine_success": false, + "transaction_outcome": "RECONCILIATION_REQUIRED", + "actuation": "dom", + "api_fallback": "the check before saving (step_039) did not pass: missing ['No matches were found.']; the save was not sent", + "step_errors": [ + [ + "step_039", + "Postconditions failed for step 'step_039' (click recorded visual target): expected screen state not reached (semantic drift) \u2014 failed: text_present 'No matches were found.' \u2014 run aborted" + ] + ], + "new_charts": 0, + "readback": "0/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": null, + "match": false + }, + { + "field": "fname", + "want": "Rosa", + "got": null, + "match": false + }, + { + "field": "lname", + "want": "Delgado", + "got": null, + "match": false + }, + { + "field": "DOB", + "want": "1958-03-22", + "got": null, + "match": false + }, + { + "field": "sex", + "want": "Female", + "got": null, + "match": false + }, + { + "field": "street", + "want": "7 MILL ROAD", + "got": null, + "match": false + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": null, + "match": false + }, + { + "field": "state", + "want": "MA", + "got": null, + "match": false + }, + { + "field": "postal_code", + "want": "02139", + "got": null, + "match": false + }, + { + "field": "phone_home", + "want": "555-0188", + "got": null, + "match": false + }, + { + "field": "email", + "want": "rosa.delgado@example.invalid", + "got": null, + "match": false + }, + { + "field": "country_code", + "want": "USA", + "got": null, + "match": false + } + ], + "approvals": 1, + "approval_kinds": "dom", + "approval_wait_s": 0.026, + "engine_s": 22.666, + "wall_s": 25.59, + "checkpoints": 39 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-3-r7-arlo-whitcombe.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-3-r7-arlo-whitcombe.json new file mode 100644 index 000000000..8c2bd8ca1 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-3-r7-arlo-whitcombe.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r7-arlo-whitcombe", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Arlo", + "got": "Arlo", + "match": true + }, + { + "field": "lname", + "want": "Whitcombe", + "got": "Whitcombe", + "match": true + }, + { + "field": "DOB", + "want": "1966-05-30", + "got": "1966-05-30", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "88 BEACON TERRACE", + "got": "88 BEACON TERRACE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0152", + "got": "555-0152", + "match": true + }, + { + "field": "email", + "want": "arlo.whitcombe@example.invalid", + "got": "arlo.whitcombe@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.03, + "engine_s": 1.125, + "wall_s": 3.803, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-4-r8-priya-raman.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-4-r8-priya-raman.json new file mode 100644 index 000000000..5d47989f4 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-4-r8-priya-raman.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r8-priya-raman", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Priya", + "got": "Priya", + "match": true + }, + { + "field": "lname", + "want": "Raman", + "got": "Raman", + "match": true + }, + { + "field": "DOB", + "want": "1975-10-14", + "got": "1975-10-14", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "15 STATION AVENUE", + "got": "15 STATION AVENUE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0167", + "got": "555-0167", + "match": true + }, + { + "field": "email", + "want": "priya.raman@example.invalid", + "got": "priya.raman@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.025, + "engine_s": 1.152, + "wall_s": 3.842, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-1-r5-grace-adeyemi.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-1-r5-grace-adeyemi.json new file mode 100644 index 000000000..b13c98f80 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-1-r5-grace-adeyemi.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r5-grace-adeyemi", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Grace", + "got": "Grace", + "match": true + }, + { + "field": "lname", + "want": "Adeyemi", + "got": "Adeyemi", + "match": true + }, + { + "field": "DOB", + "want": "1990-02-08", + "got": "1990-02-08", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "402 CEDAR ROW", + "got": "402 CEDAR ROW", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0145", + "got": "555-0145", + "match": true + }, + { + "field": "email", + "want": "grace.adeyemi@example.invalid", + "got": "grace.adeyemi@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.055, + "engine_s": 18.26, + "wall_s": 21.718, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-2-r6-rosa-delgado.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-2-r6-rosa-delgado.json new file mode 100644 index 000000000..2c2218b94 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-2-r6-rosa-delgado.json @@ -0,0 +1,99 @@ +{ + "arm": "browser", + "ref": "r6-rosa-delgado", + "mismatch": true, + "error": null, + "engine_success": false, + "transaction_outcome": "RECONCILIATION_REQUIRED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [ + [ + "step_039", + "Postconditions failed for step 'step_039' (click recorded visual target): expected screen state not reached (semantic drift) \u2014 failed: text_present 'No matches were found.' \u2014 run aborted" + ] + ], + "new_charts": 0, + "readback": "0/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": null, + "match": false + }, + { + "field": "fname", + "want": "Rosa", + "got": null, + "match": false + }, + { + "field": "lname", + "want": "Delgado", + "got": null, + "match": false + }, + { + "field": "DOB", + "want": "1958-03-22", + "got": null, + "match": false + }, + { + "field": "sex", + "want": "Female", + "got": null, + "match": false + }, + { + "field": "street", + "want": "7 MILL ROAD", + "got": null, + "match": false + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": null, + "match": false + }, + { + "field": "state", + "want": "MA", + "got": null, + "match": false + }, + { + "field": "postal_code", + "want": "02139", + "got": null, + "match": false + }, + { + "field": "phone_home", + "want": "555-0188", + "got": null, + "match": false + }, + { + "field": "email", + "want": "rosa.delgado@example.invalid", + "got": null, + "match": false + }, + { + "field": "country_code", + "want": "USA", + "got": null, + "match": false + } + ], + "approvals": 1, + "approval_kinds": "dom", + "approval_wait_s": 0.031, + "engine_s": 24.871, + "wall_s": 28.107, + "checkpoints": 39 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-3-r7-arlo-whitcombe.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-3-r7-arlo-whitcombe.json new file mode 100644 index 000000000..ee05a2c9e --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-3-r7-arlo-whitcombe.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r7-arlo-whitcombe", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Arlo", + "got": "Arlo", + "match": true + }, + { + "field": "lname", + "want": "Whitcombe", + "got": "Whitcombe", + "match": true + }, + { + "field": "DOB", + "want": "1966-05-30", + "got": "1966-05-30", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "88 BEACON TERRACE", + "got": "88 BEACON TERRACE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0152", + "got": "555-0152", + "match": true + }, + { + "field": "email", + "want": "arlo.whitcombe@example.invalid", + "got": "arlo.whitcombe@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.047, + "engine_s": 16.847, + "wall_s": 19.804, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-4-r8-priya-raman.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-4-r8-priya-raman.json new file mode 100644 index 000000000..5cfcac1c3 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-4-r8-priya-raman.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r8-priya-raman", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Priya", + "got": "Priya", + "match": true + }, + { + "field": "lname", + "want": "Raman", + "got": "Raman", + "match": true + }, + { + "field": "DOB", + "want": "1975-10-14", + "got": "1975-10-14", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "15 STATION AVENUE", + "got": "15 STATION AVENUE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0167", + "got": "555-0167", + "match": true + }, + { + "field": "email", + "want": "priya.raman@example.invalid", + "got": "priya.raman@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.048, + "engine_s": 15.714, + "wall_s": 18.419, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-1-r1-nadia-brennan.json new file mode 100644 index 000000000..da326917a --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.027, + "engine_s": 1.822, + "wall_s": 5.114, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-2-r2-marcus-feld.json new file mode 100644 index 000000000..8dc00a019 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-2-r2-marcus-feld.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Marcus", + "got": "Marcus", + "match": true + }, + { + "field": "lname", + "want": "Feld", + "got": "Feld", + "match": true + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": "1958-09-02", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": "114 QUARRY HILL DRIVE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0124", + "got": "555-0124", + "match": true + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": "marcus.feld@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.028, + "engine_s": 1.102, + "wall_s": 3.921, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-3-r3-ines-carvalho.json new file mode 100644 index 000000000..e06609ff5 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.031, + "engine_s": 1.081, + "wall_s": 3.904, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-4-r4-theo-lindqvist.json new file mode 100644 index 000000000..86214e809 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.025, + "engine_s": 1.106, + "wall_s": 3.773, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-1-r1-nadia-brennan.json new file mode 100644 index 000000000..fe0d8e75c --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.047, + "engine_s": 15.583, + "wall_s": 18.693, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-2-r2-marcus-feld.json new file mode 100644 index 000000000..59874bcad --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-2-r2-marcus-feld.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Marcus", + "got": "Marcus", + "match": true + }, + { + "field": "lname", + "want": "Feld", + "got": "Feld", + "match": true + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": "1958-09-02", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": "114 QUARRY HILL DRIVE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0124", + "got": "555-0124", + "match": true + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": "marcus.feld@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.052, + "engine_s": 15.871, + "wall_s": 18.792, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-3-r3-ines-carvalho.json new file mode 100644 index 000000000..6ee11a79c --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.047, + "engine_s": 22.049, + "wall_s": 25.165, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-4-r4-theo-lindqvist.json new file mode 100644 index 000000000..d485d931d --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.053, + "engine_s": 15.59, + "wall_s": 18.753, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-1-r5-grace-adeyemi.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-1-r5-grace-adeyemi.json new file mode 100644 index 000000000..e0ac70c6c --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-1-r5-grace-adeyemi.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r5-grace-adeyemi", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Grace", + "got": "Grace", + "match": true + }, + { + "field": "lname", + "want": "Adeyemi", + "got": "Adeyemi", + "match": true + }, + { + "field": "DOB", + "want": "1990-02-08", + "got": "1990-02-08", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "402 CEDAR ROW", + "got": "402 CEDAR ROW", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0145", + "got": "555-0145", + "match": true + }, + { + "field": "email", + "want": "grace.adeyemi@example.invalid", + "got": "grace.adeyemi@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.028, + "engine_s": 1.082, + "wall_s": 4.212, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-2-r6-rosa-delgado.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-2-r6-rosa-delgado.json new file mode 100644 index 000000000..d7d80d8e3 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-2-r6-rosa-delgado.json @@ -0,0 +1,99 @@ +{ + "arm": "api", + "ref": "r6-rosa-delgado", + "mismatch": true, + "error": null, + "engine_success": false, + "transaction_outcome": "RECONCILIATION_REQUIRED", + "actuation": "dom", + "api_fallback": "the check before saving (step_039) did not pass: missing ['No matches were found.']; the save was not sent", + "step_errors": [ + [ + "step_039", + "Postconditions failed for step 'step_039' (click recorded visual target): expected screen state not reached (semantic drift) \u2014 failed: text_present 'No matches were found.' \u2014 run aborted" + ] + ], + "new_charts": 0, + "readback": "0/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": null, + "match": false + }, + { + "field": "fname", + "want": "Rosa", + "got": null, + "match": false + }, + { + "field": "lname", + "want": "Delgado", + "got": null, + "match": false + }, + { + "field": "DOB", + "want": "1958-03-22", + "got": null, + "match": false + }, + { + "field": "sex", + "want": "Female", + "got": null, + "match": false + }, + { + "field": "street", + "want": "7 MILL ROAD", + "got": null, + "match": false + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": null, + "match": false + }, + { + "field": "state", + "want": "MA", + "got": null, + "match": false + }, + { + "field": "postal_code", + "want": "02139", + "got": null, + "match": false + }, + { + "field": "phone_home", + "want": "555-0188", + "got": null, + "match": false + }, + { + "field": "email", + "want": "rosa.delgado@example.invalid", + "got": null, + "match": false + }, + { + "field": "country_code", + "want": "USA", + "got": null, + "match": false + } + ], + "approvals": 1, + "approval_kinds": "dom", + "approval_wait_s": 0.03, + "engine_s": 22.625, + "wall_s": 25.706, + "checkpoints": 39 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-3-r7-arlo-whitcombe.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-3-r7-arlo-whitcombe.json new file mode 100644 index 000000000..a7173128c --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-3-r7-arlo-whitcombe.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r7-arlo-whitcombe", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Arlo", + "got": "Arlo", + "match": true + }, + { + "field": "lname", + "want": "Whitcombe", + "got": "Whitcombe", + "match": true + }, + { + "field": "DOB", + "want": "1966-05-30", + "got": "1966-05-30", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "88 BEACON TERRACE", + "got": "88 BEACON TERRACE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0152", + "got": "555-0152", + "match": true + }, + { + "field": "email", + "want": "arlo.whitcombe@example.invalid", + "got": "arlo.whitcombe@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.03, + "engine_s": 1.123, + "wall_s": 3.806, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-4-r8-priya-raman.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-4-r8-priya-raman.json new file mode 100644 index 000000000..5d8d09a2f --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-4-r8-priya-raman.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r8-priya-raman", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Priya", + "got": "Priya", + "match": true + }, + { + "field": "lname", + "want": "Raman", + "got": "Raman", + "match": true + }, + { + "field": "DOB", + "want": "1975-10-14", + "got": "1975-10-14", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "15 STATION AVENUE", + "got": "15 STATION AVENUE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0167", + "got": "555-0167", + "match": true + }, + { + "field": "email", + "want": "priya.raman@example.invalid", + "got": "priya.raman@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.027, + "engine_s": 1.057, + "wall_s": 3.865, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-1-r5-grace-adeyemi.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-1-r5-grace-adeyemi.json new file mode 100644 index 000000000..986266edf --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-1-r5-grace-adeyemi.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r5-grace-adeyemi", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Grace", + "got": "Grace", + "match": true + }, + { + "field": "lname", + "want": "Adeyemi", + "got": "Adeyemi", + "match": true + }, + { + "field": "DOB", + "want": "1990-02-08", + "got": "1990-02-08", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "402 CEDAR ROW", + "got": "402 CEDAR ROW", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0145", + "got": "555-0145", + "match": true + }, + { + "field": "email", + "want": "grace.adeyemi@example.invalid", + "got": "grace.adeyemi@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.048, + "engine_s": 15.514, + "wall_s": 18.949, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-2-r6-rosa-delgado.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-2-r6-rosa-delgado.json new file mode 100644 index 000000000..901860315 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-2-r6-rosa-delgado.json @@ -0,0 +1,99 @@ +{ + "arm": "browser", + "ref": "r6-rosa-delgado", + "mismatch": true, + "error": null, + "engine_success": false, + "transaction_outcome": "RECONCILIATION_REQUIRED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [ + [ + "step_039", + "Postconditions failed for step 'step_039' (click recorded visual target): expected screen state not reached (semantic drift) \u2014 failed: text_present 'No matches were found.' \u2014 run aborted" + ] + ], + "new_charts": 0, + "readback": "0/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": null, + "match": false + }, + { + "field": "fname", + "want": "Rosa", + "got": null, + "match": false + }, + { + "field": "lname", + "want": "Delgado", + "got": null, + "match": false + }, + { + "field": "DOB", + "want": "1958-03-22", + "got": null, + "match": false + }, + { + "field": "sex", + "want": "Female", + "got": null, + "match": false + }, + { + "field": "street", + "want": "7 MILL ROAD", + "got": null, + "match": false + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": null, + "match": false + }, + { + "field": "state", + "want": "MA", + "got": null, + "match": false + }, + { + "field": "postal_code", + "want": "02139", + "got": null, + "match": false + }, + { + "field": "phone_home", + "want": "555-0188", + "got": null, + "match": false + }, + { + "field": "email", + "want": "rosa.delgado@example.invalid", + "got": null, + "match": false + }, + { + "field": "country_code", + "want": "USA", + "got": null, + "match": false + } + ], + "approvals": 1, + "approval_kinds": "dom", + "approval_wait_s": 0.022, + "engine_s": 22.555, + "wall_s": 25.298, + "checkpoints": 39 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-3-r7-arlo-whitcombe.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-3-r7-arlo-whitcombe.json new file mode 100644 index 000000000..858ce9b65 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-3-r7-arlo-whitcombe.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r7-arlo-whitcombe", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Arlo", + "got": "Arlo", + "match": true + }, + { + "field": "lname", + "want": "Whitcombe", + "got": "Whitcombe", + "match": true + }, + { + "field": "DOB", + "want": "1966-05-30", + "got": "1966-05-30", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "88 BEACON TERRACE", + "got": "88 BEACON TERRACE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0152", + "got": "555-0152", + "match": true + }, + { + "field": "email", + "want": "arlo.whitcombe@example.invalid", + "got": "arlo.whitcombe@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.053, + "engine_s": 15.174, + "wall_s": 17.925, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-4-r8-priya-raman.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-4-r8-priya-raman.json new file mode 100644 index 000000000..0ead31498 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-4-r8-priya-raman.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r8-priya-raman", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Priya", + "got": "Priya", + "match": true + }, + { + "field": "lname", + "want": "Raman", + "got": "Raman", + "match": true + }, + { + "field": "DOB", + "want": "1975-10-14", + "got": "1975-10-14", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "15 STATION AVENUE", + "got": "15 STATION AVENUE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0167", + "got": "555-0167", + "match": true + }, + { + "field": "email", + "want": "priya.raman@example.invalid", + "got": "priya.raman@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.043, + "engine_s": 15.596, + "wall_s": 18.571, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-1-r1-nadia-brennan.json new file mode 100644 index 000000000..2974ad6af --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.028, + "engine_s": 1.3, + "wall_s": 4.671, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-2-r2-marcus-feld.json new file mode 100644 index 000000000..d824da622 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-2-r2-marcus-feld.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Marcus", + "got": "Marcus", + "match": true + }, + { + "field": "lname", + "want": "Feld", + "got": "Feld", + "match": true + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": "1958-09-02", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": "114 QUARRY HILL DRIVE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0124", + "got": "555-0124", + "match": true + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": "marcus.feld@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.029, + "engine_s": 1.363, + "wall_s": 4.065, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-3-r3-ines-carvalho.json new file mode 100644 index 000000000..060a97af0 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.026, + "engine_s": 1.495, + "wall_s": 4.167, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-4-r4-theo-lindqvist.json new file mode 100644 index 000000000..e6bc09193 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "api", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "api", + "api_fallback": null, + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 1, + "approval_kinds": "api_request", + "approval_wait_s": 0.026, + "engine_s": 1.533, + "wall_s": 4.193, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-1-r1-nadia-brennan.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-1-r1-nadia-brennan.json new file mode 100644 index 000000000..f00af8a91 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-1-r1-nadia-brennan.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r1-nadia-brennan", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Ms.", + "got": "Ms.", + "match": true + }, + { + "field": "fname", + "want": "Nadia", + "got": "Nadia", + "match": true + }, + { + "field": "lname", + "want": "Brennan", + "got": "Brennan", + "match": true + }, + { + "field": "DOB", + "want": "1971-04-19", + "got": "1971-04-19", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "27 WILLOW BEND ROAD", + "got": "27 WILLOW BEND ROAD", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0121", + "got": "555-0121", + "match": true + }, + { + "field": "email", + "want": "nadia.brennan@example.invalid", + "got": "nadia.brennan@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.049, + "engine_s": 20.218, + "wall_s": 23.385, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-2-r2-marcus-feld.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-2-r2-marcus-feld.json new file mode 100644 index 000000000..672a01468 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-2-r2-marcus-feld.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r2-marcus-feld", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Marcus", + "got": "Marcus", + "match": true + }, + { + "field": "lname", + "want": "Feld", + "got": "Feld", + "match": true + }, + { + "field": "DOB", + "want": "1958-09-02", + "got": "1958-09-02", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "114 QUARRY HILL DRIVE", + "got": "114 QUARRY HILL DRIVE", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0124", + "got": "555-0124", + "match": true + }, + { + "field": "email", + "want": "marcus.feld@example.invalid", + "got": "marcus.feld@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.053, + "engine_s": 18.634, + "wall_s": 21.533, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-3-r3-ines-carvalho.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-3-r3-ines-carvalho.json new file mode 100644 index 000000000..213defb27 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-3-r3-ines-carvalho.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r3-ines-carvalho", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mrs.", + "got": "Mrs.", + "match": true + }, + { + "field": "fname", + "want": "Ines", + "got": "Ines", + "match": true + }, + { + "field": "lname", + "want": "Carvalho", + "got": "Carvalho", + "match": true + }, + { + "field": "DOB", + "want": "1949-12-11", + "got": "1949-12-11", + "match": true + }, + { + "field": "sex", + "want": "Female", + "got": "Female", + "match": true + }, + { + "field": "street", + "want": "9 LANTERN COURT", + "got": "9 LANTERN COURT", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0131", + "got": "555-0131", + "match": true + }, + { + "field": "email", + "want": "ines.carvalho@example.invalid", + "got": "ines.carvalho@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.06, + "engine_s": 16.615, + "wall_s": 19.53, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-4-r4-theo-lindqvist.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-4-r4-theo-lindqvist.json new file mode 100644 index 000000000..dbfeff64e --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-4-r4-theo-lindqvist.json @@ -0,0 +1,94 @@ +{ + "arm": "browser", + "ref": "r4-theo-lindqvist", + "mismatch": false, + "error": null, + "engine_success": true, + "transaction_outcome": "COMPLETED_UNVERIFIED", + "actuation": "dom", + "api_fallback": "API steps are off for this run", + "step_errors": [], + "new_charts": 1, + "readback": "12/12", + "correct": true, + "field_check": [ + { + "field": "title", + "want": "Mr.", + "got": "Mr.", + "match": true + }, + { + "field": "fname", + "want": "Theo", + "got": "Theo", + "match": true + }, + { + "field": "lname", + "want": "Lindqvist", + "got": "Lindqvist", + "match": true + }, + { + "field": "DOB", + "want": "1983-07-26", + "got": "1983-07-26", + "match": true + }, + { + "field": "sex", + "want": "Male", + "got": "Male", + "match": true + }, + { + "field": "street", + "want": "63 FERRY STREET", + "got": "63 FERRY STREET", + "match": true + }, + { + "field": "city", + "want": "EXAMPLETON", + "got": "EXAMPLETON", + "match": true + }, + { + "field": "state", + "want": "MA", + "got": "MA", + "match": true + }, + { + "field": "postal_code", + "want": "02139", + "got": "02139", + "match": true + }, + { + "field": "phone_home", + "want": "555-0136", + "got": "555-0136", + "match": true + }, + { + "field": "email", + "want": "theo.lindqvist@example.invalid", + "got": "theo.lindqvist@example.invalid", + "match": true + }, + { + "field": "country_code", + "want": "USA", + "got": "USA", + "match": true + } + ], + "approvals": 2, + "approval_kinds": "dom+dom", + "approval_wait_s": 0.052, + "engine_s": 15.536, + "wall_s": 18.458, + "checkpoints": 41 +} \ No newline at end of file diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/runs.csv b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/runs.csv new file mode 100644 index 000000000..0835c9741 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/runs.csv @@ -0,0 +1,41 @@ +arm,block,pos,ref,mismatch,correct,readback,new_charts,engine_success,transaction_outcome,actuation,api_fallback,approvals,approval_kinds,approval_wait_s,engine_s,wall_s,checkpoints,error,load_start,load_end,gate_waited_s,engine_sha +api,1,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.023,1.114,4.369,41,,,,,352af585b +api,1,2,r2-marcus-feld,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.023,1.092,3.806,41,,,,,352af585b +api,1,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.021,1.053,3.758,41,,,,,352af585b +api,1,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.026,1.1,3.783,41,,,,,352af585b +browser,1,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.043,18.324,21.29,41,,,,,352af585b +browser,1,2,r2-marcus-feld,False,False,0/12,0,False,RECONCILIATION_REQUIRED,,API steps are off for this run,2,dom+dom,0.048,14.588,17.402,40,,,,,352af585b +browser,1,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.05,16.038,18.852,41,,,,,352af585b +browser,1,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.049,15.285,18.056,41,,,,,352af585b +api,2,1,r5-grace-adeyemi,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.025,1.188,4.307,41,,,,,352af585b +api,2,2,r6-rosa-delgado,True,True,0/12,0,False,RECONCILIATION_REQUIRED,dom,the check before saving (step_039) did not pass: missing ['No matches were found.']; the save was not sent,1,dom,0.026,22.666,25.59,39,,,,,352af585b +api,2,3,r7-arlo-whitcombe,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.03,1.125,3.803,41,,,,,352af585b +api,2,4,r8-priya-raman,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.025,1.152,3.842,41,,,,,352af585b +browser,2,1,r5-grace-adeyemi,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.055,18.26,21.718,41,,,,,352af585b +browser,2,2,r6-rosa-delgado,True,True,0/12,0,False,RECONCILIATION_REQUIRED,dom,API steps are off for this run,1,dom,0.031,24.871,28.107,39,,,,,352af585b +browser,2,3,r7-arlo-whitcombe,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.047,16.847,19.804,41,,,,,352af585b +browser,2,4,r8-priya-raman,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.048,15.714,18.419,41,,,,,352af585b +browser,3,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.047,15.583,18.693,41,,,,,352af585b +browser,3,2,r2-marcus-feld,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.052,15.871,18.792,41,,,,,352af585b +browser,3,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.047,22.049,25.165,41,,,,,352af585b +api,3,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.027,1.822,5.114,41,,,,,352af585b +api,3,2,r2-marcus-feld,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.028,1.102,3.921,41,,,,,352af585b +api,3,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.031,1.081,3.904,41,,,,,352af585b +api,3,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.025,1.106,3.773,41,,,,,352af585b +browser,3,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.053,15.59,18.753,41,,,,,352af585b +api,4,1,r5-grace-adeyemi,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.028,1.082,4.212,41,,,,,352af585b +api,4,2,r6-rosa-delgado,True,True,0/12,0,False,RECONCILIATION_REQUIRED,dom,the check before saving (step_039) did not pass: missing ['No matches were found.']; the save was not sent,1,dom,0.03,22.625,25.706,39,,,,,352af585b +api,4,3,r7-arlo-whitcombe,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.03,1.123,3.806,41,,,,,352af585b +api,4,4,r8-priya-raman,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.027,1.057,3.865,41,,,,,352af585b +browser,4,1,r5-grace-adeyemi,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.048,15.514,18.949,41,,,,,352af585b +browser,4,2,r6-rosa-delgado,True,True,0/12,0,False,RECONCILIATION_REQUIRED,dom,API steps are off for this run,1,dom,0.022,22.555,25.298,39,,,,,352af585b +browser,4,3,r7-arlo-whitcombe,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.053,15.174,17.925,41,,,,,352af585b +browser,4,4,r8-priya-raman,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.043,15.596,18.571,41,,,,,352af585b +api,5,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.028,1.3,4.671,41,,,,,352af585b +api,5,2,r2-marcus-feld,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.029,1.363,4.065,41,,,,,352af585b +api,5,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.026,1.495,4.167,41,,,,,352af585b +api,5,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,api,,1,api_request,0.026,1.533,4.193,41,,,,,352af585b +browser,5,1,r1-nadia-brennan,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.049,20.218,23.385,41,,,,,352af585b +browser,5,2,r2-marcus-feld,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.053,18.634,21.533,41,,,,,352af585b +browser,5,3,r3-ines-carvalho,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.06,16.615,19.53,41,,,,,352af585b +browser,5,4,r4-theo-lindqvist,False,True,12/12,1,True,COMPLETED_UNVERIFIED,dom,API steps are off for this run,2,dom+dom,0.052,15.536,18.458,41,,,,,352af585b diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/summary.json b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/summary.json new file mode 100644 index 000000000..ce3187035 --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/summary.json @@ -0,0 +1,43 @@ +{ + "label": "API step under each_item approval + durable, no run gate; harness answers approvals; approval wait excluded", + "engine_sha": "352af585b", + "arms": { + "browser": { + "runs": 20, + "correct": 19, + "errors": 0, + "api_saves": 0, + "approvals_per_item": [ + 1, + 2 + ], + "approval_kinds": [ + "dom", + "dom+dom" + ], + "median_engine_s": 15.95, + "median_approval_wait_s": 0.049, + "max_load_start": null, + "safe_stops": 1, + "wrong_saves": 0 + }, + "api": { + "runs": 20, + "correct": 20, + "errors": 0, + "api_saves": 18, + "approvals_per_item": [ + 1 + ], + "approval_kinds": [ + "api_request", + "dom" + ], + "median_engine_s": 1.12, + "median_approval_wait_s": 0.026, + "max_load_start": null, + "safe_stops": 0, + "wrong_saves": 0 + } + } +} diff --git a/packages/flow/benchmark/openemr_api_tier/results/approval_durable/README.md b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/README.md new file mode 100644 index 000000000..8061b73cc --- /dev/null +++ b/packages/flow/benchmark/openemr_api_tier/results/approval_durable/README.md @@ -0,0 +1,55 @@ +# OpenEMR referral: API step under per-item approval and a durable run + +**Label: API step under each_item approval + durable, no run gate; harness +answers approvals; approval wait excluded.** These are not numbers for the +governed production profile. The referral bundle doesn't pass the run gate +yet: it is refused on its GUI path, for reasons unrelated to the API step +(see the admission list in the session notes, and the local-edit proposal, +PR #1366). + +Harness: [`approval_durable.py`](../../approval_durable.py). Data: +[`2026-10-19-combined/`](2026-10-19-combined/) (`runs.csv`, `summary.json`, +and each item's `result.json` under `items/`). + +## What ran + +- The OpenEMR new-patient referral, compiled from a recording with network + capture and admitted with `--identity subject_name=lname`, on a local + OpenEMR test copy with synthetic patients only. +- Two arms with the same bundle and record check: API steps off (`browser`) + and on (`api`). Both use `ApprovalGate("each_item")` and `durable=True`. +- 5 blocks of 4 referrals per arm, interleaved, with the arm order alternating + by block. Each block resets the fixture and seeds the existing chart that + referral r6 must not duplicate. +- The harness answers each approval request as soon as it is written, + standing in for the person. The time spent waiting for that answer is + measured (`approval_wait_s`) and left out of `engine_s`. +- Correctness is checked by the test setup's own direct SQL read-back. A new + referral is correct when exactly one new chart exists and it matches all 12 + fields. r6 is correct when no chart is created. +- Each item waited until the machine's 1-minute load was under 12. The run + was paused for a shared quiet window after 19 of 40 runs and resumed into a + second directory. The interrupted item was run again from the start. Load + per item was not kept for the rebuilt rows. +- Engine: the API-step stack (PRs #1358, #1361, #1362, #1364) at `352af585b`. + It does not include the review fixes #1367–#1372. With #1370 and #1371 this + referral's API step is refused: OpenEMR sends a background write while the + form is filled, and its form runs script on submit and on the name fields. + +## Results + +| Arm | Runs | Correct by read-back | Saved through | Median engine time | Approvals per item | +| --- | --- | --- | --- | --- | --- | +| API steps off | 20 | 19, plus 1 safe stop (nothing saved) | GUI | 15.95 s | 2 (1 for r6) | +| API step | 20 | 20 | API 18, GUI 2 (r6) | 1.12 s | 1 | + +- **API step.** Each new referral sent one request the person approved, with + the operation digest bound to the request bytes, and read back 12 of 12. + For r6 the API step's duplicate pre-check found the existing chart, so the + API step was not used. The GUI steps ran instead and created nothing, as + they should. +- **API steps off.** The GUI asks twice per new referral: once at "Create New + Patient", and once at the confirming save. One item (b01, r2) stopped itself + at the save's identity check (coverage 0.81) and saved nothing. The + earlier spike saw the same fail-closed stop once in 20 runs. +- No run saved a wrong or duplicate chart. diff --git a/packages/flow/docs/API_TIER.md b/packages/flow/docs/API_TIER.md index 2f56c5a93..e6fac0e78 100644 --- a/packages/flow/docs/API_TIER.md +++ b/packages/flow/docs/API_TIER.md @@ -11,6 +11,12 @@ the benchmark scoreboard, against the same program on the GUI (`program`), with the same record check. The harness and the earlier spike are in [benchmark/openemr_api_tier](../benchmark/openemr_api_tier/README.md). +Under per-item approval and a durable run (no run gate; the harness answers +the approvals and their wait is left out), the API step saved each new +referral correctly with one approval at a median 1.12 s of engine time, +against 15.95 s and two approvals with API steps off (20 runs each, +2026-10-19): [results](../benchmark/openemr_api_tier/results/approval_durable/README.md). + ## How to use it ```bash diff --git a/packages/flow/public-artifacts.json b/packages/flow/public-artifacts.json index 9b88e04d4..a4abb3059 100644 --- a/packages/flow/public-artifacts.json +++ b/packages/flow/public-artifacts.json @@ -371,6 +371,174 @@ "path": "benchmark/openemr/results.json", "sha256": "30de6dbb43ad0db4fd8b55737d765d06747b896ffae04522b6957d567dcfd89c" }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-1-r1-nadia-brennan.json", + "sha256": "720f7b21df440c2736ef2e7a8b1fcc95647d94413f9d65ba1bab0f957095a967" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-2-r2-marcus-feld.json", + "sha256": "0ffab27cdb11594c779424d3bdbc9b498ddf740d480042f7370b727e820da030" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-3-r3-ines-carvalho.json", + "sha256": "f7721ce8ae1e57ef52638138e920a1758ac61e62699e954310cd5e21c607b240" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-4-r4-theo-lindqvist.json", + "sha256": "bc7fd5d2037d605dbf242ce4005ea784cd63c8e4b62b173362c664efb0a48f9b" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-1-r1-nadia-brennan.json", + "sha256": "f818fd9dd2a964c10a88ff69ce4c10c5e3b27af08c7f7f8d60c6a9aa665ee761" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-2-r2-marcus-feld.json", + "sha256": "13a6ba67f0fde89c5ef8370ff2c2661434d16d5947a5f42a401a3fc218bbc648" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-3-r3-ines-carvalho.json", + "sha256": "d93aa3ff2703c9eae85ce4c7de73ad3aad9ec24965c46711b06d292af9b14991" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-4-r4-theo-lindqvist.json", + "sha256": "b79c859009098cfe8e526b5775601697c1612a3280af1011d879273d97ccecc7" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-1-r5-grace-adeyemi.json", + "sha256": "ac05089531d7c3d0bc7d0a5513ac9789e9594983c24e476dd5300047633fb3aa" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-2-r6-rosa-delgado.json", + "sha256": "0e27df1fc55e4952a5e5bc7ee842005d54a7e8ccd2ab66f486a724febf7a9d72" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-3-r7-arlo-whitcombe.json", + "sha256": "4a2dbf32430ce5f4c064854472658486aa56eaffd63eb446191535c4c81b1c41" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-4-r8-priya-raman.json", + "sha256": "903fc03903e2ed8ab7a1be8cd004ae09b0fd6f16cfdd82c0fb1902667052e7b9" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-1-r5-grace-adeyemi.json", + "sha256": "2fd6d6a3fabc7987a854f925d84d9c709ce95f2c3d992bf8d04e0c18a520803c" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-2-r6-rosa-delgado.json", + "sha256": "8bd71537bf0737a2cb932753c4dc8e4e12c9fcf3012a85495f20ba4e1d9f132e" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-3-r7-arlo-whitcombe.json", + "sha256": "3e74161adf06a0b692153da436ac1d289aa65d74976989134d6189bf71291dd1" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-4-r8-priya-raman.json", + "sha256": "4e690e3e561a421130fdfee2d88e8cf0005d604b741bcc7ebcad123f4b4c92d4" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-1-r1-nadia-brennan.json", + "sha256": "56f90f6c2e1c29e542c3da5c7fa02785f471bdcbf38a0cd285889b675058bdc7" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-2-r2-marcus-feld.json", + "sha256": "a3ea8fdc6b2f25feded5e257e8f64cd644783ea8b784daaef2c9c3a15c09d674" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-3-r3-ines-carvalho.json", + "sha256": "192eb11514aa498c79ae2997243084f214f0a903d7c614b88e9a4d867f072977" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-4-r4-theo-lindqvist.json", + "sha256": "0e04941fea9db21558e7c75c37c37fe2244006fe21af5723a22984d86211dd39" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-1-r1-nadia-brennan.json", + "sha256": "f6436902635448d7014d3fb8b03f9fc5bc5026925d3bba205ae23f58772fb0fc" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-2-r2-marcus-feld.json", + "sha256": "149c85d593c8b304343b89742095fc0be38bd5dd3ba5dba65756223a42b6bccd" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-3-r3-ines-carvalho.json", + "sha256": "7713f65b351b3430e5efe337fffa1ec593357fe38301616538b9113fb79b2229" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-4-r4-theo-lindqvist.json", + "sha256": "74d05b07758e22d02bf58e8bea3bada953ba500fe5a4af69b54067630b1cbde8" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-1-r5-grace-adeyemi.json", + "sha256": "5f64b48414a8026590043ab2c4d27c1a5bb527e8f2e9cfce34ec3ff60db84b33" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-2-r6-rosa-delgado.json", + "sha256": "171e98fe4617407dc0fba49228a183caff66e0d82170cfb39a1c8692d7b0b2dc" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-3-r7-arlo-whitcombe.json", + "sha256": "484e0ddc68ceb665f1a0871d3c0c1bfae0b65ff69ccdc855d2600f0b4a5beb5f" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-4-r8-priya-raman.json", + "sha256": "6562554d027ea6771f9bc3c28c291b08de875389b0608cec71105b1a0def869b" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-1-r5-grace-adeyemi.json", + "sha256": "dacfb7eb2d7432c22cb4f713f0221b016de9bab3e7c58498b6bf775b1cccd4bd" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-2-r6-rosa-delgado.json", + "sha256": "06d6abe039bacf185863912246796197cd5c581d462fd13e80dbfff359e2b152" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-3-r7-arlo-whitcombe.json", + "sha256": "110529a961ce6a86d12c3d0bd46ff7734bcc0e3d50e5962adafe83ceecbc4c49" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-4-r8-priya-raman.json", + "sha256": "64b7250dc37ac9d15f82f49d242d1530cc0ee02b9c85d0af762a74f565aa8370" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-1-r1-nadia-brennan.json", + "sha256": "8b568206d6543fef092f12f11ecf0c07b5a1651fa31196ba6cd441f408ff71e6" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-2-r2-marcus-feld.json", + "sha256": "d5e8925e416023b05c4c2168efb63423f36e10ddd62b884f4fce88d590279a1f" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-3-r3-ines-carvalho.json", + "sha256": "41ee9b1f36bfedf2eaa22710b2b59a34fa96da02fae36e6b2d044927624243d4" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-4-r4-theo-lindqvist.json", + "sha256": "89fddbcadbb47fa601b728164e9890d7ce4d375c0e199df868843e3daa73c75a" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-1-r1-nadia-brennan.json", + "sha256": "2968a9309a5c111bc5cdbb4fe752c127b147096bc8ab6bd04d66f84e635401ff" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-2-r2-marcus-feld.json", + "sha256": "c58de6296dc893f69f5d9508e6a4348f71a2cecdd7e311b11bfa6f7b77043ae2" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-3-r3-ines-carvalho.json", + "sha256": "f7ec8cc86c8bfeaa8f7bdc8dff0b5f62cc00e735f249a0ca4a3991c14b7e304e" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-4-r4-theo-lindqvist.json", + "sha256": "09046626ea521bed7edfc1bbc316293fea2d58c9cf6726f2cc181d0252d9f604" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/runs.csv", + "sha256": "4ce59146852f10c9a076912f7d7acbfcfb6815f8ee84181edf80bd217d3e3645" + }, + { + "path": "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/summary.json", + "sha256": "dcff06ae9d8273191d18afdf042fb21746f4c5f30149d62d61187b603abe6dd0" + }, { "path": "benchmark/openemr_api_tier/spike/api_bundle/compile_report.json", "sha256": "9bea1fdb1e7caa3cd4e25d2dcf6c46bb82871a35aa18c4dbb471d2c4fb6321bc" From 914539ac5b8c517c854bcbd5b7821c69a1dbd527 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 18:26:18 -0400 Subject: [PATCH 10/13] fix(flow): type the native min/max checks separately (mypy) Co-Authored-By: Claude Opus 5.5 --- .../runtime/actuators/web_form.py | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/packages/flow/openadapt_flow/runtime/actuators/web_form.py b/packages/flow/openadapt_flow/runtime/actuators/web_form.py index 27915c01a..8473683b2 100644 --- a/packages/flow/openadapt_flow/runtime/actuators/web_form.py +++ b/packages/flow/openadapt_flow/runtime/actuators/web_form.py @@ -273,17 +273,20 @@ def native_problem(name: str, control: dict[str, Any], value: str) -> Optional[s number = _number(value) if number is None: return f"{name} is not a number" - for attr, bad in (("min", number.__lt__), ("max", number.__gt__)): - limit = _number(str(control.get(attr) or "")) - if limit is not None and bad(limit): - return f"{name} is outside the form's {attr} {control.get(attr)}" + low = _number(str(control.get("min") or "")) + high = _number(str(control.get("max") or "")) + if low is not None and number < low: + return f"{name} is outside the form's min {control.get('min')}" + if high is not None and number > high: + return f"{name} is outside the form's max {control.get('max')}" if kind == "date": if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", value): return f"{name} is not a date (YYYY-MM-DD)" - for attr, bad in (("min", value.__lt__), ("max", value.__gt__)): - limit = control.get(attr) - if limit and bad(limit): - return f"{name} is outside the form's {attr} {limit}" + low_date, high_date = control.get("min"), control.get("max") + if low_date and value < str(low_date): + return f"{name} is outside the form's min {low_date}" + if high_date and value > str(high_date): + return f"{name} is outside the form's max {high_date}" return None From 9c9e8d59cef524dc760478a5ba8e5631818eba00 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 18:45:49 -0400 Subject: [PATCH 11/13] fix(flow): bind the authorization's API-step check to the full replaced-step digest After merging main (#1361): the run authorization's admitted_api_step compared folded_checks_digest over the steps before the save, while #1372 now digests every replaced step including the save. It uses the same range as the replayer and the proposal. Tests that edit the save re-propose its digest first, as openadapt-flow api propose does. Co-Authored-By: Claude Opus 5.5 --- .../openadapt_flow/runtime/authorization.py | 4 +-- packages/flow/tests/test_api_step_durable.py | 26 +++++++++++++++++-- packages/flow/tests/test_api_step_governed.py | 2 +- 3 files changed, 27 insertions(+), 5 deletions(-) diff --git a/packages/flow/openadapt_flow/runtime/authorization.py b/packages/flow/openadapt_flow/runtime/authorization.py index 5748c7b77..2cbd94623 100644 --- a/packages/flow/openadapt_flow/runtime/authorization.py +++ b/packages/flow/openadapt_flow/runtime/authorization.py @@ -343,7 +343,7 @@ class AdmittedApiStep(BaseModel): """One API step (``web_form`` binding) admitted into a governed run. Binds the exact binding a person admitted (``admission_digest``), the - on-screen checks of the GUI steps it stands for (``folded_checks_digest``) + complete definition of the GUI steps it stands for (``folded_checks_digest``) and its record check (``effect_contract_hashes``). The runtime runs an API step under a governed authorization only when all three still match. """ @@ -545,7 +545,7 @@ def admitted_api_step(self, step: Any, workflow: Any) -> Optional[str]: row.admission_digest != binding.admission_digest() or row.folded_checks_digest != binding.web_form.folded_checks_digest or row.folded_checks_digest - != folded_checks_digest(workflow.steps[start:end]) + != folded_checks_digest(workflow.steps[start : end + 1]) or list(row.effect_contract_hashes) != [effect.contract_hash() for effect in step.effects] ): diff --git a/packages/flow/tests/test_api_step_durable.py b/packages/flow/tests/test_api_step_durable.py index 82b80900d..322c1c090 100644 --- a/packages/flow/tests/test_api_step_durable.py +++ b/packages/flow/tests/test_api_step_durable.py @@ -79,6 +79,28 @@ def actuate(self, binding, params): ) +def _reproposed(wf: Workflow) -> Workflow: + """The span program after an edit to its steps, its API step proposed again. + + The API step's admission binds the complete definition of the steps it + replaces, so an edited save needs the binding's digest recomputed, as + ``openadapt-flow api propose`` does, before it is admitted again. + """ + + from openadapt_flow.runtime.actuators.web_form import folded_checks_digest + + save = wf.steps[-1] + spec = save.api_binding.web_form.model_copy( + update={"folded_checks_digest": folded_checks_digest(wf.steps)} + ) + binding = save.api_binding.model_copy(update={"web_form": spec}) + return wf.model_copy( + update={ + "steps": [*wf.steps[:-1], save.model_copy(update={"api_binding": binding})] + } + ) + + def _absolute_check_workflow() -> Workflow: """The span program with a record check that needs no before-snapshot. @@ -91,7 +113,7 @@ def _absolute_check_workflow() -> Workflow: effect = save.effects[0].model_copy(update={"forbid_collateral_loss": False}) binding = save.api_binding.model_copy(update={"effects": [effect]}) save = save.model_copy(update={"effects": [effect], "api_binding": binding}) - wf = wf.model_copy(update={"steps": [*wf.steps[:-1], save]}) + wf = _reproposed(wf.model_copy(update={"steps": [*wf.steps[:-1], save]})) wf, _ = admit_api_step(wf, admitted_by="reviewer") return wf @@ -330,7 +352,7 @@ def test_an_identity_checked_save_with_a_right_record_contract_runs_durably(tmp_ update={"steps": [*wf.steps[:-1], save], "params": {**wf.params, "note": "api"}} ) wf, _ = admit_api_step( - wf, admitted_by="reviewer", identity={"subject_name": "note"} + _reproposed(wf), admitted_by="reviewer", identity={"subject_name": "note"} ) fx = Fixture(tmp_path, wf) try: diff --git a/packages/flow/tests/test_api_step_governed.py b/packages/flow/tests/test_api_step_governed.py index d26917e7b..2aeb259eb 100644 --- a/packages/flow/tests/test_api_step_governed.py +++ b/packages/flow/tests/test_api_step_governed.py @@ -96,7 +96,7 @@ def _workflow(*, admitted: bool = True, identity: bool = True) -> Workflow: ], controls=["csrf", "lname"], replaces_steps=[typing.id, submit.id], - folded_checks_digest=folded_checks_digest([typing]), + folded_checks_digest=folded_checks_digest([typing, submit]), ) submit = submit.model_copy( update={ From 2a5cc1fbcdf51cbfd660235c1cd5baf2f9055aa9 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 19:13:19 -0400 Subject: [PATCH 12/13] flow: classify local edits only on strict evidence; add Enter in a field and a dialog's dismiss Every pattern now needs network capture showing no write in the step's window, URL, title and page count unchanged, and equal record-store snapshots when the recorder took them. Control+a no longer classifies without capture. New patterns: confirm_entry (Enter after typing into a text field; the field_value check reads the text field's exact value) and dismiss_dialog (an informational dialog's dismiss, e.g. MockMed's What's New 'Continue to tasks'; checks the page is unchanged and the label is gone). A dismiss is reconsidered only when 'continue' or 'next' are its only write words and its risk is still the per-step keyword inference, so effect mining no longer attaches a placeholder record_written to it (#1377). Fixes the CI crash on a structural target with no frame_path and the mypy index error. Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 1 + llms-full.txt | 185 ++++++++- .../docs/design/LOCAL_EDIT_CLASSIFICATION.md | 131 ++++--- .../openadapt_flow/compiler/local_edit.py | 354 ++++++++++++------ packages/flow/openadapt_flow/risk.py | 35 ++ .../flow/openadapt_flow/runtime/replayer.py | 31 +- .../tests/test_local_edit_classification.py | 247 +++++++++++- 7 files changed, 785 insertions(+), 199 deletions(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index c42ee9af9..415000e81 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -124,6 +124,7 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile one or more demonstrations into a deterministic program | Released | Real app (OpenEMR referral, 62 timed runs) | `openadapt-flow record`, `compile`; [packages/flow](packages/flow/README.md) | +| Classify common local edits without a person (select-all, Enter in a field or dropdown, an in-form tab, an informational dialog's dismiss), only when the recording's network capture shows no write and a runtime check confirms only the expected change | In review ([#1366](https://github.com/OpenAdaptAI/OpenAdapt/pull/1366)) | Synthetic; OpenEMR referral recording re-admitted offline (violations 12 to 1 under `permissive`) | [LOCAL_EDIT_CLASSIFICATION.md](packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md), `compiler/local_edit.py` | | Right-record check before any write (structure, then OCR, then pixels) | Released | Real app and synthetic | [RESOLUTION_LADDER.md](packages/flow/docs/RESOLUTION_LADDER.md), `runtime/identity.py` | | Read the saved record back from the system of record (API, database, FHIR, separate session) | Released | Real app (OpenEMR 12 of 12 fields) | [VERIFICATION.md](packages/flow/docs/VERIFICATION.md), `runtime/effects/` | | Read values out of an app as a workflow step (READ), with a records format, a signed review queue and export to CSV or a database | On main ([#1229](https://github.com/OpenAdaptAI/OpenAdapt/pull/1229), [#1215](https://github.com/OpenAdaptAI/OpenAdapt/pull/1215), [#1218](https://github.com/OpenAdaptAI/OpenAdapt/pull/1218)–[#1224](https://github.com/OpenAdaptAI/OpenAdapt/pull/1224)) | Synthetic (mock desktop apps) | `openadapt_flow/interchange/` | diff --git a/llms-full.txt b/llms-full.txt index 7f4752fa6..5a0686033 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -416,9 +416,10 @@ shipping in the next release; a pull request number means it is being merged. review, campaign, canary, rollback), agent-demonstrated fixes for a halted run, and fleet drift patches shared with no patient data are released. Self-repair, where an agent proposes a fix and Flow probes it up to the save - without saving, is on main; fixes of up to three new steps are in review. -9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded - save around any agent's write + without saving, is on main, and so are fixes of up to three new steps. +9. **Is a safety layer underneath any agent.** An agent's MCP calls, a task + ledger that keeps a long job's rules, items, questions and checked saves on + the server ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)), a guarded save around any agent's write ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run` to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. @@ -480,6 +481,7 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile one or more demonstrations into a deterministic program | Released | Real app (OpenEMR referral, 62 timed runs) | `openadapt-flow record`, `compile`; [packages/flow](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/README.md) | +| Classify common local edits without a person (select-all, Enter in a field or dropdown, an in-form tab, an informational dialog's dismiss), only when the recording's network capture shows no write and a runtime check confirms only the expected change | In review ([#1366](https://github.com/OpenAdaptAI/OpenAdapt/pull/1366)) | Synthetic; OpenEMR referral recording re-admitted offline (violations 12 to 1 under `permissive`) | [LOCAL_EDIT_CLASSIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md), `compiler/local_edit.py` | | Right-record check before any write (structure, then OCR, then pixels) | Released | Real app and synthetic | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md), `runtime/identity.py` | | Read the saved record back from the system of record (API, database, FHIR, separate session) | Released | Real app (OpenEMR 12 of 12 fields) | [VERIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION.md), `runtime/effects/` | | Read values out of an app as a workflow step (READ), with a records format, a signed review queue and export to CSV or a database | On main ([#1229](https://github.com/OpenAdaptAI/OpenAdapt/pull/1229), [#1215](https://github.com/OpenAdaptAI/OpenAdapt/pull/1215), [#1218](https://github.com/OpenAdaptAI/OpenAdapt/pull/1218)–[#1224](https://github.com/OpenAdaptAI/OpenAdapt/pull/1224)) | Synthetic (mock desktop apps) | `openadapt_flow/interchange/` | @@ -574,6 +576,7 @@ improvising or retrying when something is off. | Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | +| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | In review ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#track-a-long-job) | | Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | | Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | | One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | @@ -591,19 +594,112 @@ improvising or retrying when something is off. ## Every command, by capability -Each `openadapt-flow` command, grouped by the capability it serves. A test -fails when a command is added without being listed here. - -- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`, `compile-agent`. -- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. -- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`, `deferred`. -- **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). -- **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). -- **Give agents the program:** `emit-skill`, `emit-mcp`, `serve-execute`. -- **Proof:** `seal`, `local-seal`, `countersign`, `attach-seal`. -- **Run on premises:** `service`, `no-outbound`. -- **Share safely with OpenAdapt Cloud:** `sanitize`, `review-sanitized`, `approve-sanitized`, `materialize-approved`, `push`, `validate-hosted`, `report-break`, `report-run`, `connect`, `login`, `connector`. -- **Try it and measure it:** `demo`, `tutorial`, `bench`, `benchmark`. +Each `openadapt-flow` command, grouped by the capability it serves. Each +command has one file under +[`packages/flow/docs/commands/`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/commands/): to add a +command, add `/.md` and run `python scripts/build_command_map.py`. +A test fails when a command has no file or this list is out of date. + + + +**Record and build a program:** + +- `compile-agent` +- `compile` +- `compose` +- `demo-record` +- `disambiguate` +- `for-each` +- `induce` +- `intake` +- `process` +- `pull` +- `record` +- `route` +- `variants` +- `visualize` + +**Check and admit:** + +- `admit` +- `api` +- `approver-grant` +- `bind-record-checks` +- `certify` +- `check` +- `jit` +- `known-ids` +- `lint` +- `propose-qualification` +- `qualify` +- `scaffold-verifier` + +**Run and answer:** + +- `approve` +- `business-decisions` +- `console` +- `deferred` +- `explain` +- `item-approval` +- `replay` +- `resume` +- `run` +- `triage` + +**Judge any executor's write:** + +- `serve-reward`: Scores agent training episodes by reading the system of record. +- `verify-run` + +**Repair and learn from halts:** + +- `fleet` +- `repair` +- `teach-branch` +- `teach` +- `undo`: The undo steps a person approves after a stop. + +**Give agents the program:** + +- `emit-mcp` +- `emit-skill` +- `serve-execute` + +**Proof:** + +- `attach-seal` +- `countersign` +- `local-seal` +- `seal` + +**Run on premises:** + +- `no-outbound` +- `service` + +**Share safely with OpenAdapt Cloud:** + +- `approve-sanitized` +- `connect` +- `connector` +- `login` +- `materialize-approved` +- `push` +- `report-break` +- `report-run` +- `review-sanitized` +- `sanitize` +- `validate-hosted` + +**Try it and measure it:** + +- `bench` +- `benchmark` +- `demo` +- `tutorial` + + ## Measured results @@ -1128,6 +1224,30 @@ halts with a named reason when the confirmation is missing before and after numbers are in [`benchmark/known_ids/KNOWN_IDS.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../benchmark/known_ids/KNOWN_IDS.md). +Before a step stops on a look-alike identifier, the replayer takes extra looks +that change nothing +([`openadapt_flow/runtime/identity_looks.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/identity_looks.py)): +the same read on two fresh screenshots, the 2x re-read, the known-ID list when +one is configured, and, where the surface allows it, a select-and-copy of the +identifier's text box (`runtime.identity_copy_look`). OCR looks can't settle +the record on their own, because a look-alike twin can read the same on every +frame and at every size. Only two looks can: the known-ID list, and a copied +identifier that equals a run parameter in the identity band. A look that names +another record stops the run. Each look is recorded on the step's identity +check. + +The copy look has limits. It needs a surface where selecting the identifier +and copying it changes nothing, and a clipboard that reaches the machine Flow +runs on (RDP or Citrix clipboard redirection). It drags across only the one +text box that holds the identifier, and the run stops if the screen changes +anywhere else during the look. A canvas with no selectable text has no copy +look. A recording's own identifier is OCR text and can be a misread (the RDP +kiosk recorded `A1001` as `Alo01`), so in a workflow that always opens the +recorded record, the copy can stop a run but never confirm one. It confirms +only a record chosen by a run parameter. Governed and qualification runs never +take it. The check just before the click takes no input; it reuses the copy +only when the identifier's pixels are unchanged since the look. + Permissive `replay` may proceed on an unreadable reversible target and reports that condition. Governed `run` requires an affirmative live verdict for every identity-required admitted step; mismatch, unreadable, or abstain halts before @@ -1156,6 +1276,39 @@ irreversible, and qualification requires its record check. An icon-only click that only navigates after data entry is therefore over-classified until an operator reclassifies it. +On a pixel surface an icon-only click carries no text that names its control. +When the recorder can move the pointer without clicking +(`Recorder(capture_hover=True)` on a backend with `hover`), it rests the +pointer on each click target first and keeps that frame. For a click with no +other identity evidence, the compiler stores what changed near the target, such +as the control's highlight or tooltip, as the step's hover evidence +([`openadapt_flow/runtime/hover_evidence.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/hover_evidence.py)). +At replay, on the pixel rungs, the runtime rests the pointer on the resolved +target, waits the same 0.8 s, and compares. A different response, or a +response where the recording saw none, stops the run before the click. A +missing response, or one that can't be compared (another render with no +readable text), is recorded on the step and changes nothing. A tooltip that +opens later than the wait looks like a different response and stops the run. +Controls that don't answer the pointer, such as most text fields, get no +evidence this way. The browser, remote-display (Citrix and RDP client window) +and FreeRDP backends can hover. The Aardwolf RDP transport and the native +Windows, macOS and Linux backends can't move the pointer without a button +yet, so they take no hover look. + +A pixel step is checked on one frame and clicked a moment later. Consequential +identity-gated clicks were already bound to the exact checked frame. Every +other pixel click now gets a check-to-click guard (`Replayer._pixel_recheck`): +right before the click, the runtime takes a fresh frame and compares the +target region, the identity region and the hover response with the frame the +step was checked on. If one changed (a dialog opened, the list re-sorted), it +doesn't click. It re-resolves the target and re-runs the identity check on +the fresh frame, with no input, and clicks only when the next frame matches +that one. After two re-observations that still see a change, or when the +target or its identity can't be confirmed again, it refuses the click. Each +change is recorded on the step (`StepResult.check_to_click`). A change smaller +than 4 pixels, or under 24 levels per channel, doesn't count. Structured +targets are bound by element tokens instead and aren't rechecked. + A consequential click must be tied to a record before Flow delivers it. Inside a table row, the row is that record. An icon-only Save that floats beside a form (openIMIS's claim Save has no text, accessible name, or tooltip) has no diff --git a/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md b/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md index 036956b04..4b94b33fc 100644 --- a/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md +++ b/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md @@ -1,6 +1,9 @@ -# Classifying local edits without a person (proposal) +# Classifying local edits without a person -Status: proposal. Not merged; the founder decides. +Status: approved by the founder on 2026-10-19, with a strict rule: the +compiler classifies a step only when the recording shows no mutating request +sent for it **and** a runtime check confirms only the expected local change. +Anything else goes to a person. ## The problem @@ -22,18 +25,38 @@ These are the same few patterns in every form-filling program, on every app. A person approving each one, per bundle, is busywork that trains people to approve without reading. -## The proposal +## The rule -The compiler classifies three local-edit patterns itself, and only when the -recording shows the action stayed local. Each classified step gets an -automatic outcome check that halts the run if, at run time, the action did -anything else. +The compiler classifies five local-edit patterns itself. Every pattern needs +all of this evidence from the recording: -| Pattern | Evidence the compiler needs (all of it) | Outcome check added to the step | +- network capture (`record --capture-network`), with no request other than + `GET`, `HEAD` or `OPTIONS` between the previous event and the next one; +- URL, title and page count recorded on both sides of the action, and + unchanged; +- when the recorder snapshotted the record store around the step, the two + snapshots equal. + +A recording without network capture classifies nothing. + +| Pattern | Step evidence | Outcome check added to the step | | --- | --- | --- | -| Select-all before typing: `Control+a` / `Meta+a` | The previous step clicked a single text field (structural role `textbox`, `searchbox` or `spinbutton`); the next step types into it; URL, title and page count unchanged across the key; no write request during the key (when the recording has network capture) | `local_edit`: after the key, the focused element is still that field, and URL, title and page count are unchanged. The next step's typed-input read-back then checks the field's whole value. | -| Confirm a typed option: `Enter` | The previous steps clicked a dropdown (role `combobox` or `listbox`) and typed an option's label (a parameter or a literal); URL, title and page count unchanged; no write request during the key (network capture **required**: `Enter` is a submission key) | `local_edit` as above, plus `field_value`: the focused dropdown shows the typed label (the run's parameter value). | -| In-form tab: a click on a tab | Role `tab`, or a `button` whose recorded click changed only the visible panel: URL, title and page count unchanged, no write request (network capture required), and the next step acts in the same frame | `local_edit` (URL, title and page count unchanged; focus need not stay). | +| `select_all`: `Control+a` / `Meta+a` | The previous step clicked a text field (role `textbox`, `searchbox` or `spinbutton`, with a selector); the next step types into it | `local_edit`: focus is still on that field; URL, title and page count unchanged. The next step's typed-input read-back then checks the field's whole value. | +| `confirm_option`: `Enter` in a dropdown | The steps before clicked a dropdown (role `combobox` or `listbox`) and typed a parameter's value | `local_edit` on the dropdown, plus `field_value`: the focused dropdown shows this run's value | +| `confirm_entry`: `Enter` in a text field | The steps before clicked a text field and typed a parameter's value | `local_edit` on the field, plus `field_value`: the focused field's value equals this run's value exactly | +| `in_form_tab`: a tab click | Role `tab`, or a `button` whose only mined check is a region change; the next acting step is in the same frame | `local_edit` (URL, title and page count unchanged; focus may move) | +| `dismiss_dialog`: an informational dialog's dismiss | A `button` or `link` labelled only "Continue", "Continue to ...", "Next", "Close", "OK", "Got it", "Dismiss", "Not now", "Maybe later" or "Skip for now" | `local_edit`, plus `text_absent` of the label: the dialog is gone | + +`continue` and `next` are write words for the risk inference, so a dismiss +labelled with them starts irreversible. It is reconsidered only when those +are its only write words and its risk is still the per-step keyword +inference. A click that a later rule made irreversible (the click that +confirms a dialog, an icon-only click presumed to save) is never +reconsidered. Because a classified dismiss is reversible, effect mining no +longer attaches a placeholder `record_written` to it (#1377). + +The runtime checks fail closed: a backend that can't read focus, the field +value, the URL, the title or the page count fails the step. What stays the same: @@ -43,7 +66,8 @@ What stays the same: `risk_review_required` and goes to a person, exactly as today. - **Write-shaped context still wins.** A key or click whose retained text names a consequential write (`submit`, `save`, `delete`, ...) classifies - irreversible before any of this runs. + irreversible before any of this runs. The one exception is a dismiss whose + only write words are `continue` or `next`, with the full evidence above. - **Explicit `risk_overrides` still win.** - **D, the duplicate-prevention (idempotency) key on the save, is untouched.** That is a business decision. @@ -55,18 +79,18 @@ evidence), and has an outcome check, so `require_reviewed_action_risk`, tab click classified this way is in-form navigation that writes nothing, so it isn't consequential and isn't entity navigation; `prohibit_unarmed_clicks` under `clinical-write` still applies to it, so it still needs an identity -check unless that policy exempts `local_edit` clicks (an open question below). +check (decision 2 below). ## Why this is safe enough - The classification is backed by what the demonstration did, not by the - label. A key that submitted the form in the recording (a POST, a URL change) - is never classified. + label. A step that sent a write request in the recording, changed the URL, + title or page count, or changed the record store is never classified. - The outcome check runs every time. If the same key does something else at run time (the app changed, a different field had focus), the step fails its postcondition and the run halts at that step. The halt comes after the key was pressed, which is the residual risk: a key that, at run time only, - commits something. For the three patterns that requires the app to have + commits something. For these patterns that requires the app to have changed the field's behavior since the recording, and the first write it makes is still caught by the save's record check or by the next step's checks. @@ -74,29 +98,52 @@ check unless that policy exempts `local_edit` clicks (an open question below). are part of the content digest), so a later edit to a classification changes the digest and needs a new admission. -## Open questions for the founder - -1. Accept the residual risk above for these three patterns? -2. Should `clinical-write`'s `prohibit_unarmed_clicks` exempt a `local_edit` - tab click (it writes nothing and stays in the form), or must a person or the - compiler still arm it with an identity check? Today it must be armed. -3. Without network capture, `Control+a` is classified on the structural and - URL evidence alone (select-all in a focused text field isn't a submission - key). `Enter` and tab clicks need network capture. Is that the right line? - -## Implementation sketch - -- `openadapt_flow.risk.local_edit_classifications(steps, events, network)`: - the three rules, returning `{step_id: (explanation, evidence)}`. -- `compile_recording` applies them after the per-step inference and before - `risk_overrides`, and adds the outcome checks. -- Two postcondition kinds: `local_edit` (structural: focus and URL, title, - page count) and `field_value` (the focused dropdown's label equals a run - parameter). Both are evaluated by the runtime like the existing structural - kinds, and both refuse rather than pass when the backend can't observe them - (unlike `url_changed`, which passes unverified). -- Qualification's draft classification reads the same evidence, so a - compiled `local_edit` step drafts as `reversible` instead of `unknown`. -- Tests: each pattern classifies with full evidence; each missing piece of - evidence leaves the step for a person; each outcome check halts when the - action did something else (focus moved, URL changed, wrong option). +## Decisions + +1. The residual risk above is accepted for these patterns (founder, + 2026-10-19). +2. `clinical-write`'s `prohibit_unarmed_clicks` still applies to a classified + tab click. On the OpenEMR referral, the Contact tab (`step_015`) still needs + an identity check or a person. +3. Network capture is required for every pattern, `Control+a` included. + +## Implementation + +- `openadapt_flow.compiler.local_edit.local_edits(steps, events, network)` + returns the classified steps with their explanation and checks. +- `compile_recording` applies them after the per-step risk inference and the + confirmation-dialog pairing, and before `risk_overrides`, which still win. + Effect mining runs after both. +- `openadapt_flow.risk.is_local_edit` and `is_sealed_dismiss` let the run + gate's risk re-inference read a classified step from what the bundle seals: + the explanation and its runtime checks. +- Postcondition kinds `local_edit` and `field_value`, evaluated by the + replayer (`_local_edit_held`, `_field_value_shown`). +- Tests: `tests/test_local_edit_classification.py`. Each pattern has a + positive case and a case where a write request was sent, which must + refuse. + +## OpenEMR referral, re-admitted on this branch + +Run gate on the recorded referral with the API step admitted +(`--identity subject_name=lname`), compared with `main`: + +| Policy | Violations on `main` | With local edits | +| --- | --- | --- | +| `permissive` | 12 | 1 | +| `clinical-write` | 32 | 9 | + +What remains: + +- `step_007` (`Control+a`): OpenEMR's background + `POST /library/ajax/execute_background_services.php` landed in its window, + so the strict rule leaves it for a person. Under `clinical-write` it also + has no outcome check. +- `step_015` (the Contact tab): classified, but unarmed under + `clinical-write`'s `prohibit_unarmed_clicks`. +- `step_039` (Create New Patient): unarmed, no identity contract and no + record check on its GUI path. +- `step_039` and `step_040`: no duplicate-prevention (idempotency) key. That + stays a person's choice. +- Encryption at rest, under the default and regulated profiles: seal the + bundle. diff --git a/packages/flow/openadapt_flow/compiler/local_edit.py b/packages/flow/openadapt_flow/compiler/local_edit.py index b48858e7e..446f82543 100644 --- a/packages/flow/openadapt_flow/compiler/local_edit.py +++ b/packages/flow/openadapt_flow/compiler/local_edit.py @@ -2,40 +2,54 @@ Proposal: docs/design/LOCAL_EDIT_CLASSIFICATION.md. A compiled program flags every special key and every unlabelled or navigation-shaped click for a -person's review, because the step alone can't prove its risk. Three patterns +person's review, because the step alone can't prove its risk. A few patterns recur in every form-filling program, and the recording can prove them local: - ``select_all``: Control+A (or Meta+A) in a text field, right before typing into it; - ``confirm_option``: Enter right after typing an option's label into a dropdown; -- ``in_form_tab``: a click on a tab of the same form. +- ``confirm_entry``: Enter right after typing into a text field; +- ``in_form_tab``: a click on a tab of the same form; +- ``dismiss_dialog``: the dismiss of an informational dialog ("Continue to + tasks", "Got it", "Close"). A step is classified only when every piece of evidence holds: the field's structural role, the steps around it, URL, title and page count unchanged -across the action, and no write request in its window (network capture is -required for Enter and for tab clicks, since either can submit). Anything +across the action, the record store unchanged when the recorder snapshotted +it, and network capture showing no write request in its window. Anything missing or mixed leaves the step for a person. A classified step gets an -outcome check (``local_edit``, and ``field_value`` for a dropdown) that halts -the run when, at run time, the action did anything else. +outcome check (``local_edit``, plus ``field_value`` after Enter, or the +dismissed label gone after a dismiss) that halts the run when, at run time, +the action did anything else. Model-free and deterministic. """ from __future__ import annotations +import re from dataclasses import dataclass, field from typing import Any, Literal, Optional, Sequence from openadapt_flow.ir import ActionKind, Postcondition, PostconditionKind, Step -from openadapt_flow.risk import LOCAL_EDIT_PREFIX, is_local_edit +from openadapt_flow.risk import ( + DISMISS_PREFIX, + LOCAL_EDIT_PREFIX, + infer_step_risk, + is_local_edit, + only_weak_write_words, + step_text, +) __all__ = ["LOCAL_EDIT_PREFIX", "LocalEdit", "is_local_edit", "local_edits"] TEXT_ROLES = frozenset({"textbox", "searchbox", "spinbutton"}) CHOICE_ROLES = frozenset({"combobox", "listbox"}) _READ_METHODS = frozenset({"GET", "HEAD", "OPTIONS"}) -Pattern = Literal["select_all", "confirm_option", "in_form_tab"] +Pattern = Literal[ + "select_all", "confirm_option", "confirm_entry", "in_form_tab", "dismiss_dialog" +] @dataclass(frozen=True) @@ -107,6 +121,73 @@ def _no_write( return True +#: A dismiss control's label: it closes an informational dialog. ``continue`` +#: and ``next`` are write words for the risk inference; only these two may be +#: reconsidered, and only with the full evidence below. +_DISMISS_LABEL = re.compile( + r"^\s*(?:continue(?:\s+to\s+\w+(?:\s+\w+)?)?|next|close|ok(?:ay)?|got\s+it|" + r"dismiss|not\s+now|maybe\s+later|skip\s+for\s+now)\s*$", + re.IGNORECASE, +) + + +def _store_unchanged(event: dict[str, Any]) -> Optional[bool]: + """The record store snapshots around the step, when the recorder took them. + + True when both were taken and are equal, False when they differ (or only + one was taken), None when neither was taken. + """ + + from openadapt_flow.compiler.effect_mining import SOR_AFTER_KEY, SOR_BEFORE_KEY + + before, after = event.get(SOR_BEFORE_KEY), event.get(SOR_AFTER_KEY) + if before is None and after is None: + return None + return before is not None and before == after + + +def _weak_write_dismiss(step: Step) -> bool: + """An irreversible click that is irreversible only for "continue" or "next". + + Its risk must still be the per-step keyword inference, unchanged: a step a + later rule made irreversible (the click that confirms a dialog, an + icon-only click presumed to save) is never reconsidered, and any other + write word in its context still wins. + """ + + if step.action is not ActionKind.CLICK: + return False + if not _DISMISS_LABEL.match(_label(step) or ""): + return False + if not only_weak_write_words(step_text(step)): + return False + inference = infer_step_risk(step) + return ( + inference.risk == "irreversible" + and inference.explanation == step.risk_explanation + and any(e.startswith("write_keyword:") for e in inference.evidence) + ) + + +def _typed_param( + events: Sequence[dict[str, Any]], step: Optional[Step] +) -> Optional[str]: + if step is None or step.action is not ActionKind.TYPE: + return None + i = _event_index(step) + if i is None or i >= len(events): + return None + value = events[i].get("param") or step.param + return value if isinstance(value, str) and value else None + + +def _frame(step: Optional[Step]) -> Optional[list[str]]: + target = _structural(step) + if target is None: + return None + return list(target.frame_path or []) + + def local_edits( steps: Sequence[Step], events: Sequence[dict[str, Any]], @@ -115,12 +196,22 @@ def local_edits( """The steps the recording proves are local edits, with their checks. ``network`` is the recording's captured requests (``network.jsonl`` - lines), or None when the recording has no capture. + lines), or None when the recording has no capture. Every pattern needs + both halves of the evidence: + + - the recording shows no mutating request in the step's window (network + capture is required), URL, title and page count unchanged, and, when + the recorder snapshotted the record store, the store unchanged; + - a check that the step made only the expected local change, which the + runtime runs every time (focus and field value, or the page unchanged + and the dialog gone). + + Anything short of that is left for a person. """ found: list[LocalEdit] = [] for k, step in enumerate(steps): - if step.risk == "irreversible" or step.effects or step.api_binding: + if step.effects or step.api_binding: continue i = _event_index(step) if i is None or i >= len(events): @@ -128,122 +219,141 @@ def local_edits( event = events[i] if not _unchanged(event): continue + if _no_write(events, i, network) is not True: + continue + if _store_unchanged(event) is False: + continue + if step.risk == "irreversible" and not _weak_write_dismiss(step): + continue prev = steps[k - 1] if k > 0 else None + before_prev = steps[k - 2] if k > 1 else None nxt = steps[k + 1] if k + 1 < len(steps) else None - no_write = _no_write(events, i, network) - if step.action is ActionKind.KEY: - chord = _key_chord(step) - target = _structural(prev) - if ( - chord in {"a+control", "a+meta"} - and prev is not None - and prev.action is ActionKind.CLICK - and target is not None - and target.selector - and (target.role or "").lower() in TEXT_ROLES - and nxt is not None - and nxt.action is ActionKind.TYPE - and no_write is not False - ): - found.append( - LocalEdit( - step.id, - "select_all", - f"{LOCAL_EDIT_PREFIX} selects the text of the field " - f"{prev.id} clicked, before {nxt.id} types into it; " - "URL, title and page count unchanged in the recording" - + ("; no write request" if no_write else ""), - [ - Postcondition( - kind=PostconditionKind.LOCAL_EDIT, - field_selector=target.selector, - ) - ], - ) - ) - continue - before_prev = steps[k - 2] if k > 1 else None - choice = _structural(before_prev) - typed_param = ( - events[_event_index(prev)].get("param") - if prev is not None - and _event_index(prev) is not None - and _event_index(prev) < len(events) # type: ignore[operator] - else None - ) - if ( - chord == "enter" - and prev is not None - and prev.action is ActionKind.TYPE - and before_prev is not None - and before_prev.action is ActionKind.CLICK - and choice is not None - and choice.selector - and (choice.role or "").lower() in CHOICE_ROLES - and isinstance(typed_param, str) - and typed_param - and no_write is True - ): - found.append( - LocalEdit( - step.id, - "confirm_option", - f"{LOCAL_EDIT_PREFIX} confirms the option typed into the " - f"dropdown {before_prev.id} clicked; URL, title and page " - "count unchanged and no write request in the recording", - [ - Postcondition( - kind=PostconditionKind.LOCAL_EDIT, - field_selector=choice.selector, - ), - Postcondition( - kind=PostconditionKind.FIELD_VALUE, - param=typed_param, - ), - ], - ) + edit = ( + _key_edit(step, prev, before_prev, nxt, events) + if step.action is ActionKind.KEY + else _click_edit(step, steps, k) + if step.action is ActionKind.CLICK + else None + ) + if edit is not None: + found.append(edit) + return found + + +_EVIDENCE = "URL, title and page count unchanged and no write request in the recording" + + +def _key_edit( + step: Step, + prev: Optional[Step], + before_prev: Optional[Step], + nxt: Optional[Step], + events: Sequence[dict[str, Any]], +) -> Optional[LocalEdit]: + chord = _key_chord(step) + clicked = _structural(prev) + if ( + chord in {"a+control", "a+meta"} + and prev is not None + and prev.action is ActionKind.CLICK + and clicked is not None + and clicked.selector + and (clicked.role or "").lower() in TEXT_ROLES + and nxt is not None + and nxt.action is ActionKind.TYPE + ): + # The next step's exact typed-input read-back checks the field's + # whole value; this one checks focus stayed in the field. + return LocalEdit( + step.id, + "select_all", + f"{LOCAL_EDIT_PREFIX} selects the text of the field {prev.id} " + f"clicked, before {nxt.id} types into it; {_EVIDENCE}", + [ + Postcondition( + kind=PostconditionKind.LOCAL_EDIT, + field_selector=clicked.selector, ) - continue - if step.action is ActionKind.CLICK: - target = _structural(step) - role = (target.role or "").lower() if target is not None else "" - # The next step that acts on a target (a scroll or wait between - # them has none) must be in the same frame: the form is still - # there. - acting = next( - ( - later - for later in steps[k + 1 : k + 4] - if later.action not in {ActionKind.SCROLL, ActionKind.WAIT} + ], + ) + field = _structural(before_prev) + typed = _typed_param(events, prev) + role = (field.role or "").lower() if field is not None else "" + if ( + chord == "enter" + and typed is not None + and before_prev is not None + and before_prev.action is ActionKind.CLICK + and field is not None + and field.selector + and role in (CHOICE_ROLES | TEXT_ROLES) + ): + kind = "dropdown" if role in CHOICE_ROLES else "field" + return LocalEdit( + step.id, + "confirm_option" if role in CHOICE_ROLES else "confirm_entry", + f"{LOCAL_EDIT_PREFIX} confirms the value typed into the {kind} " + f"{before_prev.id} clicked; {_EVIDENCE}", + [ + Postcondition( + kind=PostconditionKind.LOCAL_EDIT, field_selector=field.selector ), - None, - ) - nxt = acting - same_frame = ( - target is not None - and acting is not None - and _structural(acting) is not None - and list(_structural(acting).frame_path) == list(target.frame_path) - ) - if ( - target is not None - and role in {"tab", "button"} - and no_write is True - and same_frame - and (role == "tab" or not step.expect or _region_only(step)) - ): - found.append( - LocalEdit( - step.id, - "in_form_tab", - f"{LOCAL_EDIT_PREFIX} switches the panel of the same form " - f"({role}); URL, title and page count unchanged, no write " - f"request, and {nxt.id if nxt else 'the next step'} acts in " - "the same frame in the recording", - [Postcondition(kind=PostconditionKind.LOCAL_EDIT)], - ) - ) - return found + Postcondition(kind=PostconditionKind.FIELD_VALUE, param=typed), + ], + ) + return None + + +def _label(step: Step) -> Optional[str]: + target = _structural(step) + if target is not None and target.name: + return str(target.name) + return step.anchor.ocr_text if step.anchor is not None else None + + +def _click_edit(step: Step, steps: Sequence[Step], k: int) -> Optional[LocalEdit]: + target = _structural(step) + if target is None: + return None + role = (target.role or "").lower() + label = _label(step) or "" + if role in {"button", "link"} and _DISMISS_LABEL.match(label): + # An informational dialog's dismiss (MockMed's What's New "Continue + # to tasks"): the page stays, and the control is gone afterwards. + return LocalEdit( + step.id, + "dismiss_dialog", + f"{DISMISS_PREFIX} ({label.strip()!r}); {_EVIDENCE}", + [ + Postcondition(kind=PostconditionKind.LOCAL_EDIT), + Postcondition(kind=PostconditionKind.TEXT_ABSENT, text=label.strip()), + ], + ) + # The next step that acts on a target (a scroll or wait between them has + # none) must be in the same frame: the form is still there. + acting = next( + ( + later + for later in steps[k + 1 : k + 4] + if later.action not in {ActionKind.SCROLL, ActionKind.WAIT} + ), + None, + ) + if ( + role in {"tab", "button"} + and acting is not None + and _frame(acting) is not None + and _frame(acting) == _frame(step) + and (role == "tab" or not step.expect or _region_only(step)) + ): + return LocalEdit( + step.id, + "in_form_tab", + f"{LOCAL_EDIT_PREFIX} switches the panel of the same form ({role}), " + f"and {acting.id} acts in the same frame; {_EVIDENCE}", + [Postcondition(kind=PostconditionKind.LOCAL_EDIT)], + ) + return None def _region_only(step: Step) -> bool: diff --git a/packages/flow/openadapt_flow/risk.py b/packages/flow/openadapt_flow/risk.py index 04c472650..f3b72f4be 100644 --- a/packages/flow/openadapt_flow/risk.py +++ b/packages/flow/openadapt_flow/risk.py @@ -81,6 +81,31 @@ def is_local_edit(step: Step) -> bool: ) +#: The dismiss of an informational dialog. Its label may carry the one pair +#: of write words ("continue", "next") a dismiss commonly uses; any other +#: write word keeps the step irreversible. +DISMISS_PREFIX = LOCAL_EDIT_PREFIX + " dismisses an informational dialog" +_WEAK_WRITE_RE = re.compile(r"^(?:continue(?:s|d|ing)?|next)$", re.IGNORECASE) + + +def only_weak_write_words(text: str) -> bool: + """Every write word in ``text`` is "continue" or "next" (or there is none).""" + + return all(_WEAK_WRITE_RE.match(m.group(0)) for m in _WRITE_RE.finditer(text)) + + +def is_sealed_dismiss(step: Step) -> bool: + """A local-edit dismiss whose checks (page unchanged, label gone) are sealed.""" + + return bool( + is_local_edit(step) + and (step.risk_explanation or "").startswith(DISMISS_PREFIX) + and any(p.kind is PostconditionKind.TEXT_ABSENT and p.text for p in step.expect) + and only_weak_write_words(step_text(step)) + and _matched_token(_FILE_WRITE_RE, step_text(step)) is None + ) + + @dataclass(frozen=True) class RiskInference: """Compile-time classification plus its qualification disposition.""" @@ -751,6 +776,16 @@ def infer_step_risk( ), requires_review=True, ) + if ( + write_keyword is not None + and step.action is ActionKind.CLICK + and is_sealed_dismiss(step) + ): + return RiskInference( + "reversible", + (step.risk_explanation or LOCAL_EDIT_PREFIX)[:512], + (action_evidence, "pattern:local_edit"), + ) if write_keyword is not None and step.action in { ActionKind.CLICK, ActionKind.DOUBLE_CLICK, diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py index de6cfcc8b..e18ca7827 100644 --- a/packages/flow/openadapt_flow/runtime/replayer.py +++ b/packages/flow/openadapt_flow/runtime/replayer.py @@ -16986,24 +16986,35 @@ def _local_edit_held(self, pc: Any, start_state: dict[str, Any]) -> bool: return True def _field_value_shown(self, pc: Any) -> bool: - """The focused dropdown shows this run's value of ``pc.param``.""" + """The focused field shows this run's value of ``pc.param``. + + A dropdown's selected label matches up to spacing and case; a text + field's value must match exactly. Unobservable is a failure. + """ params = getattr(self, "_current_params", None) or {} if not pc.param or pc.param not in params: return False - reader = getattr(self.backend, "focused_select_label", None) - if not callable(reader): - return False - try: - shown = reader() - except Exception: # noqa: BLE001 - unobservable is a failure - return False + expected = runtime_param_text(params[pc.param]) + + def read(name: str) -> Optional[str]: + reader = getattr(self.backend, name, None) + if not callable(reader): + return None + try: + value = reader() + except Exception: # noqa: BLE001 - unobservable is a failure + return None + return value if isinstance(value, str) else None def squash(value: str) -> str: return " ".join(value.split()).casefold() - expected = runtime_param_text(params[pc.param]) - return isinstance(shown, str) and squash(shown) == squash(expected) + label = read("focused_select_label") + if label is not None: + return squash(label) == squash(expected) + value = read("focused_text_value") + return value is not None and value == expected @staticmethod def _describe_postcondition(pc: Any) -> str: diff --git a/packages/flow/tests/test_local_edit_classification.py b/packages/flow/tests/test_local_edit_classification.py index 24c61879c..755e63e2c 100644 --- a/packages/flow/tests/test_local_edit_classification.py +++ b/packages/flow/tests/test_local_edit_classification.py @@ -145,11 +145,23 @@ def test_select_all_without_its_evidence_stays_with_a_person( assert local_edits(steps, events, network=network) == [], why -def test_select_all_needs_no_network_capture(): +def test_select_all_needs_network_capture(): steps = [_click(0, "#lname", "textbox"), _key(1, "Control+a"), _type(2, "lname")] - assert _ids(local_edits(steps, _events(3), network=None)) == { - "step_001": "select_all" - } + assert local_edits(steps, _events(3), network=None) == [] + + +def test_select_all_with_a_write_request_sent_stays_with_a_person(): + steps = [_click(0, "#lname", "textbox"), _key(1, "Control+a"), _type(2, "lname")] + sent = [{"method": "PUT", "t": 1.5, "url": URL + "/draft"}] + assert local_edits(steps, _events(3), network=sent) == [] + + +def test_a_changed_record_store_stays_with_a_person(): + steps = [_click(0, "#lname", "textbox"), _key(1, "Control+a"), _type(2, "lname")] + events = _events(3, e1={"sor_before": {"n": 1}, "sor_after": {"n": 2}}) + assert local_edits(steps, events, network=[]) == [] + same = _events(3, e1={"sor_before": {"n": 1}, "sor_after": {"n": 1}}) + assert _ids(local_edits(steps, same, network=[])) == {"step_001": "select_all"} # -- Enter on a typed dropdown option --------------------------------------------------- @@ -177,10 +189,30 @@ def test_enter_needs_network_capture_and_no_write(): assert _dropdown(e2={"title_after": "Saved"}) == [] -def test_enter_in_a_text_field_stays_with_a_person(): +def _entry(network: Optional[list] = None, **changes): steps = [_click(0, "#q", "textbox"), _type(1, "q"), _key(2, "Enter")] - events = _events(3, e1={"param": "q"}) - assert local_edits(steps, events, network=[]) == [] + events = _events(3, e1={"param": "q"}, **changes) + return local_edits(steps, events, network=[] if network is None else network) + + +def test_enter_after_typing_into_a_text_field_is_a_local_edit(): + (edit,) = _entry() + assert edit.step_id == "step_002" and edit.pattern == "confirm_entry" + kinds = [p.kind for p in edit.postconditions] + assert kinds == [PostconditionKind.LOCAL_EDIT, PostconditionKind.FIELD_VALUE] + assert edit.postconditions[0].field_selector == "#q" + assert edit.postconditions[1].param == "q" + + +def test_enter_in_a_text_field_with_a_write_request_sent_stays_with_a_person(): + assert _entry(network=[{"method": "POST", "t": 2.4, "url": URL}]) == [] + steps = [_click(0, "#q", "textbox"), _type(1, "q"), _key(2, "Enter")] + assert local_edits(steps, _events(3, e1={"param": "q"}), network=None) == [] + + +def test_enter_after_a_literal_or_nothing_typed_stays_with_a_person(): + steps = [_click(0, "#q", "textbox"), _key(1, "Enter")] + assert local_edits(steps, _events(2), network=[]) == [] # -- in-form tabs ------------------------------------------------------------------------- @@ -202,6 +234,24 @@ def test_a_tab_click_needs_capture_the_same_frame_and_no_write(): assert local_edits([steps[0], other], _events(2), network=[]) == [] +def test_a_tab_role_click_in_the_same_frame_is_a_local_edit(): + steps = [ + _click(0, "#tab-contact", "tab", "Contact"), + _click(1, "#street", "textbox"), + ] + (edit,) = local_edits(steps, _events(2), network=[]) + assert edit.pattern == "in_form_tab" + + +def test_a_tab_click_with_a_write_request_sent_stays_with_a_person(): + steps = [ + _click(0, "#tab-contact", "tab", "Contact"), + _click(1, "#street", "textbox"), + ] + sent = [{"method": "PATCH", "t": 0.8, "url": URL + "/autosave"}] + assert local_edits(steps, _events(2), network=sent) == [] + + def test_a_write_shaped_click_is_never_a_local_edit(): save = _click(0, None, "button", "Save") save.risk = "irreversible" @@ -209,6 +259,71 @@ def test_a_write_shaped_click_is_never_a_local_edit(): assert local_edits(steps, _events(2), network=[]) == [] +# -- the dismiss of an informational dialog (#1377) ------------------------------------------ + + +def _infer(step: Step) -> Step: + inference = infer_step_risk(step) + step.risk = inference.risk + step.risk_explanation = inference.explanation + step.risk_review_required = inference.requires_review + return step + + +def _whats_new() -> list[Step]: + # MockMed's What's New notice: "Continue to tasks" closes it, the page stays. + dismiss = _infer(_click(0, "#notice-continue", "button", "Continue to tasks")) + assert dismiss.risk == "irreversible" # "continue" is a write word + return [dismiss, _click(1, "#task-search", "textbox", "Search tasks")] + + +def test_dismissing_an_informational_dialog_is_a_local_edit(): + (edit,) = local_edits(_whats_new(), _events(2), network=[]) + assert edit.step_id == "step_000" and edit.pattern == "dismiss_dialog" + local, gone = edit.postconditions + assert local.kind is PostconditionKind.LOCAL_EDIT and local.field_selector is None + assert gone.kind is PostconditionKind.TEXT_ABSENT + assert gone.text == "Continue to tasks" + + +def test_a_dismiss_with_a_write_request_sent_stays_with_a_person(): + sent = [{"method": "POST", "t": 0.4, "url": URL + "/notices/seen"}] + assert local_edits(_whats_new(), _events(2), network=sent) == [] + assert local_edits(_whats_new(), _events(2), network=None) == [] + moved = _events(2, e0={"url_after": URL + "/tasks"}) + assert local_edits(_whats_new(), moved, network=[]) == [] + + +@pytest.mark.parametrize( + "name", ["Continue to checkout", "Continue and save", "Next: submit", "Save"] +) +def test_a_dismiss_shaped_label_with_another_write_word_stays_irreversible(name): + steps = [_infer(_click(0, "#b", "button", name)), _click(1, "#x", "textbox")] + assert steps[0].risk == "irreversible" + assert local_edits(steps, _events(2), network=[]) == [] + + +def test_the_click_that_confirms_a_dialog_is_never_reconsidered(): + ok = _infer(_click(0, "#ok", "button", "OK")) + ok.risk = "irreversible" + ok.risk_explanation = "confirms the dialog opened by step_000; this click saves" + steps = [ok, _click(1, "#x", "textbox")] + assert local_edits(steps, _events(2), network=[]) == [] + + +def test_the_risk_reinference_reads_a_sealed_dismiss(): + steps = _whats_new() + (edit,) = local_edits(steps, _events(2), network=[]) + dismiss = steps[0] + dismiss.risk_explanation = edit.explanation + dismiss.expect = list(edit.postconditions) + inference = infer_step_risk(dismiss) + assert inference.risk == "reversible" and not inference.requires_review + # Without the label-gone check the write word wins again. + dismiss.expect = [edit.postconditions[0]] + assert infer_step_risk(dismiss).risk == "irreversible" + + # -- risk, tags and the outcome checks ------------------------------------------------------ @@ -260,9 +375,11 @@ def test_unset_local_edit_fields_serialize_as_before(): class _Backend: - def __init__(self, *, url=URL, focused=True, label="Massachusetts") -> None: + def __init__( + self, *, url=URL, focused=True, label="Massachusetts", text=None + ) -> None: self.url, self.page_title, self.page_count = url, "Form", 1 - self._focused, self._label = focused, label + self._focused, self._label, self._text = focused, label, text def focused_element_matches(self, selector): return self._focused @@ -270,6 +387,9 @@ def focused_element_matches(self, selector): def focused_select_label(self): return self._label + def focused_text_value(self): + return self._text + def _replayer(backend): from openadapt_flow.runtime.replayer import Replayer @@ -315,3 +435,112 @@ def test_the_field_value_check_reads_the_dropdown(label, passes): pc = Postcondition(kind=PostconditionKind.FIELD_VALUE, param="state") replayer = _replayer(_Backend(label=label)) assert replayer._postcondition_passes(pc, b"", None, START) is passes + + +@pytest.mark.parametrize( + "text, passes", + [ + ("Massachusetts", True), + ("massachusetts", False), + ("Massachusetts ", False), + (None, False), + ], +) +def test_the_field_value_check_reads_a_text_field_exactly(text, passes): + pc = Postcondition(kind=PostconditionKind.FIELD_VALUE, param="state") + replayer = _replayer(_Backend(label=None, text=text)) + assert replayer._postcondition_passes(pc, b"", None, START) is passes + + +# -- compiled end to end: the What's New dismiss gets no record_written (#1377) --------------- + + +def _whats_new_recording(tmp_path, *, dismiss_post_t: Optional[float]): + import json + + from openadapt_flow.compiler.effect_mining import SOR_AFTER_KEY, SOR_BEFORE_KEY + + page = { + "url_before": URL, + "url_after": URL, + "title_before": "MockMed", + "title_after": "MockMed", + "pages_before": 1, + "pages_after": 1, + } + chart = [{"id": "e1", "type": "Encounter", "patient_id": "p1"}] + events = [ + { + "i": 0, + "kind": "click", + "x": 129, + "y": 236, + "t": 1.0, + "structural": { + "selector": "#notice-continue", + "role": "button", + "name": "Continue to tasks", + }, + SOR_BEFORE_KEY: [], + SOR_AFTER_KEY: [], + **page, + }, + { + "i": 1, + "kind": "click", + "x": 300, + "y": 400, + "t": 3.0, + "structural": { + "selector": "#save", + "role": "button", + "name": "Save encounter", + }, + SOR_BEFORE_KEY: [], + SOR_AFTER_KEY: chart, + **page, + }, + ] + rec = tmp_path / "recording" + (rec / "frames").mkdir(parents=True) + from PIL import Image + + for i in range(len(events)): + for side in ("before", "after"): + Image.new("RGB", (1280, 800), "white").save( + rec / "frames" / f"{i:04d}_{side}.png" + ) + (rec / "events.jsonl").write_text("".join(json.dumps(e) + "\n" for e in events)) + (rec / "meta.json").write_text(json.dumps({"params": {}})) + lines = [{"method": "POST", "url": URL + "/api/encounters", "t": 3.2, "seq": 1}] + if dismiss_post_t is not None: + lines.insert( + 0, + {"method": "POST", "url": URL + "/api/seen", "t": dismiss_post_t, "seq": 0}, + ) + header = {"schema": "openadapt.network-capture/1", "exchanges": len(lines)} + (rec / "network.jsonl").write_text( + "".join(json.dumps(x) + "\n" for x in [header, *lines]) + ) + return rec + + +def test_a_compiled_whats_new_dismiss_gets_no_record_written(tmp_path): + from openadapt_flow.compiler.compile import compile_recording + from openadapt_flow.runtime.effects import EffectKind + + rec = _whats_new_recording(tmp_path, dismiss_post_t=None) + wf = compile_recording(rec, tmp_path / "bundle", name="notice", mine_effects=True) + dismiss, save = wf.steps + assert dismiss.risk == "reversible" and not dismiss.risk_review_required + assert is_local_edit(dismiss) and dismiss.effects == [] + assert EffectKind.RECORD_WRITTEN in [e.kind for e in save.effects] + + +def test_a_compiled_dismiss_that_sent_a_write_keeps_its_risk(tmp_path): + from openadapt_flow.compiler.compile import compile_recording + + rec = _whats_new_recording(tmp_path, dismiss_post_t=1.1) + wf = compile_recording(rec, tmp_path / "bundle", name="notice", mine_effects=True) + dismiss = wf.steps[0] + assert dismiss.risk == "irreversible" and not is_local_edit(dismiss) From c7851a03abe0f94c5df79b5909602acaf1a18a9f Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 19:46:32 -0400 Subject: [PATCH 13/13] batch api: register #1374's benchmark results in the artifact inventory Co-Authored-By: Claude Opus 5.5 --- packages/flow/public-artifacts.json | 42 +++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/packages/flow/public-artifacts.json b/packages/flow/public-artifacts.json index 62b620b55..787f0074c 100644 --- a/packages/flow/public-artifacts.json +++ b/packages/flow/public-artifacts.json @@ -189,6 +189,48 @@ "benchmark/o2c_recon/results.json", "benchmark/openemr/latency_cost.png", "benchmark/openemr/results.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-1-r1-nadia-brennan.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-2-r2-marcus-feld.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-3-r3-ines-carvalho.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-api-4-r4-theo-lindqvist.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-1-r1-nadia-brennan.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-2-r2-marcus-feld.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-3-r3-ines-carvalho.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b01-browser-4-r4-theo-lindqvist.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-1-r5-grace-adeyemi.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-2-r6-rosa-delgado.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-3-r7-arlo-whitcombe.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-api-4-r8-priya-raman.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-1-r5-grace-adeyemi.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-2-r6-rosa-delgado.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-3-r7-arlo-whitcombe.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b02-browser-4-r8-priya-raman.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-1-r1-nadia-brennan.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-2-r2-marcus-feld.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-3-r3-ines-carvalho.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-api-4-r4-theo-lindqvist.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-1-r1-nadia-brennan.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-2-r2-marcus-feld.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-3-r3-ines-carvalho.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b03-browser-4-r4-theo-lindqvist.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-1-r5-grace-adeyemi.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-2-r6-rosa-delgado.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-3-r7-arlo-whitcombe.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-api-4-r8-priya-raman.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-1-r5-grace-adeyemi.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-2-r6-rosa-delgado.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-3-r7-arlo-whitcombe.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b04-browser-4-r8-priya-raman.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-1-r1-nadia-brennan.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-2-r2-marcus-feld.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-3-r3-ines-carvalho.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-api-4-r4-theo-lindqvist.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-1-r1-nadia-brennan.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-2-r2-marcus-feld.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-3-r3-ines-carvalho.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/items/b05-browser-4-r4-theo-lindqvist.json", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/runs.csv", + "benchmark/openemr_api_tier/results/approval_durable/2026-10-19-combined/summary.json", "benchmark/openemr_api_tier/spike/api_bundle/compile_report.json", "benchmark/openemr_api_tier/spike/api_bundle/manifest.json", "benchmark/openemr_api_tier/spike/api_bundle/session_form.json",