feat(qualify): declarative qualify from verify.yaml (carry-over of openadapt-flow#532)

packages/flow/README.md

@@ -380,6 +380,10 @@ The full list, capability by capability: [docs/LIMITS.md](docs/LIMITS.md).
 
 ## More documentation
 
+- [Qualify a workflow](docs/QUALIFY_A_WORKFLOW.md): `openadapt-flow qualify`
+  runs a bundle against the normal case and deliberate faults, reads the write
+  back from the system of record, and certifies it under a policy. Terms are
+  defined in [docs/QUALIFICATION_TERMS.md](docs/QUALIFICATION_TERMS.md).
 - [Production setup](docs/PRODUCTION.md): qualifying a bundle from its
   recording, policies, multi-step processes, signed results, and release
   status.

packages/flow/benchmark/openemr_local/qualify_hooks.py

@@ -0,0 +1,123 @@
+"""Sign-in, reset and read-back session for the local OpenEMR fixture.
+
+``benchmark/openemr_local/verify.yaml`` names these three functions. They are
+environment setup for a made-up-patient test system, not workflow logic:
+OpenEMR's sign-in opens the new-patient form through its tab shell, the
+fixture resets by restoring a database snapshot, and the read-back client
+uses an OAuth password grant that a static header cannot express.
+
+Environment (all optional):
+
+- ``OPENEMR_FIXTURE_STATE``: fixture state directory
+  (default ``benchmark/openemr_local/state``);
+- ``OPENEMR_FIXTURE_LOCK``: a different environment lock file;
+- ``OPENEMR_UI_URL`` / ``OPENEMR_API_URL``: non-default ports.
+"""
+
+from __future__ import annotations
+
+import atexit
+import os
+import time
+from pathlib import Path
+from typing import Any, Callable
+
+HERE = Path(__file__).resolve().parent
+_fixture: Any = None
+_browser: Any = None
+
+
+def fixture() -> Any:
+    """The fixture named by the environment, created once per process."""
+
+    global _fixture
+    if _fixture is not None:
+        return _fixture
+    import benchmark.openemr_local.fixture as fx
+
+    lock = os.environ.get("OPENEMR_FIXTURE_LOCK")
+    if lock:
+        fx.LOCK_PATH = Path(lock)
+    ui = os.environ.get("OPENEMR_UI_URL")
+    api = os.environ.get("OPENEMR_API_URL")
+
+    class _Fixture(fx.OpenEMRFixture):
+        @property
+        def ui_base_url(self) -> str:
+            return ui or super().ui_base_url
+
+        @property
+        def api_base_url(self) -> str:
+            return api or super().api_base_url
+
+        def _validate_runtime_env(self) -> None:
+            if ui or api:  # non-default ports: the runtime env names them
+                self.runtime_env.chmod(0o600)
+                return
+            super()._validate_runtime_env()
+
+    _fixture = _Fixture(Path(os.environ.get("OPENEMR_FIXTURE_STATE", HERE / "state")))
+    return _fixture
+
+
+def _shared_browser() -> Any:
+    global _browser
+    if _browser is None:
+        from playwright.sync_api import sync_playwright
+
+        from openadapt_flow._browser_setup import ensure_chromium_installed
+
+        ensure_chromium_installed()
+        pw = sync_playwright().start()
+        browser = pw.chromium.launch(headless=True)
+        _browser = (pw, browser)
+
+        def _stop() -> None:
+            try:
+                browser.close()
+            finally:
+                pw.stop()
+
+        atexit.register(_stop)
+    return _browser[1]
+
+
+def open_session(config: Any) -> tuple[Any, Callable[[], None]]:
+    """A fresh signed-in context at the new-patient form."""
+
+    from scripts.openemr_local_demo import (
+        OpenEMRPlaywrightBackend,
+        _preauthenticate_browser,
+    )
+
+    fx = fixture()
+    width, height = config.app.viewport
+    context = _shared_browser().new_context(
+        viewport={"width": width, "height": height}, device_scale_factor=1
+    )
+    page = context.new_page()
+    page.goto(fx.ui_base_url)
+    backend = OpenEMRPlaywrightBackend(page)
+    _preauthenticate_browser(backend, fx)
+    return backend, context.close
+
+
+def reset() -> None:
+    """Restore the verified baseline database."""
+
+    fixture().reset()
+
+
+def oracle_session(config: Any, attempts: int = 5) -> Any:
+    """A read-only OAuth client; retries transport errors right after a reset."""
+
+    import httpx
+
+    for attempt in range(1, attempts + 1):
+        try:
+            return fixture().token_session("oracle")
+        except httpx.TransportError:
+            if attempt == attempts:
+                raise
+            time.sleep(2.0 * attempt)
+    raise AssertionError("unreachable")

packages/flow/benchmark/openemr_local/verify.yaml

@@ -0,0 +1,60 @@
+# Qualify the OpenEMR patient-registration workflow on the local fixture.
+#
+#   PYTHONPATH=. openadapt-flow qualify path/to/bundle \
+#     --verify benchmark/openemr_local/verify.yaml --out runs/openemr-qual \
+#     --certify clinical-write
+#
+# Made-up patients only. See docs/QUALIFY_A_WORKFLOW.md.
+schema_version: openadapt.verify/v1
+environment_id: openemr-local-fixture-7c96c8eefe46
+minimum_effect_strength: independent_system_of_record
+
+app:
+  target: web
+  name: OpenEMR (local fixture, made-up patients)
+  url: ${OPENEMR_UI_URL:-http://127.0.0.1:9301}
+  version: openemr@7c96c8eefe46 (benchmark/openemr_local lock)
+  # OpenEMR's sign-in opens the form through its tab shell, and the fixture
+  # resets by restoring a database snapshot; both are small functions.
+  session: benchmark.openemr_local.qualify_hooks:open_session
+  reset:
+    python: benchmark.openemr_local.qualify_hooks:reset
+
+# Read the patient back through OpenEMR's REST API with a separate,
+# read-only OAuth client. This is the existing RestRecordVerifier.
+verifier:
+  kind: rest
+  base_url: ${OPENEMR_API_URL:-https://127.0.0.1:9300}
+  records_path: /apis/default/api/patient?_limit=1000
+  records_key: data
+  session: benchmark.openemr_local.qualify_hooks:oracle_session
+
+# What the save writes; expands to the effect contract list (one
+# record_written plus a field_equals per field). A bare value is a parameter.
+record:
+  name: patient
+  match: [email, lname]
+  duplicate_key: email   # OpenEMR has no idempotency column; one chart per email
+  fields:
+    email: email
+    lname: lname
+    title: title
+    fname: fname
+    DOB: DOB
+    sex: sex
+    street: street
+    city: city
+    state: {literal: MA}  # the select shows "Massachusetts"; the record stores MA
+    postal_code: postal_code
+    phone_home: phone_home
+    country_code: country_code
+
+save_step: auto
+
+inputs:
+  source: recorded
+
+campaign:
+  id: openemr-local-patient-registration-v2
+  trials: 3
+  policy: clinical-write

packages/flow/docs/QUALIFICATION_TERMS.md

@@ -0,0 +1,126 @@
+# Qualification terms
+
+This page defines the words `openadapt-flow qualify` uses. Other pages link
+here instead of repeating them.
+
+## Qualify
+
+Qualifying a workflow means running the compiled bundle against the normal
+case and a set of deliberate faults before it may run unattended on a real
+system. It passes only when:
+
+- the normal case completes, and the write is confirmed by reading it back
+  from the system of record (the application's API or database), not from the
+  screen;
+- every fault case halts before anything is written.
+
+The faults are:
+
+| Case | What changes | Must |
+|---|---|---|
+| `representative` | nothing | complete, verified by read-back |
+| `fault-ambiguity` | a second copy of the save button appears | halt |
+| `fault-wrong-identity` | the save button is relabeled to a different action | halt |
+| `fault-stale-identity` | the screen goes blank at the save | halt |
+| `fault-weak-effect` | the write check is swapped for one that only reads the screen | halt |
+| `fault-missing-effect` | the write check is removed | halt |
+
+Each case runs several times (three by default). After each trial an
+independent oracle reads the system of record and checks that the normal case
+wrote exactly one correct record and that no fault case wrote anything.
+
+**When you need it:** before a workflow makes consequential or irreversible
+writes (creating a patient, submitting a claim, sending a payment) without a
+person watching. It is optional for read-only tasks.
+
+**What you get:** a signed report (`qualification/report.json`) and a bundle
+whose `workflow.json` carries the qualification project and its signed case
+results. With certification (below), the runner admits that bundle for
+unattended runs.
+
+**What it does not prove:** anything beyond the environment it was qualified
+in. The report records the application version, browser, viewport and
+runtime. A different application version, a changed screen, or a different
+system of record needs a new qualification. A pass also does not show the
+workflow handles inputs or screens that the cases did not exercise.
+
+## Certify
+
+Certifying adds one step to qualifying: the result is checked against a
+certification policy and signed under it. `openadapt-flow qualify ...
+--certify clinical-write` refuses to certify unless all six cases ran and
+passed and the bundle satisfies every rule of the policy. Nothing is marked
+certified when it refuses; the refusal reasons are in
+`qualification/verdict.txt`.
+
+What is signed is `qualification/result.json`
+(`openadapt.qualification-result/v1`, RFC-0002): the coverage report without
+free text, the digest of the read-back contract it was judged against, and
+one read-back verdict per trial. Its digest goes into the existing
+qualification admission envelope; there is no separate signing scheme.
+
+The keys `qualify` signs with are local test keys. Case results are signed
+with runner and fault-driver keys made for one campaign and held in memory.
+The admission envelope is signed with Flow's well-known local-dev
+qualification key, which every production trust map refuses. None of them is
+a production trust root; a production deployment admits results signed by its
+own authority (see [PRODUCTION.md](PRODUCTION.md)).
+
+## Certification policy
+
+A certification policy is a named set of rules a bundle must meet before it is
+certified. The rules live in a YAML file and are enforced by
+`openadapt_flow/policy.py`. They cover:
+
+- **identity**: which steps must check that the target on screen is the right
+  one before acting (`require_identity_for`, `prohibit_unarmed_clicks`);
+- **effect verification**: which steps must confirm their write in the system
+  of record, not only on screen (`require_system_effects_for`,
+  `require_effects_for_irreversible`, `require_effect_verification_for`), and
+  which need an idempotency key so a retry cannot write twice
+  (`require_idempotency_key_for`);
+- **screen checks**: which steps must also check the screen afterwards
+  (`require_screen_postconditions_for`, `prohibit_vacuous_postconditions`,
+  `max_unverified_steps`);
+- **operator review**: effect bindings the compiler could not derive must be
+  confirmed by a person (`prohibit_unconfirmed_effect_bindings`), and steps
+  found on thin evidence need sign-off
+  (`require_human_approval_below_confidence`).
+
+Which faults must halt, the minimum effect tier and the trial count are not
+policy rules today. `qualify` always requires all six cases for
+certification, sets the minimum effect tier to the tier of the read-back you
+configure (a REST or SQL read-back is the strongest, "independent system"),
+and runs `campaign.trials` trials per case (three by default).
+
+### Built-in policies
+
+| Policy | Requires | Use it for |
+|---|---|---|
+| `clinical-write` | identity on every navigation and write step; no unchecked clicks; a system-of-record check and an idempotency key on every write; screen checks on write, save, submit, create, update, delete, confirm and send steps; no step without a check; no unconfirmed effect bindings; sign-off for steps found with confidence below 0.8 | unattended writes to a system of record: EHR, claims, billing, anything where a wrong or duplicate write is costly |
+| `permissive` | only that write steps carry some check of their outcome | development, CI smoke tests, read-only tasks |
+
+The `qualify propose` and `qualify from-demo` commands also take a
+`--policy-pack` that bundles a policy with a minimum effect tier and signer:
+
+| Pack | Policy | Minimum effect tier | Signer |
+|---|---|---|---|
+| `community` | `permissive` | persisted-state reacquisition | local dev signer allowed |
+| `cloud` | `clinical-write` | independent session | production signer |
+| `regulated` | `clinical-write` | independent system | production signer |
+
+### How to choose
+
+Use `clinical-write` whenever the workflow writes to a system of record and
+will run without a person approving each write. Use `permissive` while you are
+building the workflow, or for tasks that only read.
+
+### Your own policy
+
+You can write your own. `--certify` takes a built-in name or a path to a YAML
+file with the same keys; unknown keys are rejected. Start from a copy of
+`openadapt_flow/policies/clinical-write.yaml`, and tighten rather than loosen
+it for production writes.
+
+See also: [Qualify a workflow](QUALIFY_A_WORKFLOW.md),
+[QUALIFICATION_PROJECT.md](QUALIFICATION_PROJECT.md).