diff --git a/CAPABILITIES.md b/CAPABILITIES.md index c9125ded3..20e8806f6 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -181,6 +181,7 @@ improvising or retrying when something is off. | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | | Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | | Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-an-approval-mode) | | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | diff --git a/llms-full.txt b/llms-full.txt index 623dfd5b6..a2cf209b6 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -375,7 +375,8 @@ shipping in the next release; a pull request number means it is being merged. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles the verified run into a program (`openadapt_flow.jit`, released; - prompt-to-program `compile-agent`, landing). Agents can drive the recording + prompt-to-program `compile-agent`, landing; the agent can act through code + instead of screenshots, [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)). Agents can drive the recording on web, native Windows, RDP and Citrix. 2. **Compiles to the fastest layer the app allows.** Page structure first (DOM, Windows UI Automation, macOS Accessibility), direct API calls where the app's @@ -436,9 +437,10 @@ shipping in the next release; a pull request number means it is being merged. | Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | +| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. It is arm 7 of the scoreboard; its figures are quoted from there once its rows are in. | `CODE_AS_ACTION.md` in [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | -| Ships as one monorepo with separately installable packages (flow, agent, types, capture, and privacy and desktop moving in). | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | +| Ships as one monorepo with separately installable packages (flow, agent, types, capture, privacy and desktop), each released from the monorepo. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | | Makes no model calls on healthy runs. Model training (openadapt-ml) is separate research. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | ## The full map @@ -474,6 +476,7 @@ improvising or retrying when something is off. | Compile one or more demonstrations into a deterministic program | Released | Real app (OpenEMR referral, 62 timed runs) | `openadapt-flow record`, `compile`; [packages/flow](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/README.md) | | Right-record check before any write (structure, then OCR, then pixels) | Released | Real app and synthetic | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md), `runtime/identity.py` | | Read the saved record back from the system of record (API, database, FHIR, separate session) | Released | Real app (OpenEMR 12 of 12 fields) | [VERIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION.md), `runtime/effects/` | +| Read values out of an app as a workflow step (READ), with a records format, a signed review queue and export to CSV or a database | On main ([#1229](https://github.com/OpenAdaptAI/OpenAdapt/pull/1229), [#1215](https://github.com/OpenAdaptAI/OpenAdapt/pull/1215), [#1218](https://github.com/OpenAdaptAI/OpenAdapt/pull/1218)–[#1224](https://github.com/OpenAdaptAI/OpenAdapt/pull/1224)) | Synthetic (mock desktop apps) | `openadapt_flow/interchange/` | | Stop before the effect instead of improvising (`HALTED_BEFORE_EFFECT`) | Released | Real app and synthetic | `runtime/replayer.py` | | Never retry an uncertain save (`RECONCILIATION_REQUIRED`) | Released | Synthetic (fault server) | [VERIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION.md) | | One check-level scale, 0 to 5, from "unchecked" to "counterparty confirmed" | Released | Unit tests | `openadapt_flow/check_level.py` | @@ -508,7 +511,7 @@ improvising or retrying when something is off. | Watch many real runs in the background and mine candidate branches (shadow mode) | On main ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)) | Synthetic | [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Fill a workflow's inputs from a referral PDF, fax image or email | Released | Synthetic | `openadapt-flow intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | | Use spoken narration while recording to propose the rule behind a branch | In review ([#1291](https://github.com/OpenAdaptAI/OpenAdapt/pull/1291)) | Synthetic (generated audio) | `openadapt-capture[audio,privacy]` | -| Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | In review ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276)) | Synthetic (MockMed) | `induce --probe-on` | +| Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | On main ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276), [#1289](https://github.com/OpenAdaptAI/OpenAdapt/pull/1289)) | Synthetic (MockMed; desktop and remote test copies) | `induce --probe-on` | | Route each item to the right verified variant of a program | On main ([#1264](https://github.com/OpenAdaptAI/OpenAdapt/pull/1264)) | Synthetic | `variants`, `route` | ## Surfaces @@ -519,7 +522,8 @@ improvising or retrying when something is off. | Native Windows (UI Automation) | Released, early | Synthetic (WinForms, 3 of 3 trials) | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md) | | Native macOS (Accessibility) | Released, early | Synthetic (TextEdit, 3 of 3 trials) | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md) | | RDP (pixels; Aardwolf or headless FreeRDP with a session pool) | Released, early | Real Windows 11 host, 3 of 3, plus a FreeRDP batch | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md), `backends/xfreerdp.py` | -| Citrix | Exploratory | A no-DOM stand-in only; no real ICA/HDX session yet | [LIMITS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md) | +| Citrix and other virtual desktops, with OpenAdapt installed inside the session (`in_session`): the same structure-first path as a local desktop | Released, early | Same evidence as the Windows path; no live Citrix session yet | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md) | +| Citrix from outside the session, pixels only (`remote_display`), the fallback when nothing can be installed inside | Exploratory | A no-DOM stand-in only; no real ICA/HDX session yet | [LIMITS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md) | | Chain programs across surfaces, passing confirmed values | Released | Synthetic | `openadapt-flow compose`, `process`; [PROCESS_CONTRACT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/PROCESS_CONTRACT.md) | | One program that switches surfaces, tabs and popups mid-run | On main ([#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197)) | Real Chromium; Windows switch on a mock | [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Halt when a workflow qualified with page structure runs on a pixels-only surface | Released | Synthetic | `openadapt-flow qualify` | @@ -533,7 +537,11 @@ improvising or retrying when something is off. | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | | Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | | Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | -| Run a worklist on several workers at once, one signed-in session each, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--workers` | +| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-an-approval-mode) | +| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| Record where each step's time goes in every run report | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | ## When the app changes @@ -544,7 +552,7 @@ improvising or retrying when something is off. | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | In review ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | | Self-repair across several new steps (a new dialog, a control moved into a menu) | In progress | Synthetic (MockMed drift modes) | bridge branch, stacked on #1265 | -| Keep pixel-path runs working across display scaling and font smoothing changes | In review ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (RDP and Citrix-style renderings) | `vision/match.py` | +| Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | In review ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | | Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) | @@ -553,9 +561,11 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | +| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | -| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Synthetic | `compile-agent` | +| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | +| Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | | One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | ## Recording, privacy and proof @@ -565,6 +575,7 @@ improvising or retrying when something is off. | Record mouse, keyboard, screen video and accessibility trees on one clock (Windows, macOS, Linux X11) | Released (capture 1.3.0) | Real OS | [packages/capture](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/README.md); Wayland is not supported | | Find and redact personal and health data in text, screenshots and structure | Released (privacy 1.1.3) | Unit tests | [packages/privacy](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/privacy/README.md), [PRIVACY.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/PRIVACY.md) | | Scrub by default before anything leaves the machine | On main ([#1262](https://github.com/OpenAdaptAI/OpenAdapt/pull/1262)) | Unit tests | `openadapt-privacy` | +| Each package installs on its own within a size budget, checked in CI (privacy without desktop, capture without plotting) | On main ([#1270](https://github.com/OpenAdaptAI/OpenAdapt/pull/1270), [#1271](https://github.com/OpenAdaptAI/OpenAdapt/pull/1271), [#1279](https://github.com/OpenAdaptAI/OpenAdapt/pull/1279)) | CI | `.github/workflows/packages.yml` | | Signed receipts (Seals), countersigned and timestamped, offline by default | Released | Unit tests, real TSA tokens | `openadapt-flow seal`, `countersign`, `local-seal` | | Strict no-outbound mode and an unattended on-prem service | Released | Synthetic | `openadapt-flow no-outbound`, `service`; [ON_PREM.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/ON_PREM.md) | @@ -574,7 +585,7 @@ Each `openadapt-flow` command, grouped by the capability it serves. A test fails when a command is added without being listed here. - **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`. -- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. +- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. - **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`. - **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). - **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). @@ -962,6 +973,31 @@ class named with `--allow-halt`. A handoff copies an effect-bound parameter from a confirmed receipt. Missing evidence HALTs. Subflows and worklists stay inside each child, on that child's surface. +## Parallel worklists + +`--workers N` runs a worklist's items on N signed-in sessions at once +([`THROUGHPUT.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md)). Every item keeps every check. The run +refuses N above the deployment's `worklist.max_workers`, fewer accounts than +workers, two accounts for the same user, two workers sharing one session, a +program with steps outside its loop, and rows not declared +`items_independent`. It can't tell on its own whether two rows touch the same +record; that declaration is yours. A halt on any worker stops new items on +all of them, and an item claimed before a crash is never run again +automatically: it is marked `uncertain` for a person to check. The pipelined +read-back window applies only to an item's last save and only to read-backs +that don't use the screen; durable runs, which include the Standard and +Regulated profiles, always read back inline. Parallel RDP runs use the +FreeRDP session pool and have unit tests only. Throughput figures come from +one machine and a local OpenEMR test copy, not from a production EMR. + +## Short parameter values + +The right-record check binds a typed value into its expected text only when +the value has at least 4 characters. An item whose bound value is shorter, +such as the first name "Bo", halts at the identity check instead of saving. +This is deliberate: a short value matches too much unrelated text to identify +a record. + ## Save triage `openadapt-flow triage` orders verified saves for review under @@ -1296,12 +1332,16 @@ grade is `screen_readback`, never `independent_readback`. **Without a read that is independent of the acting session, the run cannot be `VERIFIED`.** That read is either a system-of-record read (REST, FHIR, SQL, -file; tier 1) or the application's record page read in a separate signed-in -session (`ui_readback` for web, `fresh_session` for desktop and RDP, -tested on mocks only; tier 2). The second goes through the same -application, so it is not a system-of-record read and has its own limits -([`EFFECT_KIT.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-ui-read-back-ui_readback)). -A pixel-only Citrix or RDP session with neither is that case. Reopening the +file; tier 1) or the application's own screens read in a separate signed-in +session (tier 2): a record page in a separate browser (`ui_readback`), a +relaunched desktop app or a second RDP session (`fresh_session`, tested on +mocks only), or a look-up in a separate desktop, RDP or Citrix session +(`desktop_readback`, tier 2 with its own RDP logon). None of them goes around +the application, so none is a system-of-record read, and each has its own +limits ([`EFFECT_KIT.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-ui-read-back-ui_readback), +[`desktop_readback`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback)). +If a person can look the record up, the check can too. A pixel-only Citrix or +RDP deployment that declares no such check is the remaining case. Reopening the record in the acting session (tier 3) is reported as its own check level, `reopened` ("Reopened in the same session"), and is not `VERIFIED`: a confirmed reopen ends `COMPLETED_UNVERIFIED`, and the agent result says diff --git a/packages/flow/docs/API_TIER.md b/packages/flow/docs/API_TIER.md index abd6d0b91..560c78a0a 100644 --- a/packages/flow/docs/API_TIER.md +++ b/packages/flow/docs/API_TIER.md @@ -100,14 +100,46 @@ RECONCILIATION_REQUIRED` unless the record check proves the effect. The record check (`effects`) then confirms the save with the same verifier a GUI save uses. +## Under an approval mode + +Under `each_item` (the default) and `problems_only`, the person approves the +request the API step sends, not a screenshot of the form. The API step fills +no form, so a screenshot would show an empty one. + +1. Before it asks, the run prepares the request: it reads the form page, + builds the body and runs the pre-checks. Only reads are sent. If the + request can't be prepared, nobody is asked and the GUI steps run; their + save asks the person as before. +2. The approval request (`item-approval/request-.json`) has + `form_state_kind: "api_request"` and an `api_request` block with the + method, the path and `payload_sha256`, the digest of every byte the save + sends (method, path, content type, encoded body and referer, session + values included). Its `form_state_sha256` is the operation digest: the + payload digest, the record the check reads back (the record check's + selector fields and their values for this run) and the record check's + contract hashes. +3. The person sees the values this run supplies, the record the check reads + back and the request line, as rows on the phone and as the request's + image in Desktop. Session values are counted, never shown. +4. After the approval, the run prepares the request again (fresh form, fresh + pre-checks). The last-point gate compares its operation digest with the + approved one. A different request, record or record check stops the item + before the save (`form_changed_after_approval`). A request that can't be + prepared again stops it too (`form_state_unreadable`). In both cases the + save isn't sent, and the GUI steps don't run in its place. +5. The run sends exactly the request it just checked, once. + +An application that issues a new session value on every form read can't be +approved this way: the second read changes the digest, and the item stops +before saving. + ## When it doesn't run The API step runs only in a plain linear run. It doesn't run, and the GUI steps run instead, when: -- approval mode is `each_item` (the default) or `problems_only`. The person - approves what the screen shows, and an API step leaves the form empty, so - API steps run only with `--approval-mode not_gated`; +- an approval mode is on and the API actuator can't prepare the request + before sending it (a deployment's own actuator without `prepare`); - the run is governed by qualification, is durable, or is resumed; - the step isn't admitted, or the program's steps, on-screen checks or the save's record check changed since it was proposed or admitted; diff --git a/packages/flow/docs/APPROVAL_MODES.md b/packages/flow/docs/APPROVAL_MODES.md index 3cb726142..79c86b118 100644 --- a/packages/flow/docs/APPROVAL_MODES.md +++ b/packages/flow/docs/APPROVAL_MODES.md @@ -38,6 +38,12 @@ in its run directory and waits: (at most 4 hours). - `item-approval/.png`: the filled form the person is shown. +When the save is an API step ([`API_TIER.md`](API_TIER.md)), the person +approves the request it sends instead of a screenshot: the image and the +phone's rows list the values, the record the check reads back and the +request line. The request's `form_state_kind` is `api_request`, and its +digest binds every byte the save sends, the record and the record check. + Desktop reads the requests and answers one of them: ```bash @@ -63,7 +69,9 @@ openadapt-flow item-approval decline RUN_DIR ... --reason wrong_values and right-record check again. At the last point before the input, the run reads the form again; a field that changed while waiting stops the item before the save. In a browser the run reads the page's form controls; - elsewhere it reads the words on the screen. + elsewhere it reads the words on the screen. An API step prepares its + request again instead; a request whose digest changed stops the item + before the save, and the GUI steps don't run in its place. - A decline stops the item before that save. Earlier saves of the same item stay as they are. - `--approval-wait SECONDS` (default 900, at most 14,400) sets how long the diff --git a/packages/flow/openadapt_flow/approval_mode/gate.py b/packages/flow/openadapt_flow/approval_mode/gate.py index 82b86d6e4..a29eda6ca 100644 --- a/packages/flow/openadapt_flow/approval_mode/gate.py +++ b/packages/flow/openadapt_flow/approval_mode/gate.py @@ -50,6 +50,7 @@ ) from openadapt_flow.approval_mode.item_approval import ( DEFAULT_APPROVAL_WAIT_S, + ApiRequestRef, ItemApprovalRefused, ItemApprovalRequest, ItemApprovalStore, @@ -199,6 +200,10 @@ class StepContext: #: The values the save writes, shown on the phone (see #: :data:`SaveFieldsReader`). Bound into the request digest. save_fields: SaveFieldsReader = _no_save_fields + #: The request an API step sends, when the save is one. ``frame`` then + #: returns an image of that request and ``form_state`` its operation + #: digest, so the person approves the request rather than a screenshot. + api_request: Optional[ApiRequestRef] = None @dataclass(frozen=True) @@ -631,6 +636,7 @@ def wait_refusal(detail: str) -> GateVerdict: qualification_case_kind=ctx.qualification_case_kind, step_class=step_class_for(ctx.step, policy_name=self._policy_name), fields=ctx.save_fields(), + api_request=ctx.api_request, ) except (OSError, ValueError, ItemApprovalRefused): return wait_refusal("approval_request_unwritable") diff --git a/packages/flow/openadapt_flow/approval_mode/item_approval.py b/packages/flow/openadapt_flow/approval_mode/item_approval.py index 8200c60b3..71c2c9443 100644 --- a/packages/flow/openadapt_flow/approval_mode/item_approval.py +++ b/packages/flow/openadapt_flow/approval_mode/item_approval.py @@ -158,6 +158,23 @@ class SaveField(BaseModel): after: Optional[str] = Field(default=None, max_length=_MAX_FIELD_VALUE) +class ApiRequestRef(BaseModel): + """The request an API step sends, when the save is an API step. + + The person approves this request, not a screenshot of a form: the API + step fills no form. ``payload_sha256`` covers every byte the request + sends; the request's ``form_state_sha256`` is the operation digest that + binds it to the record and the record check + (:func:`openadapt_flow.runtime.actuators.web_form.operation_digest`). + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + method: str = Field(min_length=1, max_length=16) + path: str = Field(min_length=1, max_length=1024) + payload_sha256: str = Field(pattern=PREFIXED_SHA256_PATTERN) + + class ItemApprovalRequest(BaseModel): """What the person is asked to approve: one save of one item.""" @@ -179,7 +196,9 @@ class ItemApprovalRequest(BaseModel): screenshot_sha256: str = Field(pattern=_HEX_SHA256) #: How the field state was read: the page's own form controls, or the #: text on the screen where the application exposes no form controls. - form_state_kind: Literal["dom", "screen_text"] + #: ``api_request`` when the save is an API step: the digest is then the + #: exact request, the record and the record check (``api_request``). + form_state_kind: Literal["dom", "screen_text", "api_request"] form_state_sha256: str = Field(pattern=PREFIXED_SHA256_PATTERN) created_at: str expires_at: str @@ -195,6 +214,9 @@ class ItemApprovalRequest(BaseModel): fields: Optional[tuple[SaveField, ...]] = Field( default=None, min_length=1, max_length=MAX_SAVE_FIELDS ) + #: The request an API step sends, when this save is one. Left out of the + #: file when absent, so earlier requests keep their digest. + api_request: Optional[ApiRequestRef] = None @model_serializer(mode="wrap") def _serialize_compatible(self, handler: Any) -> dict[str, Any]: @@ -203,6 +225,8 @@ def _serialize_compatible(self, handler: Any) -> dict[str, Any]: data.pop("step_class", None) if self.fields is None: data.pop("fields", None) + if self.api_request is None: + data.pop("api_request", None) return data @model_validator(mode="after") @@ -215,6 +239,8 @@ def _times(self) -> "ItemApprovalRequest": raise ValueError("an approval request may not outlive its window") if self.screenshot != f"{APPROVAL_DIRNAME}/{self.request_id}.png": raise ValueError("the request screenshot must be its own file") + if (self.form_state_kind == "api_request") != (self.api_request is not None): + raise ValueError("an API step's request names the request it sends") return self def digest(self) -> str: @@ -691,11 +717,13 @@ def open_request( ttl_s: float = REQUEST_TTL_S, step_class: StepClass = StepClass.CONSEQUENTIAL, fields: Optional[Sequence[SaveField]] = None, + api_request: Optional[ApiRequestRef] = None, ) -> ItemApprovalRequest: """Write the screenshot and the request. Returns the request. ``fields`` are the values the save writes, shown on the phone. An - empty list is stored as none. + empty list is stored as none. ``api_request`` names the request an + API step sends; ``form_state`` is then ``("api_request", digest)``. """ self._checked_dir(create=True) @@ -721,6 +749,7 @@ def open_request( qualification_case_kind=qualification_case_kind, step_class=step_class, fields=tuple(fields) if fields else None, + api_request=api_request, ) _write_exclusive(self.dir / f"{request_id}.png", frame_png) _write_exclusive( @@ -1100,6 +1129,11 @@ def request_summary( "qualification_case_kind": request.qualification_case_kind, "step_class": request.step_class.value, "decided_by": decided_by_summary(store, request), + "api_request": ( + request.api_request.model_dump(mode="json") + if request.api_request is not None + else None + ), } diff --git a/packages/flow/openadapt_flow/runtime/actuators/web_form.py b/packages/flow/openadapt_flow/runtime/actuators/web_form.py index 2c0285b6f..e356b8f90 100644 --- a/packages/flow/openadapt_flow/runtime/actuators/web_form.py +++ b/packages/flow/openadapt_flow/runtime/actuators/web_form.py @@ -36,9 +36,11 @@ from __future__ import annotations +import hashlib import json import re import urllib.parse as up +from dataclasses import dataclass from html.parser import HTMLParser from typing import Any, Optional, Protocol @@ -247,6 +249,129 @@ def post_form(self, path: str, body: str, *, referer: str) -> Response: ) +# -- the prepared save ---------------------------------------------------------- + + +@dataclass(frozen=True) +class PreparedSave: + """The exact save request, built and checked, not yet sent. + + :meth:`payload_sha256` covers every byte the save sends: the method, the + path, the content type, the encoded body (session values included) and + the referer. A person who approves an API step approves this digest (see + :func:`operation_digest`), and the run refuses the send when the request + it would send has another digest. + """ + + method: str + path: str + body: tuple[tuple[str, str], ...] + referer: str + #: Fields whose value the server issued to this session (CSRF tokens). + #: In the digest; never shown to the person. + session_fields: frozenset[str] = frozenset() + + def encoded(self) -> str: + return up.urlencode(list(self.body)) + + def payload_sha256(self) -> str: + payload = { + "method": self.method.upper(), + "path": self.path, + "content_type": "application/x-www-form-urlencoded", + "body": self.encoded(), + "referer": self.referer, + } + canonical = json.dumps(payload, sort_keys=True, separators=(",", ":")) + return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def operation_digest( + prepared: PreparedSave, + *, + record_check: list[str], + record: list[tuple[str, str]], +) -> str: + """What a person approves for an API step, as one digest. + + Binds the exact request (:meth:`PreparedSave.payload_sha256`), the record + the save is checked against (``record``: the record check's selector + fields and their values for this run) and the record check itself + (``record_check``: each effect's contract hash). + """ + + payload = { + "schema": "openadapt.api-step-operation/v1", + "payload_sha256": prepared.payload_sha256(), + "record": [list(row) for row in record], + "record_check": list(record_check), + } + canonical = json.dumps(payload, sort_keys=True, separators=(",", ":")) + return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def operation_rows( + binding: Any, + prepared: PreparedSave, + params: dict[str, str], + *, + record: list[tuple[str, str]], +) -> list[tuple[str, str]]: + """``(label, value)`` rows a person reads to approve an API step. + + The values this run supplies (parameters and options chosen by label), + then the record the check reads back. The form's own defaults and the + session values are counted, not listed: they are the recorded form's, + and the digest covers them. + """ + + spec = binding.web_form + sent = dict(prepared.body) + rows: list[tuple[str, str]] = [] + defaults = 0 + for field in spec.fields: + if field.source == "param": + rows.append((field.name, sent.get(field.name, ""))) + elif field.source == "select_label": + label = str(params.get(field.param or "", "")) + rows.append((field.name, label)) + elif field.source == "form_default": + defaults += 1 + for name, value in record: + rows.append((f"Record check: {name}", value)) + rows.append( + ( + "Request", + f"{prepared.method.upper()} {prepared.path}; {defaults} form default(s) " + f"and {len(prepared.session_fields)} session value(s) as recorded", + ) + ) + return rows + + +def render_operation_png(title: str, rows: list[tuple[str, str]]) -> bytes: + """A plain image of ``rows``, shown where an approval shows a screenshot. + + Desktop and the phone show the request's image. An API step fills no + form, so the image is the request itself, not the screen. + """ + + import io + + from PIL import Image, ImageDraw + + line_h, pad, width = 18, 16, 900 + lines = [title, ""] + [f"{label}: {value}" for label, value in rows] + height = pad * 2 + line_h * len(lines) + image = Image.new("RGB", (width, height), "white") + draw = ImageDraw.Draw(image) + for i, line in enumerate(lines): + draw.text((pad, pad + i * line_h), line[:140], fill="black") + out = io.BytesIO() + image.save(out, format="PNG") + return out.getvalue() + + # -- actuator -------------------------------------------------------------------- @@ -262,6 +387,22 @@ def __init__(self, transport: Transport) -> None: self.log: list[dict[str, Any]] = [] def actuate(self, binding: Any, params: dict[str, str]) -> ApiActuationResult: + """Prepare the save and send it: :meth:`prepare`, then :meth:`send`.""" + + prepared = self.prepare(binding, params) + if isinstance(prepared, ApiActuationResult): + return prepared + return self.send(binding, prepared) + + def prepare( + self, binding: Any, params: dict[str, str] + ) -> "PreparedSave | ApiActuationResult": + """Everything before the save: reads only, nothing that writes. + + Returns the exact request :meth:`send` would send, or an + ``UNAVAILABLE`` result saying why the save can't be sent. + """ + spec = binding.web_form summary = f"{binding.method} {binding.url_template}" @@ -323,13 +464,24 @@ def unavailable(reason: str) -> ApiActuationResult: return unavailable(f"the run has no value for {exc}") except Exception as exc: # noqa: BLE001 - only reads were sent return unavailable(f"a read before the save failed ({type(exc).__name__})") + return PreparedSave( + method=binding.method, + path=binding.url_template, + body=tuple(body), + referer=self.transport.base_url + spec.form_path, + session_fields=frozenset( + f.name for f in spec.fields if f.source == "session" + ), + ) + + def send(self, binding: Any, prepared: "PreparedSave") -> ApiActuationResult: + """Send exactly ``prepared``, once. Never retried.""" - encoded = up.urlencode(body) + spec = binding.web_form + summary = f"{binding.method} {binding.url_template}" try: resp = self.transport.post_form( - binding.url_template, - encoded, - referer=self.transport.base_url + spec.form_path, + prepared.path, prepared.encoded(), referer=prepared.referer ) except Exception as exc: # noqa: BLE001 - after dispatch began self.log.append({"stage": "save", "error": type(exc).__name__}) diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py index fe4105905..02b3e546a 100644 --- a/packages/flow/openadapt_flow/runtime/replayer.py +++ b/packages/flow/openadapt_flow/runtime/replayer.py @@ -47,6 +47,7 @@ import time import uuid from copy import deepcopy +from dataclasses import dataclass from datetime import date, datetime, timezone from pathlib import Path from typing import ( @@ -56,6 +57,7 @@ Literal, Mapping, Optional, + Sequence, TypeVar, cast, ) @@ -641,6 +643,18 @@ def better_identity_read(first: Any, retry: Any) -> Any: return first +@dataclass(frozen=True) +class _WebFormApprovalView: + """An API step's prepared request, as a person is asked to approve it.""" + + prepared: Any + #: :func:`~openadapt_flow.runtime.actuators.web_form.operation_digest`. + digest: str + rows: list[tuple[str, str]] + png: bytes + ref: Any + + class Replayer: """Replays a Workflow against a Backend using injected vision. @@ -1033,6 +1047,12 @@ def __init__( self.web_form_actuator = web_form_actuator self._web_form_span_active = False self._web_form_fallback_reason: Optional[str] = None + # Under an approval mode, the API request a person was asked about + # (prepared before the gate), keyed by the save step's result, and + # the operation digest the gate's last-point check re-reads. + self._web_form_approval: dict[int, Any] = {} + self._web_form_form_state: Optional[tuple[str, str]] = None + self._web_form_form_state_set = False # Durable tiered runtime (RFC §5, Tier 3), OFF by default. When True, # run() writes a RunCheckpoint after each VERIFIED step and, on a halt, # a PendingEscalation instead of just dying -- so the run can be @@ -2387,16 +2407,15 @@ def _web_form_span_at( reason = "a durable or resumed run uses the GUI steps" elif getattr(self, "_surface_router", None) is not None: reason = "a run across several windows or tabs uses the GUI steps" - elif ( - self.approval_gate is not None - and str(getattr(getattr(self.approval_gate, "mode", None), "value", "")) - != "not_gated" + elif self._approval_gated() and not callable( + getattr(self._actuator_for(binding), "prepare", None) ): - # A person approves what the screen shows; an API step would - # leave the form empty. Only an ungated run uses API steps. + # Under an approval mode the person approves the exact request + # the API step sends. An actuator that can't prepare the + # request before sending it can't show it. reason = ( - "approval mode shows the person the screen before the save, " - "so the GUI steps run" + "approval mode needs the exact request before the save, and " + "this API actuator can't prepare it, so the GUI steps run" ) elif self.governed_authorization is not None: reason = "a governed run uses its qualified GUI path" @@ -2459,6 +2478,7 @@ def _run_web_form_span( ) finally: self._web_form_span_active = False + self._web_form_approval.clear() if result.api_fallback is not None: self._web_form_fallback_reason = result.api_fallback return None @@ -7106,8 +7126,23 @@ def _run_step_on_active_surface( # approved step then takes a fresh frame and resolves its target # and right-record check again. The last-point gate still re-reads # the form before the input is delivered. + api_view = None + if ( + self._web_form_span_active + and step.api_binding is not None + and self._approval_gated() + ): + # An API step fills no form: the person approves the exact + # request it sends. If it can't be prepared, nothing was sent + # and nobody was asked; the GUI steps run instead. + api_view = self._web_form_approval_view(step, params) + if isinstance(api_view, str): + result.api_fallback = api_view + result.elapsed_ms = (time.monotonic() - t0) * 1000.0 + return result + self._web_form_approval[id(result)] = api_view approval_refusal = self._approval_step_refusal( - workflow, step, step_index, result + workflow, step, step_index, result, api_view=api_view ) if approval_refusal is not None: result.ok = False @@ -8717,7 +8752,46 @@ def _try_api_tier( current_step=overlay_current, total_steps=overlay_total, ) - refusal = self._delivery_authorization_refusal(workflow, params, step, result) + approved_view = self._web_form_approval.pop(id(result), None) + send_prepared: Optional[Any] = None + again: Any = None + if approved_view is not None: + # Under an approval mode: prepare the request again now (fresh + # form, fresh pre-checks). The last-point gate compares its + # operation digest with the one a person approved; a request that + # can't be prepared again reads as an unreadable form. + again = self._web_form_approval_view(step, params) + self._web_form_form_state = ( + None if isinstance(again, str) else ("api_request", again.digest) + ) + self._web_form_form_state_set = True + try: + refusal = self._delivery_authorization_refusal( + workflow, params, step, result + ) + finally: + self._web_form_form_state = None + self._web_form_form_state_set = False + if refusal is None and approved_view is not None: + if isinstance(again, str) or again.digest != approved_view.digest: + # Also refused when the gate let it through (for example a + # save that passed problems_only's checks without a person): + # only the exact request prepared before the gate is sent. + why = ( + again + if isinstance(again, str) + else "the request, the record or the record check differs" + ) + result.safety_halt = True + result.failure_category = "governed_refusal" + refusal = ( + f"Step '{step.id}' ({step.intent}): the API request changed " + f"after the approval gate ({why}); the save was not sent " + "and the GUI steps do not run in its place" + ) + else: + # Exactly the request whose digest was approved and re-checked. + send_prepared = again.prepared if refusal is not None: result.ok = False result.error = refusal @@ -8732,10 +8806,14 @@ def _try_api_tier( result.delivery_attempted = True try: self._require_qualification_environment_current() - outcome = api_actuator.actuate( - binding, - runtime_params_for_gui(params), - ) + if send_prepared is not None: + # Exactly the request whose digest was approved and re-checked. + outcome = api_actuator.send(binding, send_prepared) + else: + outcome = api_actuator.actuate( + binding, + runtime_params_for_gui(params), + ) except Exception as exc: # noqa: BLE001 - external actuator boundary # The actuator contract is no-throw, but a deployment adapter can # still violate it. The delivery boundary was crossed immediately @@ -11525,11 +11603,81 @@ def frame() -> Optional[bytes]: "Approval mode: this item stopped before saving." ) + def _approval_gated(self) -> bool: + """Whether an approval mode other than ``not_gated`` governs this run.""" + + gate = self.approval_gate + return gate is not None and ( + str(getattr(getattr(gate, "mode", None), "value", "")) != "not_gated" + ) + + def _web_form_approval_view( + self, step: Step, params: Mapping[str, RuntimeParamScalar] + ) -> "Any | str": + """Prepare an API step's request for a person to approve. + + Returns the view (the prepared request, its operation digest, the + rows and image the person sees), or the reason it can't be prepared. + Only reads are sent. + """ + + from openadapt_flow.approval_mode.item_approval import ApiRequestRef + from openadapt_flow.runtime.actuators import ApiActuationResult + from openadapt_flow.runtime.actuators.web_form import ( + operation_digest, + operation_rows, + render_operation_png, + ) + + binding = step.api_binding + assert binding is not None + actuator = self._actuator_for(binding) + if actuator is None or not callable(getattr(actuator, "prepare", None)): + return "this API actuator can't prepare the request for approval" + gui_params = runtime_params_for_gui(params) + try: + prepared = actuator.prepare(binding, gui_params) + except Exception as exc: # noqa: BLE001 - only reads were sent + return f"preparing the request failed ({type(exc).__name__})" + if isinstance(prepared, ApiActuationResult): + return prepared.reason or "the API step is unavailable" + effects = self._resolve_effects(list(binding.effects), params) + record = [ + (key, str(expr)) + for effect in effects + for key, expr in sorted(effect.match.items()) + ] + digest = operation_digest( + prepared, + record_check=[effect.contract_hash() for effect in effects], + record=record, + ) + rows = operation_rows(binding, prepared, gui_params, record=record) + return _WebFormApprovalView( + prepared=prepared, + digest=digest, + rows=rows, + png=render_operation_png( + f"Approve this request: {step.intent or step.id}", rows + ), + ref=ApiRequestRef( + method=prepared.method.upper(), + path=prepared.path, + payload_sha256=prepared.payload_sha256(), + ), + ) + def _approval_form_state( self, frame_png: Optional[bytes] ) -> Optional[tuple[str, str]]: - """The form's current field state digest, for the approval gate.""" + """The form's current field state digest, for the approval gate. + For an API step under an approval mode it is the operation digest of + the request about to be sent (set just before the last-point gate). + """ + + if self._web_form_form_state_set: + return self._web_form_form_state from openadapt_flow.approval_mode.form_state import capture_form_state try: @@ -11560,8 +11708,14 @@ def _approval_step_refusal( step: Step, step_index: int, result: StepResult, + *, + api_view: Optional[Any] = None, ) -> Optional[str]: - """Ask the approval gate before a save step resolves its target.""" + """Ask the approval gate before a save step resolves its target. + + ``api_view`` is the prepared request of an API step: the person is + then shown that request, and its operation digest is the "form". + """ gate = self.approval_gate assert gate is not None @@ -11569,11 +11723,35 @@ def _approval_step_refusal( drill_fault_step = self._is_qualification_fault_step(step) def frame() -> Optional[bytes]: + if api_view is not None: + return api_view.png try: return self.vision.wait_settled(self.backend) except Exception: # noqa: BLE001 - no frame means no approval return None + def form_state(frame_png: Optional[bytes]) -> Optional[tuple[str, str]]: + if api_view is not None: + return ("api_request", api_view.digest) + return self._approval_form_state(frame_png) + + def save_fields() -> Optional[Sequence[Any]]: + if api_view is None: + return None + from openadapt_flow.approval_mode.item_approval import ( + MAX_SAVE_FIELDS, + SaveField, + ) + + rows = api_view.rows + if len(rows) > MAX_SAVE_FIELDS: + # The image lists every row; the phone shows it alone. + return None + return [ + SaveField(label=label[:128] or "-", after=value[:512]) + for label, value in rows + ] + try: from openadapt_flow.approval_mode.gate import StepContext @@ -11585,7 +11763,9 @@ def frame() -> Optional[bytes]: step_index=step_index, run_id=self._run_id or "", frame=frame, - form_state=self._approval_form_state, + form_state=form_state, + save_fields=save_fields, + api_request=api_view.ref if api_view is not None else None, drill_fault_step=drill_fault_step, qualification_case_kind=( authorization.qualification_case_kind diff --git a/packages/flow/tests/test_api_step_approval.py b/packages/flow/tests/test_api_step_approval.py new file mode 100644 index 000000000..98f69e5e7 --- /dev/null +++ b/packages/flow/tests/test_api_step_approval.py @@ -0,0 +1,317 @@ +"""An API step under an approval mode (docs/API_TIER.md, docs/APPROVAL_MODES.md). + +The person approves the exact request the API step sends: the values, the +record the check reads back and the record check, bound by one digest to the +request bytes. Not a screenshot of a form the API step never fills. + +- the run prepares the request (reads only), asks, and sends exactly that + request once after a person's approval; +- a request that changes while the person looks is not sent, and the GUI + steps don't run in its place; +- a decline sends nothing; +- a request that can't be prepared asks nobody: the GUI steps run, and their + save asks the person as before. + +The transport is a fake that writes to the MockMed fault server; the values +are made up. +""" + +from __future__ import annotations + +import hashlib +import json +import urllib.parse as up +from typing import Any, Optional + +import requests + +from openadapt_flow.approval_mode.gate import ApprovalGate +from openadapt_flow.approval_mode.item_approval import ( + ItemApprovalStore, + make_answer, +) +from openadapt_flow.approval_mode.model import ItemDecisionKind +from openadapt_flow.compiler.api_proposal import admit_api_step +from openadapt_flow.ir import WebFormField, Workflow +from openadapt_flow.mockmed.fault_server import serve as fault_serve +from openadapt_flow.runtime.actuators.web_form import WebFormActuator +from openadapt_flow.runtime.effects import RestRecordVerifier +from openadapt_flow.runtime.replayer import Replayer +from tests.test_api_tier_web_form import TARGET, EnterWritesBackend, _span_workflow +from tests.test_replayer import FakeVision + +APPROVE: dict[str, Any] = dict( + decision="approve", + decided_by="human", + route="desktop_run_it", + approver_id="cloud-user-0001", + auth_method="cloud_aal2", +) +PARAMS = {"lname": "Marsh"} +FORM = """
+ + +
""" + + +class SessionTransport: + """The live session: a form page, a duplicate search and the save.""" + + base_url = "http://app" + + def __init__(self, sor_url: str) -> None: + self.sor_url = sor_url + self.token = "session-1" + self.dup = "No matches were found." + self.posts: list[tuple[str, str]] = [] + + def get(self, path: str) -> Any: + text = FORM.format(token=self.token) if path == "/new.php" else self.dup + return _Resp(200, text) + + def post_form(self, path: str, body: str, *, referer: str) -> Any: + self.posts.append((path, body)) + requests.post( + f"{self.sor_url}/api/encounter", + json={**TARGET, "note": "api"}, + timeout=5, + ) + return _Resp(200, "saved") + + +class _Resp: + def __init__(self, status: int, text: str) -> None: + self.status, self.text, self.headers = status, text, {} + + +class Person: + """Answers each waiting request; ``on_wait`` runs first, each poll.""" + + def __init__(self, decisions: list[Optional[dict[str, Any]]]) -> None: + self.decisions = list(decisions) + self.clock = 0.0 + self.store: Optional[ItemApprovalStore] = None + self.on_wait: Any = None + self.seen: list[str] = [] + + def monotonic(self) -> float: + return self.clock + + def sleep(self, seconds: float) -> None: + self.clock += seconds + if self.on_wait is not None: + self.on_wait() + assert self.store is not None + for request in self.store.requests(): + if request.request_id in self.seen: + continue + self.seen.append(request.request_id) + decision = self.decisions.pop(0) if self.decisions else None + if decision is not None: + self.store.write_answer(make_answer(request, **decision)) + + +def _workflow() -> Workflow: + """The span program, its save also sending the session's CSRF value.""" + + wf = _span_workflow(admitted=False) + save = wf.steps[-1] + binding = save.api_binding + spec = binding.web_form.model_copy( + update={ + "fields": [ + WebFormField(name="csrf", source="session"), + *binding.web_form.fields, + ] + } + ) + binding = binding.model_copy(update={"web_form": spec}) + wf = wf.model_copy( + update={ + "steps": [*wf.steps[:-1], save.model_copy(update={"api_binding": binding})] + } + ) + wf, _ = admit_api_step(wf, admitted_by="reviewer") + return wf + + +def _run(tmp_path, person: Person, *, transport_setup=None): + url, db, stop = fault_serve() + url = url.rstrip("/") + try: + run_dir = tmp_path / "run" + store = ItemApprovalStore(run_dir) + person.store = store + transport = SessionTransport(url) + if transport_setup is not None: + transport_setup(transport) + backend = EnterWritesBackend(url) + gate = ApprovalGate( + "each_item", + approval_store=store, + approval_wait_s=30.0, + approval_poll_s=1.0, + sleep=person.sleep, + monotonic=person.monotonic, + ) + bundle = tmp_path / "bundle" + (bundle / "templates").mkdir(parents=True, exist_ok=True) + report = Replayer( + backend, + vision=FakeVision(), + effect_verifier=RestRecordVerifier(url), + web_form_actuator=WebFormActuator(transport), + approval_gate=gate, + poll_interval_s=0.01, + ).run(_workflow(), bundle_dir=bundle, run_dir=run_dir, params=PARAMS) + return report, backend, transport, store, db.snapshot()["records"] + finally: + stop() + + +def test_the_person_approves_the_exact_request_and_it_is_sent_once(tmp_path): + report, backend, transport, store, records = _run(tmp_path, Person([APPROVE])) + + assert report.success is True, [r.error for r in report.results] + assert backend.actions == [] + ((path, body),) = transport.posts + assert path == "/save.php" and body == "csrf=session-1&lname=Marsh" + folded, save = report.results + assert folded.actuation == "api_folded" and save.actuation == "api" + assert save.effect_verified is True + assert [r["note"] for r in records] == ["api"] + # A person approved the one save that was delivered. (This fixture runs + # without an execution profile, so the outcome reads "Finished, not + # checked" and the item still goes to a person; the Regulated profile + # case is in the compatibility matrix.) + assert [row.decision for row in report.approval.approvals] == ["approved"] + assert report.approval.decision_reason != "save_not_approved" + + (request,) = store.requests() + assert request.form_state_kind == "api_request" + assert store.status(request) == "used" + # The approved digest is the digest of the bytes that were sent. + sent = { + "method": "POST", + "path": path, + "content_type": "application/x-www-form-urlencoded", + "body": body, + "referer": "http://app/new.php", + } + canonical = json.dumps(sent, sort_keys=True, separators=(",", ":")) + assert request.api_request.payload_sha256 == ( + "sha256:" + hashlib.sha256(canonical.encode()).hexdigest() + ) + # The person reads the values, the record and the request, not a form. + rows = {field.label: field.after for field in request.fields} + assert rows["lname"] == "Marsh" + assert rows["Record check: patient_id"] == TARGET["patient_id"] + assert rows["Request"].startswith("POST /save.php") + # The session value is in the digest, never in front of the person. + assert "session-1" not in json.dumps(rows) + assert (store.run_dir / request.screenshot).read_bytes()[:4] == b"\x89PNG" + # Nothing about the request reaches the report. + saved = (tmp_path / "run" / "report.json").read_text() + assert request.api_request.payload_sha256 not in saved + assert "Marsh" not in json.dumps(json.loads(saved)["approval"]) + + +def test_a_request_that_changes_while_the_person_looks_is_not_sent(tmp_path): + person = Person([APPROVE]) + holder: dict[str, SessionTransport] = {} + + def keep(transport: SessionTransport) -> None: + holder["t"] = transport + + def the_session_value_changes() -> None: + holder["t"].token = "session-2" + + person.on_wait = the_session_value_changes + report, backend, transport, store, records = _run( + tmp_path, person, transport_setup=keep + ) + + assert transport.posts == [] and records == [] + # Not redone through the GUI either: the person approved this request. + assert backend.actions == [] + assert report.success is False + assert report.transaction_outcome == "HALTED_BEFORE_EFFECT" + assert report.approval.decision is ItemDecisionKind.STOP_BEFORE_SAVING + failed = [c for c in report.approval.pre_checks if not c.passed] + assert failed[-1].detail == "form_changed_after_approval" + + +def test_a_failed_check_after_approval_is_not_sent(tmp_path): + person = Person([APPROVE]) + holder: dict[str, SessionTransport] = {} + + def keep(transport: SessionTransport) -> None: + holder["t"] = transport + + def a_duplicate_appears() -> None: + holder["t"].dup = "1 match found" + + person.on_wait = a_duplicate_appears + report, backend, transport, store, records = _run( + tmp_path, person, transport_setup=keep + ) + + assert transport.posts == [] and records == [] and backend.actions == [] + assert report.approval.decision is ItemDecisionKind.STOP_BEFORE_SAVING + failed = [c for c in report.approval.pre_checks if not c.passed] + assert failed[-1].detail == "form_state_unreadable" + + +def test_a_decline_sends_nothing(tmp_path): + decline = dict(APPROVE, decision="decline", decline_reason="wrong_values") + report, backend, transport, store, records = _run(tmp_path, Person([decline])) + + assert transport.posts == [] and records == [] and backend.actions == [] + assert report.approval.decision is ItemDecisionKind.DECLINED + + +def test_a_request_that_cant_be_prepared_asks_nobody_and_runs_the_gui(tmp_path): + def duplicate(transport: SessionTransport) -> None: + transport.dup = "1 match found" + + # Nobody answers: the GUI save then waits for a person and stops. + report, backend, transport, store, records = _run( + tmp_path, Person([None]), transport_setup=duplicate + ) + + assert transport.posts == [] + first = report.results[0] + assert "did not pass" in first.api_fallback + assert ("press", "Tab") in [(a[0], a[1]) for a in backend.actions] + # The one request the person saw is the GUI save's, not an API request. + assert [r.form_state_kind for r in store.requests()] in ([], ["screen_text"]) + assert all(r.api_request is None for r in store.requests()) + + +def test_operation_digest_binds_the_record_check(): + from openadapt_flow.runtime.actuators.web_form import ( + PreparedSave, + operation_digest, + ) + + prepared = PreparedSave( + method="POST", + path="/save.php", + body=(("lname", "Marsh"),), + referer="http://app/new.php", + ) + base = operation_digest(prepared, record_check=["h1"], record=[("id", "p1")]) + assert base != operation_digest( + prepared, record_check=["h2"], record=[("id", "p1")] + ) + assert base != operation_digest( + prepared, record_check=["h1"], record=[("id", "p2")] + ) + changed = PreparedSave( + method="POST", + path="/save.php", + body=(("lname", "Marsh "),), + referer="http://app/new.php", + ) + assert base != operation_digest(changed, record_check=["h1"], record=[("id", "p1")]) + assert up.parse_qs(prepared.encoded()) == {"lname": ["Marsh"]} diff --git a/packages/flow/tests/test_api_tier_web_form.py b/packages/flow/tests/test_api_tier_web_form.py index f5b897d16..f0f0fe58a 100644 --- a/packages/flow/tests/test_api_tier_web_form.py +++ b/packages/flow/tests/test_api_tier_web_form.py @@ -672,18 +672,21 @@ def test_gui_steps_run_when_the_api_step_may_not(tmp_path, mutate, kw, reason): assert ("press", "Tab") in [(a[0], a[1]) for a in backend.actions] -@pytest.mark.parametrize( - "mode, runs", [("each_item", False), ("problems_only", False), ("not_gated", True)] -) -def test_only_an_ungated_run_uses_api_steps(mode, runs): +@pytest.mark.parametrize("mode", ["each_item", "problems_only", "not_gated"]) +@pytest.mark.parametrize("can_prepare", [True, False]) +def test_a_gated_run_uses_api_steps_that_can_show_their_request(mode, can_prepare): from openadapt_flow.approval_mode.model import ApprovalMode - replayer = Replayer(FakeBackend(), vision=FakeVision(), web_form_actuator=object()) + actuator = WebFormActuator(FakeTransport()) if can_prepare else object() + replayer = Replayer(FakeBackend(), vision=FakeVision(), web_form_actuator=actuator) replayer.approval_gate = type("Gate", (), {"mode": ApprovalMode(mode)})() _end, _save, reason = replayer._web_form_span_at(_span_workflow(), 0, durable=False) + # Under an approval mode the person approves the exact request + # (tests/test_api_step_approval.py), so the actuator must prepare it. + runs = can_prepare or mode == "not_gated" assert (reason is None) is runs if not runs: - assert "approval mode" in reason + assert "approval mode needs the exact request" in reason class LostResponseActuator(ScriptedActuator):