From 08afcc96de581eac35929f80810023ef1dd09e96 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 12:03:53 -0400 Subject: [PATCH 1/3] flow: approve an API step's exact request under per-item approval Under each_item and problems_only the API step used to fall back to the GUI steps, because the person approved a screenshot of a form the API step never fills. Flow 2 gates every saving run, so the API step never ran in the default profile. Now the run prepares the request first (reads only: form page, body, pre-checks) and the person approves that request. The approval request has form_state_kind "api_request", an api_request block (method, path, payload_sha256 over every byte sent) and an operation digest that binds the payload, the record the check reads back and the record check's contract hashes. After the approval the run prepares the request again; the last-point gate compares digests, and a changed or unpreparable request stops the item before the save, with no GUI redo. The run then sends exactly the request it checked, once. WebFormActuator splits into prepare() and send(); actuate() is both. An actuator without prepare() still falls back to the GUI under a gate. Governed, durable and multi-surface runs still use the GUI steps; those come in the next pull requests. Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 1 + packages/flow/docs/API_TIER.md | 38 ++- packages/flow/docs/APPROVAL_MODES.md | 10 +- .../flow/openadapt_flow/approval_mode/gate.py | 6 + .../approval_mode/item_approval.py | 38 ++- .../runtime/actuators/web_form.py | 160 ++++++++- .../flow/openadapt_flow/runtime/replayer.py | 214 +++++++++++- packages/flow/tests/test_api_step_approval.py | 317 ++++++++++++++++++ packages/flow/tests/test_api_tier_web_form.py | 15 +- 9 files changed, 766 insertions(+), 33 deletions(-) create mode 100644 packages/flow/tests/test_api_step_approval.py diff --git a/CAPABILITIES.md b/CAPABILITIES.md index c9125ded3..d2c6419f0 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -181,6 +181,7 @@ improvising or retrying when something is off. | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | | Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | | Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-an-approval-mode) | | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | diff --git a/packages/flow/docs/API_TIER.md b/packages/flow/docs/API_TIER.md index abd6d0b91..560c78a0a 100644 --- a/packages/flow/docs/API_TIER.md +++ b/packages/flow/docs/API_TIER.md @@ -100,14 +100,46 @@ RECONCILIATION_REQUIRED` unless the record check proves the effect. The record check (`effects`) then confirms the save with the same verifier a GUI save uses. +## Under an approval mode + +Under `each_item` (the default) and `problems_only`, the person approves the +request the API step sends, not a screenshot of the form. The API step fills +no form, so a screenshot would show an empty one. + +1. Before it asks, the run prepares the request: it reads the form page, + builds the body and runs the pre-checks. Only reads are sent. If the + request can't be prepared, nobody is asked and the GUI steps run; their + save asks the person as before. +2. The approval request (`item-approval/request-.json`) has + `form_state_kind: "api_request"` and an `api_request` block with the + method, the path and `payload_sha256`, the digest of every byte the save + sends (method, path, content type, encoded body and referer, session + values included). Its `form_state_sha256` is the operation digest: the + payload digest, the record the check reads back (the record check's + selector fields and their values for this run) and the record check's + contract hashes. +3. The person sees the values this run supplies, the record the check reads + back and the request line, as rows on the phone and as the request's + image in Desktop. Session values are counted, never shown. +4. After the approval, the run prepares the request again (fresh form, fresh + pre-checks). The last-point gate compares its operation digest with the + approved one. A different request, record or record check stops the item + before the save (`form_changed_after_approval`). A request that can't be + prepared again stops it too (`form_state_unreadable`). In both cases the + save isn't sent, and the GUI steps don't run in its place. +5. The run sends exactly the request it just checked, once. + +An application that issues a new session value on every form read can't be +approved this way: the second read changes the digest, and the item stops +before saving. + ## When it doesn't run The API step runs only in a plain linear run. It doesn't run, and the GUI steps run instead, when: -- approval mode is `each_item` (the default) or `problems_only`. The person - approves what the screen shows, and an API step leaves the form empty, so - API steps run only with `--approval-mode not_gated`; +- an approval mode is on and the API actuator can't prepare the request + before sending it (a deployment's own actuator without `prepare`); - the run is governed by qualification, is durable, or is resumed; - the step isn't admitted, or the program's steps, on-screen checks or the save's record check changed since it was proposed or admitted; diff --git a/packages/flow/docs/APPROVAL_MODES.md b/packages/flow/docs/APPROVAL_MODES.md index 3cb726142..79c86b118 100644 --- a/packages/flow/docs/APPROVAL_MODES.md +++ b/packages/flow/docs/APPROVAL_MODES.md @@ -38,6 +38,12 @@ in its run directory and waits: (at most 4 hours). - `item-approval/.png`: the filled form the person is shown. +When the save is an API step ([`API_TIER.md`](API_TIER.md)), the person +approves the request it sends instead of a screenshot: the image and the +phone's rows list the values, the record the check reads back and the +request line. The request's `form_state_kind` is `api_request`, and its +digest binds every byte the save sends, the record and the record check. + Desktop reads the requests and answers one of them: ```bash @@ -63,7 +69,9 @@ openadapt-flow item-approval decline RUN_DIR ... --reason wrong_values and right-record check again. At the last point before the input, the run reads the form again; a field that changed while waiting stops the item before the save. In a browser the run reads the page's form controls; - elsewhere it reads the words on the screen. + elsewhere it reads the words on the screen. An API step prepares its + request again instead; a request whose digest changed stops the item + before the save, and the GUI steps don't run in its place. - A decline stops the item before that save. Earlier saves of the same item stay as they are. - `--approval-wait SECONDS` (default 900, at most 14,400) sets how long the diff --git a/packages/flow/openadapt_flow/approval_mode/gate.py b/packages/flow/openadapt_flow/approval_mode/gate.py index 82b86d6e4..a29eda6ca 100644 --- a/packages/flow/openadapt_flow/approval_mode/gate.py +++ b/packages/flow/openadapt_flow/approval_mode/gate.py @@ -50,6 +50,7 @@ ) from openadapt_flow.approval_mode.item_approval import ( DEFAULT_APPROVAL_WAIT_S, + ApiRequestRef, ItemApprovalRefused, ItemApprovalRequest, ItemApprovalStore, @@ -199,6 +200,10 @@ class StepContext: #: The values the save writes, shown on the phone (see #: :data:`SaveFieldsReader`). Bound into the request digest. save_fields: SaveFieldsReader = _no_save_fields + #: The request an API step sends, when the save is one. ``frame`` then + #: returns an image of that request and ``form_state`` its operation + #: digest, so the person approves the request rather than a screenshot. + api_request: Optional[ApiRequestRef] = None @dataclass(frozen=True) @@ -631,6 +636,7 @@ def wait_refusal(detail: str) -> GateVerdict: qualification_case_kind=ctx.qualification_case_kind, step_class=step_class_for(ctx.step, policy_name=self._policy_name), fields=ctx.save_fields(), + api_request=ctx.api_request, ) except (OSError, ValueError, ItemApprovalRefused): return wait_refusal("approval_request_unwritable") diff --git a/packages/flow/openadapt_flow/approval_mode/item_approval.py b/packages/flow/openadapt_flow/approval_mode/item_approval.py index 8200c60b3..71c2c9443 100644 --- a/packages/flow/openadapt_flow/approval_mode/item_approval.py +++ b/packages/flow/openadapt_flow/approval_mode/item_approval.py @@ -158,6 +158,23 @@ class SaveField(BaseModel): after: Optional[str] = Field(default=None, max_length=_MAX_FIELD_VALUE) +class ApiRequestRef(BaseModel): + """The request an API step sends, when the save is an API step. + + The person approves this request, not a screenshot of a form: the API + step fills no form. ``payload_sha256`` covers every byte the request + sends; the request's ``form_state_sha256`` is the operation digest that + binds it to the record and the record check + (:func:`openadapt_flow.runtime.actuators.web_form.operation_digest`). + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + method: str = Field(min_length=1, max_length=16) + path: str = Field(min_length=1, max_length=1024) + payload_sha256: str = Field(pattern=PREFIXED_SHA256_PATTERN) + + class ItemApprovalRequest(BaseModel): """What the person is asked to approve: one save of one item.""" @@ -179,7 +196,9 @@ class ItemApprovalRequest(BaseModel): screenshot_sha256: str = Field(pattern=_HEX_SHA256) #: How the field state was read: the page's own form controls, or the #: text on the screen where the application exposes no form controls. - form_state_kind: Literal["dom", "screen_text"] + #: ``api_request`` when the save is an API step: the digest is then the + #: exact request, the record and the record check (``api_request``). + form_state_kind: Literal["dom", "screen_text", "api_request"] form_state_sha256: str = Field(pattern=PREFIXED_SHA256_PATTERN) created_at: str expires_at: str @@ -195,6 +214,9 @@ class ItemApprovalRequest(BaseModel): fields: Optional[tuple[SaveField, ...]] = Field( default=None, min_length=1, max_length=MAX_SAVE_FIELDS ) + #: The request an API step sends, when this save is one. Left out of the + #: file when absent, so earlier requests keep their digest. + api_request: Optional[ApiRequestRef] = None @model_serializer(mode="wrap") def _serialize_compatible(self, handler: Any) -> dict[str, Any]: @@ -203,6 +225,8 @@ def _serialize_compatible(self, handler: Any) -> dict[str, Any]: data.pop("step_class", None) if self.fields is None: data.pop("fields", None) + if self.api_request is None: + data.pop("api_request", None) return data @model_validator(mode="after") @@ -215,6 +239,8 @@ def _times(self) -> "ItemApprovalRequest": raise ValueError("an approval request may not outlive its window") if self.screenshot != f"{APPROVAL_DIRNAME}/{self.request_id}.png": raise ValueError("the request screenshot must be its own file") + if (self.form_state_kind == "api_request") != (self.api_request is not None): + raise ValueError("an API step's request names the request it sends") return self def digest(self) -> str: @@ -691,11 +717,13 @@ def open_request( ttl_s: float = REQUEST_TTL_S, step_class: StepClass = StepClass.CONSEQUENTIAL, fields: Optional[Sequence[SaveField]] = None, + api_request: Optional[ApiRequestRef] = None, ) -> ItemApprovalRequest: """Write the screenshot and the request. Returns the request. ``fields`` are the values the save writes, shown on the phone. An - empty list is stored as none. + empty list is stored as none. ``api_request`` names the request an + API step sends; ``form_state`` is then ``("api_request", digest)``. """ self._checked_dir(create=True) @@ -721,6 +749,7 @@ def open_request( qualification_case_kind=qualification_case_kind, step_class=step_class, fields=tuple(fields) if fields else None, + api_request=api_request, ) _write_exclusive(self.dir / f"{request_id}.png", frame_png) _write_exclusive( @@ -1100,6 +1129,11 @@ def request_summary( "qualification_case_kind": request.qualification_case_kind, "step_class": request.step_class.value, "decided_by": decided_by_summary(store, request), + "api_request": ( + request.api_request.model_dump(mode="json") + if request.api_request is not None + else None + ), } diff --git a/packages/flow/openadapt_flow/runtime/actuators/web_form.py b/packages/flow/openadapt_flow/runtime/actuators/web_form.py index 2c0285b6f..e356b8f90 100644 --- a/packages/flow/openadapt_flow/runtime/actuators/web_form.py +++ b/packages/flow/openadapt_flow/runtime/actuators/web_form.py @@ -36,9 +36,11 @@ from __future__ import annotations +import hashlib import json import re import urllib.parse as up +from dataclasses import dataclass from html.parser import HTMLParser from typing import Any, Optional, Protocol @@ -247,6 +249,129 @@ def post_form(self, path: str, body: str, *, referer: str) -> Response: ) +# -- the prepared save ---------------------------------------------------------- + + +@dataclass(frozen=True) +class PreparedSave: + """The exact save request, built and checked, not yet sent. + + :meth:`payload_sha256` covers every byte the save sends: the method, the + path, the content type, the encoded body (session values included) and + the referer. A person who approves an API step approves this digest (see + :func:`operation_digest`), and the run refuses the send when the request + it would send has another digest. + """ + + method: str + path: str + body: tuple[tuple[str, str], ...] + referer: str + #: Fields whose value the server issued to this session (CSRF tokens). + #: In the digest; never shown to the person. + session_fields: frozenset[str] = frozenset() + + def encoded(self) -> str: + return up.urlencode(list(self.body)) + + def payload_sha256(self) -> str: + payload = { + "method": self.method.upper(), + "path": self.path, + "content_type": "application/x-www-form-urlencoded", + "body": self.encoded(), + "referer": self.referer, + } + canonical = json.dumps(payload, sort_keys=True, separators=(",", ":")) + return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def operation_digest( + prepared: PreparedSave, + *, + record_check: list[str], + record: list[tuple[str, str]], +) -> str: + """What a person approves for an API step, as one digest. + + Binds the exact request (:meth:`PreparedSave.payload_sha256`), the record + the save is checked against (``record``: the record check's selector + fields and their values for this run) and the record check itself + (``record_check``: each effect's contract hash). + """ + + payload = { + "schema": "openadapt.api-step-operation/v1", + "payload_sha256": prepared.payload_sha256(), + "record": [list(row) for row in record], + "record_check": list(record_check), + } + canonical = json.dumps(payload, sort_keys=True, separators=(",", ":")) + return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def operation_rows( + binding: Any, + prepared: PreparedSave, + params: dict[str, str], + *, + record: list[tuple[str, str]], +) -> list[tuple[str, str]]: + """``(label, value)`` rows a person reads to approve an API step. + + The values this run supplies (parameters and options chosen by label), + then the record the check reads back. The form's own defaults and the + session values are counted, not listed: they are the recorded form's, + and the digest covers them. + """ + + spec = binding.web_form + sent = dict(prepared.body) + rows: list[tuple[str, str]] = [] + defaults = 0 + for field in spec.fields: + if field.source == "param": + rows.append((field.name, sent.get(field.name, ""))) + elif field.source == "select_label": + label = str(params.get(field.param or "", "")) + rows.append((field.name, label)) + elif field.source == "form_default": + defaults += 1 + for name, value in record: + rows.append((f"Record check: {name}", value)) + rows.append( + ( + "Request", + f"{prepared.method.upper()} {prepared.path}; {defaults} form default(s) " + f"and {len(prepared.session_fields)} session value(s) as recorded", + ) + ) + return rows + + +def render_operation_png(title: str, rows: list[tuple[str, str]]) -> bytes: + """A plain image of ``rows``, shown where an approval shows a screenshot. + + Desktop and the phone show the request's image. An API step fills no + form, so the image is the request itself, not the screen. + """ + + import io + + from PIL import Image, ImageDraw + + line_h, pad, width = 18, 16, 900 + lines = [title, ""] + [f"{label}: {value}" for label, value in rows] + height = pad * 2 + line_h * len(lines) + image = Image.new("RGB", (width, height), "white") + draw = ImageDraw.Draw(image) + for i, line in enumerate(lines): + draw.text((pad, pad + i * line_h), line[:140], fill="black") + out = io.BytesIO() + image.save(out, format="PNG") + return out.getvalue() + + # -- actuator -------------------------------------------------------------------- @@ -262,6 +387,22 @@ def __init__(self, transport: Transport) -> None: self.log: list[dict[str, Any]] = [] def actuate(self, binding: Any, params: dict[str, str]) -> ApiActuationResult: + """Prepare the save and send it: :meth:`prepare`, then :meth:`send`.""" + + prepared = self.prepare(binding, params) + if isinstance(prepared, ApiActuationResult): + return prepared + return self.send(binding, prepared) + + def prepare( + self, binding: Any, params: dict[str, str] + ) -> "PreparedSave | ApiActuationResult": + """Everything before the save: reads only, nothing that writes. + + Returns the exact request :meth:`send` would send, or an + ``UNAVAILABLE`` result saying why the save can't be sent. + """ + spec = binding.web_form summary = f"{binding.method} {binding.url_template}" @@ -323,13 +464,24 @@ def unavailable(reason: str) -> ApiActuationResult: return unavailable(f"the run has no value for {exc}") except Exception as exc: # noqa: BLE001 - only reads were sent return unavailable(f"a read before the save failed ({type(exc).__name__})") + return PreparedSave( + method=binding.method, + path=binding.url_template, + body=tuple(body), + referer=self.transport.base_url + spec.form_path, + session_fields=frozenset( + f.name for f in spec.fields if f.source == "session" + ), + ) + + def send(self, binding: Any, prepared: "PreparedSave") -> ApiActuationResult: + """Send exactly ``prepared``, once. Never retried.""" - encoded = up.urlencode(body) + spec = binding.web_form + summary = f"{binding.method} {binding.url_template}" try: resp = self.transport.post_form( - binding.url_template, - encoded, - referer=self.transport.base_url + spec.form_path, + prepared.path, prepared.encoded(), referer=prepared.referer ) except Exception as exc: # noqa: BLE001 - after dispatch began self.log.append({"stage": "save", "error": type(exc).__name__}) diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py index fe4105905..02b3e546a 100644 --- a/packages/flow/openadapt_flow/runtime/replayer.py +++ b/packages/flow/openadapt_flow/runtime/replayer.py @@ -47,6 +47,7 @@ import time import uuid from copy import deepcopy +from dataclasses import dataclass from datetime import date, datetime, timezone from pathlib import Path from typing import ( @@ -56,6 +57,7 @@ Literal, Mapping, Optional, + Sequence, TypeVar, cast, ) @@ -641,6 +643,18 @@ def better_identity_read(first: Any, retry: Any) -> Any: return first +@dataclass(frozen=True) +class _WebFormApprovalView: + """An API step's prepared request, as a person is asked to approve it.""" + + prepared: Any + #: :func:`~openadapt_flow.runtime.actuators.web_form.operation_digest`. + digest: str + rows: list[tuple[str, str]] + png: bytes + ref: Any + + class Replayer: """Replays a Workflow against a Backend using injected vision. @@ -1033,6 +1047,12 @@ def __init__( self.web_form_actuator = web_form_actuator self._web_form_span_active = False self._web_form_fallback_reason: Optional[str] = None + # Under an approval mode, the API request a person was asked about + # (prepared before the gate), keyed by the save step's result, and + # the operation digest the gate's last-point check re-reads. + self._web_form_approval: dict[int, Any] = {} + self._web_form_form_state: Optional[tuple[str, str]] = None + self._web_form_form_state_set = False # Durable tiered runtime (RFC §5, Tier 3), OFF by default. When True, # run() writes a RunCheckpoint after each VERIFIED step and, on a halt, # a PendingEscalation instead of just dying -- so the run can be @@ -2387,16 +2407,15 @@ def _web_form_span_at( reason = "a durable or resumed run uses the GUI steps" elif getattr(self, "_surface_router", None) is not None: reason = "a run across several windows or tabs uses the GUI steps" - elif ( - self.approval_gate is not None - and str(getattr(getattr(self.approval_gate, "mode", None), "value", "")) - != "not_gated" + elif self._approval_gated() and not callable( + getattr(self._actuator_for(binding), "prepare", None) ): - # A person approves what the screen shows; an API step would - # leave the form empty. Only an ungated run uses API steps. + # Under an approval mode the person approves the exact request + # the API step sends. An actuator that can't prepare the + # request before sending it can't show it. reason = ( - "approval mode shows the person the screen before the save, " - "so the GUI steps run" + "approval mode needs the exact request before the save, and " + "this API actuator can't prepare it, so the GUI steps run" ) elif self.governed_authorization is not None: reason = "a governed run uses its qualified GUI path" @@ -2459,6 +2478,7 @@ def _run_web_form_span( ) finally: self._web_form_span_active = False + self._web_form_approval.clear() if result.api_fallback is not None: self._web_form_fallback_reason = result.api_fallback return None @@ -7106,8 +7126,23 @@ def _run_step_on_active_surface( # approved step then takes a fresh frame and resolves its target # and right-record check again. The last-point gate still re-reads # the form before the input is delivered. + api_view = None + if ( + self._web_form_span_active + and step.api_binding is not None + and self._approval_gated() + ): + # An API step fills no form: the person approves the exact + # request it sends. If it can't be prepared, nothing was sent + # and nobody was asked; the GUI steps run instead. + api_view = self._web_form_approval_view(step, params) + if isinstance(api_view, str): + result.api_fallback = api_view + result.elapsed_ms = (time.monotonic() - t0) * 1000.0 + return result + self._web_form_approval[id(result)] = api_view approval_refusal = self._approval_step_refusal( - workflow, step, step_index, result + workflow, step, step_index, result, api_view=api_view ) if approval_refusal is not None: result.ok = False @@ -8717,7 +8752,46 @@ def _try_api_tier( current_step=overlay_current, total_steps=overlay_total, ) - refusal = self._delivery_authorization_refusal(workflow, params, step, result) + approved_view = self._web_form_approval.pop(id(result), None) + send_prepared: Optional[Any] = None + again: Any = None + if approved_view is not None: + # Under an approval mode: prepare the request again now (fresh + # form, fresh pre-checks). The last-point gate compares its + # operation digest with the one a person approved; a request that + # can't be prepared again reads as an unreadable form. + again = self._web_form_approval_view(step, params) + self._web_form_form_state = ( + None if isinstance(again, str) else ("api_request", again.digest) + ) + self._web_form_form_state_set = True + try: + refusal = self._delivery_authorization_refusal( + workflow, params, step, result + ) + finally: + self._web_form_form_state = None + self._web_form_form_state_set = False + if refusal is None and approved_view is not None: + if isinstance(again, str) or again.digest != approved_view.digest: + # Also refused when the gate let it through (for example a + # save that passed problems_only's checks without a person): + # only the exact request prepared before the gate is sent. + why = ( + again + if isinstance(again, str) + else "the request, the record or the record check differs" + ) + result.safety_halt = True + result.failure_category = "governed_refusal" + refusal = ( + f"Step '{step.id}' ({step.intent}): the API request changed " + f"after the approval gate ({why}); the save was not sent " + "and the GUI steps do not run in its place" + ) + else: + # Exactly the request whose digest was approved and re-checked. + send_prepared = again.prepared if refusal is not None: result.ok = False result.error = refusal @@ -8732,10 +8806,14 @@ def _try_api_tier( result.delivery_attempted = True try: self._require_qualification_environment_current() - outcome = api_actuator.actuate( - binding, - runtime_params_for_gui(params), - ) + if send_prepared is not None: + # Exactly the request whose digest was approved and re-checked. + outcome = api_actuator.send(binding, send_prepared) + else: + outcome = api_actuator.actuate( + binding, + runtime_params_for_gui(params), + ) except Exception as exc: # noqa: BLE001 - external actuator boundary # The actuator contract is no-throw, but a deployment adapter can # still violate it. The delivery boundary was crossed immediately @@ -11525,11 +11603,81 @@ def frame() -> Optional[bytes]: "Approval mode: this item stopped before saving." ) + def _approval_gated(self) -> bool: + """Whether an approval mode other than ``not_gated`` governs this run.""" + + gate = self.approval_gate + return gate is not None and ( + str(getattr(getattr(gate, "mode", None), "value", "")) != "not_gated" + ) + + def _web_form_approval_view( + self, step: Step, params: Mapping[str, RuntimeParamScalar] + ) -> "Any | str": + """Prepare an API step's request for a person to approve. + + Returns the view (the prepared request, its operation digest, the + rows and image the person sees), or the reason it can't be prepared. + Only reads are sent. + """ + + from openadapt_flow.approval_mode.item_approval import ApiRequestRef + from openadapt_flow.runtime.actuators import ApiActuationResult + from openadapt_flow.runtime.actuators.web_form import ( + operation_digest, + operation_rows, + render_operation_png, + ) + + binding = step.api_binding + assert binding is not None + actuator = self._actuator_for(binding) + if actuator is None or not callable(getattr(actuator, "prepare", None)): + return "this API actuator can't prepare the request for approval" + gui_params = runtime_params_for_gui(params) + try: + prepared = actuator.prepare(binding, gui_params) + except Exception as exc: # noqa: BLE001 - only reads were sent + return f"preparing the request failed ({type(exc).__name__})" + if isinstance(prepared, ApiActuationResult): + return prepared.reason or "the API step is unavailable" + effects = self._resolve_effects(list(binding.effects), params) + record = [ + (key, str(expr)) + for effect in effects + for key, expr in sorted(effect.match.items()) + ] + digest = operation_digest( + prepared, + record_check=[effect.contract_hash() for effect in effects], + record=record, + ) + rows = operation_rows(binding, prepared, gui_params, record=record) + return _WebFormApprovalView( + prepared=prepared, + digest=digest, + rows=rows, + png=render_operation_png( + f"Approve this request: {step.intent or step.id}", rows + ), + ref=ApiRequestRef( + method=prepared.method.upper(), + path=prepared.path, + payload_sha256=prepared.payload_sha256(), + ), + ) + def _approval_form_state( self, frame_png: Optional[bytes] ) -> Optional[tuple[str, str]]: - """The form's current field state digest, for the approval gate.""" + """The form's current field state digest, for the approval gate. + For an API step under an approval mode it is the operation digest of + the request about to be sent (set just before the last-point gate). + """ + + if self._web_form_form_state_set: + return self._web_form_form_state from openadapt_flow.approval_mode.form_state import capture_form_state try: @@ -11560,8 +11708,14 @@ def _approval_step_refusal( step: Step, step_index: int, result: StepResult, + *, + api_view: Optional[Any] = None, ) -> Optional[str]: - """Ask the approval gate before a save step resolves its target.""" + """Ask the approval gate before a save step resolves its target. + + ``api_view`` is the prepared request of an API step: the person is + then shown that request, and its operation digest is the "form". + """ gate = self.approval_gate assert gate is not None @@ -11569,11 +11723,35 @@ def _approval_step_refusal( drill_fault_step = self._is_qualification_fault_step(step) def frame() -> Optional[bytes]: + if api_view is not None: + return api_view.png try: return self.vision.wait_settled(self.backend) except Exception: # noqa: BLE001 - no frame means no approval return None + def form_state(frame_png: Optional[bytes]) -> Optional[tuple[str, str]]: + if api_view is not None: + return ("api_request", api_view.digest) + return self._approval_form_state(frame_png) + + def save_fields() -> Optional[Sequence[Any]]: + if api_view is None: + return None + from openadapt_flow.approval_mode.item_approval import ( + MAX_SAVE_FIELDS, + SaveField, + ) + + rows = api_view.rows + if len(rows) > MAX_SAVE_FIELDS: + # The image lists every row; the phone shows it alone. + return None + return [ + SaveField(label=label[:128] or "-", after=value[:512]) + for label, value in rows + ] + try: from openadapt_flow.approval_mode.gate import StepContext @@ -11585,7 +11763,9 @@ def frame() -> Optional[bytes]: step_index=step_index, run_id=self._run_id or "", frame=frame, - form_state=self._approval_form_state, + form_state=form_state, + save_fields=save_fields, + api_request=api_view.ref if api_view is not None else None, drill_fault_step=drill_fault_step, qualification_case_kind=( authorization.qualification_case_kind diff --git a/packages/flow/tests/test_api_step_approval.py b/packages/flow/tests/test_api_step_approval.py new file mode 100644 index 000000000..98f69e5e7 --- /dev/null +++ b/packages/flow/tests/test_api_step_approval.py @@ -0,0 +1,317 @@ +"""An API step under an approval mode (docs/API_TIER.md, docs/APPROVAL_MODES.md). + +The person approves the exact request the API step sends: the values, the +record the check reads back and the record check, bound by one digest to the +request bytes. Not a screenshot of a form the API step never fills. + +- the run prepares the request (reads only), asks, and sends exactly that + request once after a person's approval; +- a request that changes while the person looks is not sent, and the GUI + steps don't run in its place; +- a decline sends nothing; +- a request that can't be prepared asks nobody: the GUI steps run, and their + save asks the person as before. + +The transport is a fake that writes to the MockMed fault server; the values +are made up. +""" + +from __future__ import annotations + +import hashlib +import json +import urllib.parse as up +from typing import Any, Optional + +import requests + +from openadapt_flow.approval_mode.gate import ApprovalGate +from openadapt_flow.approval_mode.item_approval import ( + ItemApprovalStore, + make_answer, +) +from openadapt_flow.approval_mode.model import ItemDecisionKind +from openadapt_flow.compiler.api_proposal import admit_api_step +from openadapt_flow.ir import WebFormField, Workflow +from openadapt_flow.mockmed.fault_server import serve as fault_serve +from openadapt_flow.runtime.actuators.web_form import WebFormActuator +from openadapt_flow.runtime.effects import RestRecordVerifier +from openadapt_flow.runtime.replayer import Replayer +from tests.test_api_tier_web_form import TARGET, EnterWritesBackend, _span_workflow +from tests.test_replayer import FakeVision + +APPROVE: dict[str, Any] = dict( + decision="approve", + decided_by="human", + route="desktop_run_it", + approver_id="cloud-user-0001", + auth_method="cloud_aal2", +) +PARAMS = {"lname": "Marsh"} +FORM = """
+ + +
""" + + +class SessionTransport: + """The live session: a form page, a duplicate search and the save.""" + + base_url = "http://app" + + def __init__(self, sor_url: str) -> None: + self.sor_url = sor_url + self.token = "session-1" + self.dup = "No matches were found." + self.posts: list[tuple[str, str]] = [] + + def get(self, path: str) -> Any: + text = FORM.format(token=self.token) if path == "/new.php" else self.dup + return _Resp(200, text) + + def post_form(self, path: str, body: str, *, referer: str) -> Any: + self.posts.append((path, body)) + requests.post( + f"{self.sor_url}/api/encounter", + json={**TARGET, "note": "api"}, + timeout=5, + ) + return _Resp(200, "saved") + + +class _Resp: + def __init__(self, status: int, text: str) -> None: + self.status, self.text, self.headers = status, text, {} + + +class Person: + """Answers each waiting request; ``on_wait`` runs first, each poll.""" + + def __init__(self, decisions: list[Optional[dict[str, Any]]]) -> None: + self.decisions = list(decisions) + self.clock = 0.0 + self.store: Optional[ItemApprovalStore] = None + self.on_wait: Any = None + self.seen: list[str] = [] + + def monotonic(self) -> float: + return self.clock + + def sleep(self, seconds: float) -> None: + self.clock += seconds + if self.on_wait is not None: + self.on_wait() + assert self.store is not None + for request in self.store.requests(): + if request.request_id in self.seen: + continue + self.seen.append(request.request_id) + decision = self.decisions.pop(0) if self.decisions else None + if decision is not None: + self.store.write_answer(make_answer(request, **decision)) + + +def _workflow() -> Workflow: + """The span program, its save also sending the session's CSRF value.""" + + wf = _span_workflow(admitted=False) + save = wf.steps[-1] + binding = save.api_binding + spec = binding.web_form.model_copy( + update={ + "fields": [ + WebFormField(name="csrf", source="session"), + *binding.web_form.fields, + ] + } + ) + binding = binding.model_copy(update={"web_form": spec}) + wf = wf.model_copy( + update={ + "steps": [*wf.steps[:-1], save.model_copy(update={"api_binding": binding})] + } + ) + wf, _ = admit_api_step(wf, admitted_by="reviewer") + return wf + + +def _run(tmp_path, person: Person, *, transport_setup=None): + url, db, stop = fault_serve() + url = url.rstrip("/") + try: + run_dir = tmp_path / "run" + store = ItemApprovalStore(run_dir) + person.store = store + transport = SessionTransport(url) + if transport_setup is not None: + transport_setup(transport) + backend = EnterWritesBackend(url) + gate = ApprovalGate( + "each_item", + approval_store=store, + approval_wait_s=30.0, + approval_poll_s=1.0, + sleep=person.sleep, + monotonic=person.monotonic, + ) + bundle = tmp_path / "bundle" + (bundle / "templates").mkdir(parents=True, exist_ok=True) + report = Replayer( + backend, + vision=FakeVision(), + effect_verifier=RestRecordVerifier(url), + web_form_actuator=WebFormActuator(transport), + approval_gate=gate, + poll_interval_s=0.01, + ).run(_workflow(), bundle_dir=bundle, run_dir=run_dir, params=PARAMS) + return report, backend, transport, store, db.snapshot()["records"] + finally: + stop() + + +def test_the_person_approves_the_exact_request_and_it_is_sent_once(tmp_path): + report, backend, transport, store, records = _run(tmp_path, Person([APPROVE])) + + assert report.success is True, [r.error for r in report.results] + assert backend.actions == [] + ((path, body),) = transport.posts + assert path == "/save.php" and body == "csrf=session-1&lname=Marsh" + folded, save = report.results + assert folded.actuation == "api_folded" and save.actuation == "api" + assert save.effect_verified is True + assert [r["note"] for r in records] == ["api"] + # A person approved the one save that was delivered. (This fixture runs + # without an execution profile, so the outcome reads "Finished, not + # checked" and the item still goes to a person; the Regulated profile + # case is in the compatibility matrix.) + assert [row.decision for row in report.approval.approvals] == ["approved"] + assert report.approval.decision_reason != "save_not_approved" + + (request,) = store.requests() + assert request.form_state_kind == "api_request" + assert store.status(request) == "used" + # The approved digest is the digest of the bytes that were sent. + sent = { + "method": "POST", + "path": path, + "content_type": "application/x-www-form-urlencoded", + "body": body, + "referer": "http://app/new.php", + } + canonical = json.dumps(sent, sort_keys=True, separators=(",", ":")) + assert request.api_request.payload_sha256 == ( + "sha256:" + hashlib.sha256(canonical.encode()).hexdigest() + ) + # The person reads the values, the record and the request, not a form. + rows = {field.label: field.after for field in request.fields} + assert rows["lname"] == "Marsh" + assert rows["Record check: patient_id"] == TARGET["patient_id"] + assert rows["Request"].startswith("POST /save.php") + # The session value is in the digest, never in front of the person. + assert "session-1" not in json.dumps(rows) + assert (store.run_dir / request.screenshot).read_bytes()[:4] == b"\x89PNG" + # Nothing about the request reaches the report. + saved = (tmp_path / "run" / "report.json").read_text() + assert request.api_request.payload_sha256 not in saved + assert "Marsh" not in json.dumps(json.loads(saved)["approval"]) + + +def test_a_request_that_changes_while_the_person_looks_is_not_sent(tmp_path): + person = Person([APPROVE]) + holder: dict[str, SessionTransport] = {} + + def keep(transport: SessionTransport) -> None: + holder["t"] = transport + + def the_session_value_changes() -> None: + holder["t"].token = "session-2" + + person.on_wait = the_session_value_changes + report, backend, transport, store, records = _run( + tmp_path, person, transport_setup=keep + ) + + assert transport.posts == [] and records == [] + # Not redone through the GUI either: the person approved this request. + assert backend.actions == [] + assert report.success is False + assert report.transaction_outcome == "HALTED_BEFORE_EFFECT" + assert report.approval.decision is ItemDecisionKind.STOP_BEFORE_SAVING + failed = [c for c in report.approval.pre_checks if not c.passed] + assert failed[-1].detail == "form_changed_after_approval" + + +def test_a_failed_check_after_approval_is_not_sent(tmp_path): + person = Person([APPROVE]) + holder: dict[str, SessionTransport] = {} + + def keep(transport: SessionTransport) -> None: + holder["t"] = transport + + def a_duplicate_appears() -> None: + holder["t"].dup = "1 match found" + + person.on_wait = a_duplicate_appears + report, backend, transport, store, records = _run( + tmp_path, person, transport_setup=keep + ) + + assert transport.posts == [] and records == [] and backend.actions == [] + assert report.approval.decision is ItemDecisionKind.STOP_BEFORE_SAVING + failed = [c for c in report.approval.pre_checks if not c.passed] + assert failed[-1].detail == "form_state_unreadable" + + +def test_a_decline_sends_nothing(tmp_path): + decline = dict(APPROVE, decision="decline", decline_reason="wrong_values") + report, backend, transport, store, records = _run(tmp_path, Person([decline])) + + assert transport.posts == [] and records == [] and backend.actions == [] + assert report.approval.decision is ItemDecisionKind.DECLINED + + +def test_a_request_that_cant_be_prepared_asks_nobody_and_runs_the_gui(tmp_path): + def duplicate(transport: SessionTransport) -> None: + transport.dup = "1 match found" + + # Nobody answers: the GUI save then waits for a person and stops. + report, backend, transport, store, records = _run( + tmp_path, Person([None]), transport_setup=duplicate + ) + + assert transport.posts == [] + first = report.results[0] + assert "did not pass" in first.api_fallback + assert ("press", "Tab") in [(a[0], a[1]) for a in backend.actions] + # The one request the person saw is the GUI save's, not an API request. + assert [r.form_state_kind for r in store.requests()] in ([], ["screen_text"]) + assert all(r.api_request is None for r in store.requests()) + + +def test_operation_digest_binds_the_record_check(): + from openadapt_flow.runtime.actuators.web_form import ( + PreparedSave, + operation_digest, + ) + + prepared = PreparedSave( + method="POST", + path="/save.php", + body=(("lname", "Marsh"),), + referer="http://app/new.php", + ) + base = operation_digest(prepared, record_check=["h1"], record=[("id", "p1")]) + assert base != operation_digest( + prepared, record_check=["h2"], record=[("id", "p1")] + ) + assert base != operation_digest( + prepared, record_check=["h1"], record=[("id", "p2")] + ) + changed = PreparedSave( + method="POST", + path="/save.php", + body=(("lname", "Marsh "),), + referer="http://app/new.php", + ) + assert base != operation_digest(changed, record_check=["h1"], record=[("id", "p1")]) + assert up.parse_qs(prepared.encoded()) == {"lname": ["Marsh"]} diff --git a/packages/flow/tests/test_api_tier_web_form.py b/packages/flow/tests/test_api_tier_web_form.py index f5b897d16..f0f0fe58a 100644 --- a/packages/flow/tests/test_api_tier_web_form.py +++ b/packages/flow/tests/test_api_tier_web_form.py @@ -672,18 +672,21 @@ def test_gui_steps_run_when_the_api_step_may_not(tmp_path, mutate, kw, reason): assert ("press", "Tab") in [(a[0], a[1]) for a in backend.actions] -@pytest.mark.parametrize( - "mode, runs", [("each_item", False), ("problems_only", False), ("not_gated", True)] -) -def test_only_an_ungated_run_uses_api_steps(mode, runs): +@pytest.mark.parametrize("mode", ["each_item", "problems_only", "not_gated"]) +@pytest.mark.parametrize("can_prepare", [True, False]) +def test_a_gated_run_uses_api_steps_that_can_show_their_request(mode, can_prepare): from openadapt_flow.approval_mode.model import ApprovalMode - replayer = Replayer(FakeBackend(), vision=FakeVision(), web_form_actuator=object()) + actuator = WebFormActuator(FakeTransport()) if can_prepare else object() + replayer = Replayer(FakeBackend(), vision=FakeVision(), web_form_actuator=actuator) replayer.approval_gate = type("Gate", (), {"mode": ApprovalMode(mode)})() _end, _save, reason = replayer._web_form_span_at(_span_workflow(), 0, durable=False) + # Under an approval mode the person approves the exact request + # (tests/test_api_step_approval.py), so the actuator must prepare it. + runs = can_prepare or mode == "not_gated" assert (reason is None) is runs if not runs: - assert "approval mode" in reason + assert "approval mode needs the exact request" in reason class LostResponseActuator(ScriptedActuator): From b17b50aee28d9c3f87ab22a34ff72ec95e4e236d Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 12:04:11 -0400 Subject: [PATCH 2/3] docs: refresh llms-full.txt Co-Authored-By: Claude Opus 5.5 --- llms-full.txt | 68 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 54 insertions(+), 14 deletions(-) diff --git a/llms-full.txt b/llms-full.txt index 623dfd5b6..aa1162799 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -375,7 +375,8 @@ shipping in the next release; a pull request number means it is being merged. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles the verified run into a program (`openadapt_flow.jit`, released; - prompt-to-program `compile-agent`, landing). Agents can drive the recording + prompt-to-program `compile-agent`, landing; the agent can act through code + instead of screenshots, [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)). Agents can drive the recording on web, native Windows, RDP and Citrix. 2. **Compiles to the fastest layer the app allows.** Page structure first (DOM, Windows UI Automation, macOS Accessibility), direct API calls where the app's @@ -436,9 +437,10 @@ shipping in the next release; a pull request number means it is being merged. | Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | +| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. It is arm 7 of the scoreboard; its figures are quoted from there once its rows are in. | `CODE_AS_ACTION.md` in [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | -| Ships as one monorepo with separately installable packages (flow, agent, types, capture, and privacy and desktop moving in). | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | +| Ships as one monorepo with separately installable packages (flow, agent, types, capture, privacy and desktop), each released from the monorepo. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | | Makes no model calls on healthy runs. Model training (openadapt-ml) is separate research. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | ## The full map @@ -474,6 +476,7 @@ improvising or retrying when something is off. | Compile one or more demonstrations into a deterministic program | Released | Real app (OpenEMR referral, 62 timed runs) | `openadapt-flow record`, `compile`; [packages/flow](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/README.md) | | Right-record check before any write (structure, then OCR, then pixels) | Released | Real app and synthetic | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md), `runtime/identity.py` | | Read the saved record back from the system of record (API, database, FHIR, separate session) | Released | Real app (OpenEMR 12 of 12 fields) | [VERIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION.md), `runtime/effects/` | +| Read values out of an app as a workflow step (READ), with a records format, a signed review queue and export to CSV or a database | On main ([#1229](https://github.com/OpenAdaptAI/OpenAdapt/pull/1229), [#1215](https://github.com/OpenAdaptAI/OpenAdapt/pull/1215), [#1218](https://github.com/OpenAdaptAI/OpenAdapt/pull/1218)–[#1224](https://github.com/OpenAdaptAI/OpenAdapt/pull/1224)) | Synthetic (mock desktop apps) | `openadapt_flow/interchange/` | | Stop before the effect instead of improvising (`HALTED_BEFORE_EFFECT`) | Released | Real app and synthetic | `runtime/replayer.py` | | Never retry an uncertain save (`RECONCILIATION_REQUIRED`) | Released | Synthetic (fault server) | [VERIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION.md) | | One check-level scale, 0 to 5, from "unchecked" to "counterparty confirmed" | Released | Unit tests | `openadapt_flow/check_level.py` | @@ -508,7 +511,7 @@ improvising or retrying when something is off. | Watch many real runs in the background and mine candidate branches (shadow mode) | On main ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)) | Synthetic | [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Fill a workflow's inputs from a referral PDF, fax image or email | Released | Synthetic | `openadapt-flow intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | | Use spoken narration while recording to propose the rule behind a branch | In review ([#1291](https://github.com/OpenAdaptAI/OpenAdapt/pull/1291)) | Synthetic (generated audio) | `openadapt-capture[audio,privacy]` | -| Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | In review ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276)) | Synthetic (MockMed) | `induce --probe-on` | +| Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | On main ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276), [#1289](https://github.com/OpenAdaptAI/OpenAdapt/pull/1289)) | Synthetic (MockMed; desktop and remote test copies) | `induce --probe-on` | | Route each item to the right verified variant of a program | On main ([#1264](https://github.com/OpenAdaptAI/OpenAdapt/pull/1264)) | Synthetic | `variants`, `route` | ## Surfaces @@ -519,7 +522,8 @@ improvising or retrying when something is off. | Native Windows (UI Automation) | Released, early | Synthetic (WinForms, 3 of 3 trials) | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md) | | Native macOS (Accessibility) | Released, early | Synthetic (TextEdit, 3 of 3 trials) | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md) | | RDP (pixels; Aardwolf or headless FreeRDP with a session pool) | Released, early | Real Windows 11 host, 3 of 3, plus a FreeRDP batch | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md), `backends/xfreerdp.py` | -| Citrix | Exploratory | A no-DOM stand-in only; no real ICA/HDX session yet | [LIMITS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md) | +| Citrix and other virtual desktops, with OpenAdapt installed inside the session (`in_session`): the same structure-first path as a local desktop | Released, early | Same evidence as the Windows path; no live Citrix session yet | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md) | +| Citrix from outside the session, pixels only (`remote_display`), the fallback when nothing can be installed inside | Exploratory | A no-DOM stand-in only; no real ICA/HDX session yet | [LIMITS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md) | | Chain programs across surfaces, passing confirmed values | Released | Synthetic | `openadapt-flow compose`, `process`; [PROCESS_CONTRACT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/PROCESS_CONTRACT.md) | | One program that switches surfaces, tabs and popups mid-run | On main ([#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197)) | Real Chromium; Windows switch on a mock | [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Halt when a workflow qualified with page structure runs on a pixels-only surface | Released | Synthetic | `openadapt-flow qualify` | @@ -533,7 +537,11 @@ improvising or retrying when something is off. | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | | Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | | Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | -| Run a worklist on several workers at once, one signed-in session each, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--workers` | +| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-an-approval-mode) | +| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| Record where each step's time goes in every run report | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | ## When the app changes @@ -544,7 +552,7 @@ improvising or retrying when something is off. | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | In review ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | | Self-repair across several new steps (a new dialog, a control moved into a menu) | In progress | Synthetic (MockMed drift modes) | bridge branch, stacked on #1265 | -| Keep pixel-path runs working across display scaling and font smoothing changes | In review ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (RDP and Citrix-style renderings) | `vision/match.py` | +| Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | In review ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | | Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) | @@ -553,9 +561,11 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | +| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | -| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Synthetic | `compile-agent` | +| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | +| Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | | One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | ## Recording, privacy and proof @@ -565,6 +575,7 @@ improvising or retrying when something is off. | Record mouse, keyboard, screen video and accessibility trees on one clock (Windows, macOS, Linux X11) | Released (capture 1.3.0) | Real OS | [packages/capture](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/README.md); Wayland is not supported | | Find and redact personal and health data in text, screenshots and structure | Released (privacy 1.1.3) | Unit tests | [packages/privacy](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/privacy/README.md), [PRIVACY.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/PRIVACY.md) | | Scrub by default before anything leaves the machine | On main ([#1262](https://github.com/OpenAdaptAI/OpenAdapt/pull/1262)) | Unit tests | `openadapt-privacy` | +| Each package installs on its own within a size budget, checked in CI (privacy without desktop, capture without plotting) | On main ([#1270](https://github.com/OpenAdaptAI/OpenAdapt/pull/1270), [#1271](https://github.com/OpenAdaptAI/OpenAdapt/pull/1271), [#1279](https://github.com/OpenAdaptAI/OpenAdapt/pull/1279)) | CI | `.github/workflows/packages.yml` | | Signed receipts (Seals), countersigned and timestamped, offline by default | Released | Unit tests, real TSA tokens | `openadapt-flow seal`, `countersign`, `local-seal` | | Strict no-outbound mode and an unattended on-prem service | Released | Synthetic | `openadapt-flow no-outbound`, `service`; [ON_PREM.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/ON_PREM.md) | @@ -574,7 +585,7 @@ Each `openadapt-flow` command, grouped by the capability it serves. A test fails when a command is added without being listed here. - **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`. -- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. +- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. - **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`. - **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). - **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). @@ -962,6 +973,31 @@ class named with `--allow-halt`. A handoff copies an effect-bound parameter from a confirmed receipt. Missing evidence HALTs. Subflows and worklists stay inside each child, on that child's surface. +## Parallel worklists + +`--workers N` runs a worklist's items on N signed-in sessions at once +([`THROUGHPUT.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md)). Every item keeps every check. The run +refuses N above the deployment's `worklist.max_workers`, fewer accounts than +workers, two accounts for the same user, two workers sharing one session, a +program with steps outside its loop, and rows not declared +`items_independent`. It can't tell on its own whether two rows touch the same +record; that declaration is yours. A halt on any worker stops new items on +all of them, and an item claimed before a crash is never run again +automatically: it is marked `uncertain` for a person to check. The pipelined +read-back window applies only to an item's last save and only to read-backs +that don't use the screen; durable runs, which include the Standard and +Regulated profiles, always read back inline. Parallel RDP runs use the +FreeRDP session pool and have unit tests only. Throughput figures come from +one machine and a local OpenEMR test copy, not from a production EMR. + +## Short parameter values + +The right-record check binds a typed value into its expected text only when +the value has at least 4 characters. An item whose bound value is shorter, +such as the first name "Bo", halts at the identity check instead of saving. +This is deliberate: a short value matches too much unrelated text to identify +a record. + ## Save triage `openadapt-flow triage` orders verified saves for review under @@ -1296,12 +1332,16 @@ grade is `screen_readback`, never `independent_readback`. **Without a read that is independent of the acting session, the run cannot be `VERIFIED`.** That read is either a system-of-record read (REST, FHIR, SQL, -file; tier 1) or the application's record page read in a separate signed-in -session (`ui_readback` for web, `fresh_session` for desktop and RDP, -tested on mocks only; tier 2). The second goes through the same -application, so it is not a system-of-record read and has its own limits -([`EFFECT_KIT.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-ui-read-back-ui_readback)). -A pixel-only Citrix or RDP session with neither is that case. Reopening the +file; tier 1) or the application's own screens read in a separate signed-in +session (tier 2): a record page in a separate browser (`ui_readback`), a +relaunched desktop app or a second RDP session (`fresh_session`, tested on +mocks only), or a look-up in a separate desktop, RDP or Citrix session +(`desktop_readback`, tier 2 with its own RDP logon). None of them goes around +the application, so none is a system-of-record read, and each has its own +limits ([`EFFECT_KIT.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-ui-read-back-ui_readback), +[`desktop_readback`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback)). +If a person can look the record up, the check can too. A pixel-only Citrix or +RDP deployment that declares no such check is the remaining case. Reopening the record in the acting session (tier 3) is reported as its own check level, `reopened` ("Reopened in the same session"), and is not `VERIFIED`: a confirmed reopen ends `COMPLETED_UNVERIFIED`, and the agent result says From 70d7ca39efbd6bfe986ac8c7b6591f161da03c32 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 12:04:27 -0400 Subject: [PATCH 3/3] docs: name #1355 in the capability row Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 2 +- llms-full.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index d2c6419f0..20e8806f6 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -181,7 +181,7 @@ improvising or retrying when something is off. | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | | Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | | Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | -| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-an-approval-mode) | +| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-an-approval-mode) | | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | diff --git a/llms-full.txt b/llms-full.txt index aa1162799..a2cf209b6 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -537,7 +537,7 @@ improvising or retrying when something is off. | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | | Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | | Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | -| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-an-approval-mode) | +| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-an-approval-mode) | | Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | | Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | | A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) |