diff --git a/packages/flow/README.md b/packages/flow/README.md index eb378fe9c..ffcdc4d46 100644 --- a/packages/flow/README.md +++ b/packages/flow/README.md @@ -289,9 +289,12 @@ Install the extras for the surface that records and replays the workflow: each surface, and [docs/PRODUCT_STATUS.md](docs/PRODUCT_STATUS.md) lists the evidence for each one. A bundle is bound to the surface it was recorded on. -One bundle drives one surface: one browser tab, one desktop app, or one -remote desktop session. For a task that crosses surfaces, record one bundle -per surface and chain them with `openadapt-flow compose`. Each child starts +A bundle drives one surface by default: one browser tab, one desktop app, or +one remote desktop session. A bundle can also declare more than one surface: +popups and new tabs it opens, or another backend it switches to with an +explicit, identity-checked step. See [docs/MULTI_SURFACE.md](docs/MULTI_SURFACE.md) +for this and its limits. Otherwise, record one bundle per surface and chain +them with `openadapt-flow compose`. Each child starts only after the one before it ends with a checked save, unless you allow a specific stop with `--allow-halt`. diff --git a/packages/flow/docs/BROWSER_RECORDING.md b/packages/flow/docs/BROWSER_RECORDING.md index 6368a71a3..f19b8a481 100644 --- a/packages/flow/docs/BROWSER_RECORDING.md +++ b/packages/flow/docs/BROWSER_RECORDING.md @@ -76,6 +76,12 @@ Flow selects the sole open HTTP or HTTPS tab on the `--url` origin. It does not navigate the tab. It also does not close the tab or browser when recording finishes. +Attach mode follows a popup or tab that the selected tab (or a page it already +follows) opened, on the declared origin. These pages are recorded as surfaces, +as they are in a browser Flow launched (`record --url`); see +[MULTI_SURFACE.md](MULTI_SURFACE.md). A tab the person opens by hand is +refused. + If two or more open tabs have that origin, Flow refuses to guess. Supply the exact current URL: @@ -133,8 +139,10 @@ silently acting in a new, signed-out one. - The selected tab must stay on that origin for the full recording. A cross-origin navigation stops the recording and does not produce complete metadata. -- Do not open a popup or a new tab in the selected tab's browser context while - recording. Flow currently binds one page. It installs a new-page latch on +- Do not open a new tab by hand in the selected tab's browser context while + recording. A popup or tab that a followed page opens is followed, judged by + `window.opener` or the browser target's opener id. Any other new page is + refused. Flow installs a new-page latch on every candidate context before it reads the accepted page baseline or selects the recording tab. Listener registration does not replay pre-existing tabs, so those tabs remain allowed. The selected-context latch records every later @@ -371,7 +379,7 @@ Attach mode supports local Chromium-family browsers that expose a CDP endpoint. It requires a browser process started with remote debugging and a separate user-data directory. It does not claim Firefox, WebKit, arbitrary Chrome extensions, an ordinary browser process that was not started for local -debugging, cross-origin tab selection, multi-page/popup recording, or direct +debugging, cross-origin tab selection, cross-origin popups, or direct extension replay. Same-origin iframes are recorded: every emitted coordinate is composed into @@ -395,8 +403,11 @@ Refused loudly, never recorded silently wrong: dropdown UI that produces no recordable action events, so the choice would be silently absent from the compiled workflow. The recording stops with an explicit error at the moment of selection. -- A new tab or popup. The recording is bound to one tab and stops when - another page appears. +- In attach mode, a new tab that no followed page opened, or a popup on + another origin. Popups and tabs that a followed page opens are recorded as + surfaces ([MULTI_SURFACE.md](MULTI_SURFACE.md)). A popup that closes itself + is followed by an explicit switch back. A popup whose first action closes + it, before any frame of it was captured, stops the recording. - An action in a frame whose page-space position cannot be proven (see above). diff --git a/packages/flow/docs/LIMITS.md b/packages/flow/docs/LIMITS.md index 3dc6927e3..49fbd3dea 100644 --- a/packages/flow/docs/LIMITS.md +++ b/packages/flow/docs/LIMITS.md @@ -463,7 +463,7 @@ and exposes independently verifiable outcomes. | --- | --- | | Zoom, DPI, font, layout, viewport, or monitor changes | Browser attach recording accepts a stable viewport or device-scale change between actions, starts a new exact geometry epoch, and refuses an action that overlaps the transition. Native offline conversion can normalize a stable source-window move, resize, monitor change, or scale change into one fixed output viewport when every timeline row carries valid mapping metadata. It refuses a malformed timeline or a changed output viewport. RDP and Citrix live backends can rebaseline geometry between actions and refuse a change during the exact-frame lease. Other rescale or reflow behavior remains workflow-specific. | | Native select menus, file choosers, permission prompts, and secure desktops | OS or browser chrome may not appear in the captured surface and may be unrecordable or undrivable. Prefer an application-level or keyboard/API path. | -| New tabs and windows | Opening a tab can be observed structurally where the backend supports it; interaction inside additional windows and multi-window coordination are not a general supported path. | +| New tabs and windows | A bundle recorded with `Recorder` can declare the popups and tabs a step opens, act in them after an explicit surface switch, and halt on any page it did not declare ([MULTI_SURFACE.md](MULTI_SURFACE.md)). Such bundles run governed, qualified, and durable. Resume re-binds surfaces by identity, and managed Cloud dispatch needs only the `surfaces:` map. A program (graph) resume is refused. `record --url` and attach mode (`--cdp`, only pages a followed page opened) capture popups and tabs, and `--surface NAME=KIND[:TARGET]` binds backend surfaces at replay. Native windows and JavaScript dialogs are screen states, not surfaces. A dialog is answered by the step that raises it (browser backend, scripted recorder), and a hidden native window is reached by an explicit focus step. | | Drag and drop or gesture-heavy controls | Not a general supported primitive. Use a structured/API alternative or validate a purpose-built workflow. | | Dates, locale-sensitive fields, and auto-formatting | The same keystrokes can produce different values across platforms or locales. Require an explicit read-back or system effect and test the deployment locale. | | Slow or asynchronous applications | A state that outlasts configured waits can halt. Increasing a timeout does not establish correctness. | diff --git a/packages/flow/docs/MULTI_SURFACE.md b/packages/flow/docs/MULTI_SURFACE.md new file mode 100644 index 000000000..b31198831 --- /dev/null +++ b/packages/flow/docs/MULTI_SURFACE.md @@ -0,0 +1,604 @@ +# Multi-surface workflows: tabs, popups, and backend switches + +Status: recorder (scripted, interactive, and attach mode), compiler, IR, +replayer, CLI binding, governed/qualified/durable runs, resume by identity, +and managed dispatch. This page is the design and its +limits. The audit that motivated it is at the end. + +A bundle used to run on exactly one surface: one browser tab, one desktop app, +or one remote-desktop window. Real tasks cross that boundary. A referral opens +a payer portal in a new tab, a "Sign in with..." button opens an OAuth popup, +a print button opens a print-preview window, and a back-office task moves from +a web app into a native Windows app or a Citrix window. This design lets one +bundle span those surfaces without weakening any guarantee a single-surface +bundle has. + +Two rules come before coverage: + +1. **Every surface gets the same gates.** The identity ladder, the effect + gates, risk, approval, and postconditions run on whatever surface a step + runs on. Nothing is skipped because a step is in a popup or a second + backend. +2. **Halt, never guess.** Flow never decides by itself which window to drive. + A page nobody declared, a declared page that did not open, a page whose + URL does not match, a backend of the wrong kind, or a step that would run + on a surface other than the active one all halt the run with a reason. + +A healthy multi-surface replay stays $0: no model call is needed to open, +claim, identify, or switch a surface. + +## Vision first: what a surface is + +A bundle-level surface exists only at an **actuator boundary**: a browser +driven through its pages, a local desktop, or a remote RDP/Citrix session. +Tabs, popups, native windows and dialogs inside one actuator are **screen +states** of that actuator. Per-step visual identity checks and preconditions +handle them, the same way they handle a screen that changed inside one window. +This design adds no surface kind. + +**A surface is a name plus an identity check, never a live handle.** + +- *Binding* means finding the one live target that passes the check. When no + target passes, or several do, the run halts. Flow never guesses. +- *The check is vision first.* It looks at screen text and pixels first: the + step's anchor template, or for a switch the screen text mined from the first + frame of the target. Structured attributes (origin, URL pattern, title + pattern, opener link, window class) are used only where the actuator exposes + them. When both kinds are available, both must pass. +- *Handles are a cache.* The runtime keeps the page or backend it bound, and + while that page stays open and is the only candidate, the next step uses it + without another check. The cache is never trusted across a process boundary + or a closed page. It is rebuilt by identity. +- *A page surface rebinds only to a page this run claimed*, and only when the + step's visual evidence (anchor template or recorded screen text) shows on + it. Structured attributes alone never rebind a page. The one exception is a + resume: before a surface is first bound again, the pages the earlier run + opened are candidates too, with the same visual-evidence rule. In attach + mode this keeps the person's own tabs out of the run. +- *Checkpoints store the name and the last-seen observation, never a handle* + (see "Resume"). + +### Routing a step to a page + +Inside the browser backend each step goes to the page whose observation passes +that step's own visual identity check (its anchor template), with structured +attributes as an additional condition. Several passing pages halt. So does a +held page that fails its check while no other candidate passes, and a step +with no passing page and no live cached page. Steady state costs nothing +extra: a live cached page that is the only candidate is used as is, and its +step's own identity ladder checks it. A step never screenshots every page. +A step never runs on a page that closed under it, whatever the surface kind: +only an explicit switch moves on. + +### Explicit switches, never automatic returns + +Execution moves only through a recorded SWITCH_SURFACE step that has its own +identity check. If a popup closes itself (an OAuth or approval window) and the +person or agent acts on the original window again, the recorder writes an +explicit switch back with that window's identity check. Nothing returns to an +opener on its own. A bundle that omits the switch back is not a valid +workflow. Once a page surface closes, it shows a blank frame, so a +postcondition that expected the closed page still to be there fails. + +### Dialogs + +A JavaScript dialog (`alert`, `confirm`, `prompt`, `beforeunload`) is not a +surface. It is modal state of the page that raised it, and it blocks that page +until it is answered. **On the browser backend** the step whose action raises +it carries `Step.dialog = {response: accept|dismiss, message | message_pattern}`: + +- before the input edge, the backend arms a one-shot answer; +- the dialog's message is its identity: only a matching message is answered as + recorded. The compiler keeps a message exactly only when it has no digits + and no demonstrated parameter value. Otherwise it seals a full-match + `message_pattern` in which each parameter value matches any text and each + digit run matches any digits, so a record number or a typed value never + enters the bundle. A message the privacy pass still flags after that is + refused at compile time; make that value a parameter; +- a different message is **dismissed, never accepted**, and the step halts. + The step also halts if the dialog never appears. The halt names a code + (`dialog_mismatch`, `dialog_missing`), never the dialog's text, because the + error lands in `report.json` and `REPORT.md`; +- an unarmed dialog keeps Playwright's default (dismiss), as before. + +The answer lives on the triggering step rather than a separate step because +the dialog blocks the page's script until it is answered. A later step could +answer it only by looking ahead or by guessing. **On screen-driven backends** +(desktop, RDP, Citrix) a dialog is pixels, recorded as the clicks that answer +it. A `dialog` on a step whose backend cannot see browser dialogs is refused +before any input. The scripted recorder records it with +`recorder.expect_dialog("accept")` before the action. + +### Two overlapping native windows on one desktop + +They are one surface: one desktop, one actuator. To act on the hidden window, +the bundle needs an explicit, checkable focus step: a click on its visible part +or a hotkey, whose postcondition is the target window's screen text. The next +step then has the usual anchor and identity checks. No window enumeration and +no new kind are involved. + +### Rejected alternatives + +- **Surface lifetimes** (long-lived vs transient surface types, automatic + return to the opener when a transient one closes). A lifetime is a guess + about the future. A popup that is "transient" in one run stays open in the + next, and an automatic return drives a window nobody checked. Explicit + switches with identity checks cover the same cases without a second state + machine. +- **OS window enumeration and set-diffing in the recorder** (list top-level + windows before and after each action, call the difference a new surface). + It is platform-specific (Win32, AX, X11/Wayland, and nothing at all through + RDP/Citrix pixels). It races windows that appear late or briefly. And it + promotes screen states (dialogs, tool windows, a second window of the same + app) into surfaces that then need lifetimes. Vision checks already see what + the person saw. +- **Routing every step by screenshotting every page.** Doing this on every + step costs one capture per page per step, and it can steal focus or wake + background tabs. The cache plus a check only on ambiguity or closure gives + the same safety. + +## The model + +### Surfaces are declared once + +`Workflow.surfaces` lists every surface, primary first: + +| Shape | How it is recognised | Example | +| --- | --- | --- | +| Primary | the first entry; the backend the run starts with | the app's main tab | +| Page | `opened_by_step` names the step whose action opened it | `popup_1`, `tab_1` | +| Backend | not opened by a step; the caller supplies it by id | `windows_1`, `rdp_1` | + +A page surface records how to identify it at replay: + +- `opened_by_step` and `open_order`: which step opened it, and its position + among the pages that step opened (the browser's creation order); +- `opener` and `opener_link`: the surface it was opened from, and whether + `window.opener` pointed at that page (`window_opener`, for `window.open`) or + not (`none`, for a `target=_blank` link, which implies `noopener`); +- `origin`: exact `scheme://host[:port]`; +- `url_pattern`: a full-match regular expression for the URL. Recorded + parameter values, and path segments that look like identifiers (any digit, + as in `/patient/12345`, or a long opaque token), match any one segment. The + query and fragment are not asserted; +- `title_pattern`: an optional full-match regular expression for the title. + The recorder no longer writes titles (they carry record names), so new + recordings leave it unset. + +The recorder writes each page's URL to `meta.json` without its query or +fragment (OAuth `code` and `state`, search terms). URLs can echo application +content. They live only inside +`workflow.json`, which is sealed with the rest of an encrypted bundle. They are +never copied into the manifest, a run report, or a refusal message. + +### Steps name their surface + +`Step.surface` is the surface a step runs on (`None` means the primary). +`Step.opens` lists, in order, the page surfaces the step's action opens. + +### A surface change is a typed step + +`ActionKind.SWITCH_SURFACE` with `surface=` is the only way execution +moves. It delivers no input and carries no anchor, text, key, effect, API +binding, guard, or irreversible risk. Its identity check is: + +1. the target is declared and is available on this path (a page only after + the step that opens it, a backend only when the caller supplied it); +2. a page target is bound by identity: it is the cached page if that page is + still open and passes, otherwise the one open, unbound page that passes. + Several matches halt. Passing means the screen text the step expects is + visible on it, and, where exposed, its origin equals `surface_origin` (the + origin the demonstration saw when it switched) and its URL/title patterns + and opener link match. Only a page claimed on this run is a candidate + (see "Identity, not handles"). Leaving a page that closed itself is + allowed; +3. a backend target is of the declared kind (a Windows surface needs a Windows + backend; a backend whose kind cannot be determined is refused); +4. the step's `expect` postconditions, mined from the first frame of the new + surface, hold on the new surface. A switch with neither such screen + evidence nor a recorded origin is refused, whatever the target kind: there + would be nothing to show that the right page or app is in front. + +Because the value is a new enum member, an older Flow that does not know it +refuses to load the bundle at all. That is deliberate: an older runtime would +otherwise ignore `Step.surface` and drive every step in the first tab. + +`surface_contract_errors()` (run whenever a workflow is built or loaded) +rejects contracts that do not line up: an undeclared surface, a page whose +opener or opening step does not match, a step whose `opens` list disagrees +with the declared pages, and, on a linear bundle, any step that would run on a +surface other than the active one or a switch to a page before it opened. + +## Recording + +`Recorder` (the scripted and agent-driven recorder used by authoring sessions, +demo drivers, and the Windows conformance tests) now: + +- notes the browser context's pages before each action on a browser surface, + and after it settles, claims every new page in creation order: it waits for + the page's first navigation, then records its opener relation and its URL + without the query or fragment, and adds `"opens": [...]` to that event; +- records `switch_surface(surface_id, backend=None, kind=None)` as its own + event. A page surface is switched to by id; a backend surface is passed in + (`recorder.switch_surface("windows_1", backend=windows_backend)`), and its + kind is read from the backend class (or given explicitly); +- stamps `"surface"` on every event once a second surface exists, and writes + `"surfaces"` to `meta.json`. + +A recording that never opens a page or switches writes the same events and +meta as before. + +### Interactive recording (`record --url`) + +When Flow launched the browser (`record --url`, headed or headless), the +interactive recorder follows popups and new tabs as surfaces and writes the +same events and `meta.json` the scripted `Recorder` writes, because it records +through that same `Recorder`: + +- the event binding and the listener script are installed on the browser + CONTEXT, so every page the app opens carries the same session id, privacy + masking, secret boundaries, and scrubbed structural reads; +- a new page is claimed (`opens`) when the action that opened it is written, + via `Recorder.record_observed`; each claimed page gets a backend with the + same screenshot masks and guard as the start tab; +- when the user acts in a different page, the pending action is written + first and a `switch_surface` event is recorded (before frame: the page the + user left; after frame: the page they acted in), then the action. + +Chromium runs a context init script in a popup's initial `about:blank` +document but not in the document its first same-origin navigation commits +into the same Window (HTML window reuse). The recorder therefore checks the +followed pages on every poll tick and installs the listener where it is +missing; the listener arms once per document. + +Design choice: reuse the scripted `Recorder`'s page claiming and switch +events, and let the user's own action (which page the event came from) decide +when a switch is recorded. Rejected: a separate interactive bundle format +(two shapes to compile and audit); asking the user to mark switches (an extra +step people forget, and the event source already says it); recording every +focus change as a switch (focus without an action is not a step). + +**Attach mode (`record --cdp`, the person's own browser)** follows popups and +tabs through the same path. The browser's target list decides which pages are +followed. A new page is followed only when one of the followed pages opened +it: `window.opener` for a popup, or the target's opener id for a `noopener` +tab. The binding and listener script are installed on that page alone, so the +rest of the person's browser is left alone. Every page keeps attach mode's +contract: the declared origin check on every event, its own exact CSS-viewport +geometry, rebound at each switch, and the same masks and guards. A tab the +person opens by hand, even one that closes again before the next poll, is +still refused. So is a page that is still open when the recording ends and +that no recorded action opened. + +**A popup that closes itself** (it writes back to its opener and calls +`window.close()`) is recorded honestly. Its closing click gets a blank after +frame and no structural state, and the next action on the original window is +recorded as an explicit `switch_surface` back. One case is refused with a +clear reason: a popup whose *first* action closes it before Flow captured any +frame of it. Nothing would anchor that action. Act in the popup once more +first, or record that step with the scripted recorder. + +A click in a new page within the first poll tick after it loads can arrive +before the listener is installed and is not recorded. Replay then halts at the +next step rather than skipping it silently. + +## Compiling + +`compile_recording` turns `meta.json` `surfaces` into `Workflow.surfaces`, each +event's `surface` and `opens` into the step's fields, and each `switch_surface` +event into a SWITCH_SURFACE step. For a switch step it mines at most one +stable TEXT_PRESENT postcondition from the first frame on the new surface +(the same volatility, persistence, parameter, and PHI filters as every other +step), and records `surface_origin` for a browser target. A switch is never a +write, so it is classified reversible. The per-event viewport check is +skipped for a switch, whose before and after frames come from different +surfaces. + +## Replay + +`Replayer(..., surface_backends={"windows_1": backend})` supplies backend +surfaces (the CLI builds this map; see "Binding backends from the CLI"). The primary backend is the one passed to the constructor. Page +surfaces need nothing from the caller: the runtime watches the browser +context of every browser surface it holds. + +For every step in a multi-surface workflow: + +- **On entry**, the runtime halts if the browser has opened a page that no + step declared, if the active page or backend has closed, or if the step + names a surface other than the active one. The step is then routed (see + "Routing a step to a page"). A page surface is bound by the step's own + anchor template plus its structured attributes. Zero or several matches + halt before any input. +- **After the action**, a step with `opens` waits (bounded by the step + timeout) for exactly that many new pages and claims them in creation order. + Each one must match its opener relation, origin, URL pattern, and title + pattern. Too few, too many, or a mismatch halts. A step without `opens` that + opened a page halts. +- **SWITCH_SURFACE** runs the identity check above, then makes the target the + active backend: the page is brought to the front, the settle seed, the + read-back verifier, and every guarded pointer or keyboard lease are reset + so nothing observed on one surface is reused on another, and the step's + postconditions are checked on the new surface. + +Remote-surface safety rules (the guarded remote actuation lease, the opaque +remote option-selection contract) now read the ACTIVE surface's kind and +execution mode instead of the bundle-level `Workflow.surface`, so an RDP +segment inside a web-first bundle keeps the remote rules. + +At the end of the run the primary backend is restored on the replayer. + +## Unexpected popups + +An unexpected popup (an OAuth window that was not part of the demonstration, +a "your session is about to expire" window, a print preview) is a new page no +step declared. The run halts at the first check that sees it, after the step +that caused it or at the next step's entry, with a reason that names the +surface the run was on and the number of undeclared pages, never their URL or +title. Flow does not close the page, switch to it, or carry on in the opener. + +## Governance, qualification, and verification + +A multi-surface run reaches `VERIFIED` under exactly the rules a +single-surface run does, and no gate is weakened: + +- **The surface map is part of the sealed contract the authorization already + binds.** `workflow.json` (with `surfaces`, each step's `surface` and + `opens`, and every SWITCH_SURFACE step) is sealed into the bundle content + digest, and the governed authorization binds that digest. A changed surface + map is a different bundle and is refused at admission like any other edit. +- **Evidence names the surface of every step.** In a multi-surface run each + `StepResult` records `surface`, the surface the step actually ran on. + `classify_execution_outcome` returns `COMPLETED_UNVERIFIED` unless every + result's `surface` equals its step's declared surface. A single-surface + report never carries the field, and one that does is not `VERIFIED`. +- **Effects stay keyed by step.** An effect contract names its own system of + record, so the same `independent_readback` rule applies: only a non-UI read + of the system of record (tier 2 or stronger) reaches `VERIFIED`; a persisted + re-read or any screen read never does, on any surface. On-screen readback + state is reset at every switch so nothing read on one surface counts for + another. +- **Qualification gets one environment boundary per surface.** A + `QualificationProject` may carry `surface_environments: {surface_id: + EnvironmentBoundary}`. A qualification case of a multi-surface bundle must + declare a boundary for every non-primary surface, with the right kind and no + extra ids. At each switch the runtime observes the target surface against + its boundary (the primary against `environment`), records an + `ObservedSurfaceEnvironment` in `report.observed_surface_environments`, and + moves the qualification input guard to the newly active backend. A mismatch + halts at the switch as a governed refusal; case evidence is + `CASE_ENVIRONMENT_CHANGED` unless every used surface has a matching, + recomputed observation. +- **Durable runs work.** Checkpoints are written per step as usual. The step + results carry their surface. + +### Resume + +Each linear checkpoint of a multi-surface run records: + +- the step's `surface`; +- the `active_surface` after the step; +- `surface_observations`: per surface, the step after which it was last + observed, its kind, its origin, and a perceptual hash of its screen. + +It holds no handle, URL path, title, or screen text. On resume, the +checkpoint's active surface is re-bound **by identity** before the approval +is consumed. For a page, the identity is the one open page that shows the +retained postconditions of the checkpointed step (visual evidence is +required) and passes the recorded structured attributes. The retained postconditions are then revalidated on +that surface. Several passing pages (an ambiguous re-bind) or none raise +`StateDiverged`, and nothing is driven. Every other surface is re-bound +lazily, at its next switch or step, by the same rule. + +A resume after a save that may have gone through still stops first at the +existing **"Check the record"** outcome (`RECONCILIATION_REQUIRED`). A person +confirms the record, then approves the resume. Re-binding never replaces that +stop. + +Still refused, with a reason: + +- a linear checkpoint without a surface observation (written by an earlier + Flow): there is nothing to re-bind by; +- resume of a multi-surface *program* (graph) run: program checkpoints do not + carry surface observations yet. + +### Cloud dispatch + +The old refusal of managed dispatch was only about handles, and it is gone. +A Cloud run needs nothing but the deployment config's `surfaces:` map: + +- the bundle digest, which the managed authorization binds, seals the surface + map; +- the hosted deployment profile, whose bytes are hashed into the dispatch, + carries the backend bindings; +- page surfaces are bound by identity at run time. + +One refusal stays because it guards something real. An unattended managed +run will not start while a declared backend surface (a Windows app, an RDP +session) has no binding. Starting anyway would reach that switch with no +backend and no person to supply one, after earlier steps may already have +written. + +Design choice: bind surfaces through the digest the authorization already +seals, record the surface per step, and add per-surface environment +boundaries only where qualification needs a new observation. Rejected: a +per-surface effect-verifier map (redundant with effect contracts, and two +sources that could disagree); a separate surface binding inside the +authorization (the bundle digest already seals the map, so a second binding +could only drift); a bundle schema bump (single-surface bytes and hashes must +stay identical, and they do: every new field is omitted when empty). + +## Binding backends from the CLI + +`replay` and `run` take a repeatable `--surface NAME=KIND[:TARGET]`, and a +deployment config takes a `surfaces:` map of full backend configs: + +```bash +openadapt-flow replay bundle/ --url https://portal.example \ + --surface windows_1=windows:http://10.0.0.5:8000 \ + --surface report=https://portal.example/report +``` + +```yaml +surfaces: + windows_1: {kind: windows, agent_url: http://10.0.0.5:8000} + billing: {kind: macos, macos_app: Billing, macos_window_title: Invoices} +``` + +- `NAME` must be a backend surface the bundle declares; the primary and page + surfaces (which the runtime claims itself) are refused, as is an undeclared + name or a kind that differs from the bundle's. +- `KIND` defaults from the bundle, so `NAME=TARGET` is enough. `TARGET` fills + the kind's usual field (web `url`, windows `agent_url`, macos `macos_app`, + linux `linux_app`, rdp `rdp_host`, citrix `rdp_window`). Flags apply on top + of config entries. +- Every backend is built by the same factory as the primary + (`backends/factory.py`); a web surface opens as a new page in the primary's + browser. No per-workflow code is involved. +- A declared backend surface left unbound is named on stderr before the run, + and the run halts at its switch with a hint naming the flag. + +Design choice: one generic flag plus one config map, both feeding the +existing backend factory. Rejected: per-kind flag families (`--windows-agent` +per surface multiplies flags with every backend); guessing a default backend +for an unbound surface (a guessed backend is a wrong action, so the run halts +instead). + +**Native windows and JS dialogs.** These are screen states, not surfaces (see +"Vision first"). A window inside one desktop is reached with an explicit, +checkable focus step. A browser dialog is answered by `Step.dialog`. A +separate *actuator*, such as a second desktop agent or an RDP session, is a +backend surface bound here. + +## Backward compatibility + +- A bundle with no `surfaces` is single-surface. Every new field defaults + empty, `Workflow.surfaces` is omitted from serialized bytes when empty, and + the four new `Step` fields (`surface`, `opens`, `surface_origin`, `dialog`) + have reviewed, version-scoped sealed-digest + omissions (`bundle_validation.py`), so existing sealed v2 bundles still + verify byte-for-byte (`tests/test_bundle_schema_v2.py`). +- The bundle schema stays at v2. The change is additive for every existing + bundle, and a bundle that uses it cannot be misread by an older runtime + because SWITCH_SURFACE is required for any surface change and an older + runtime refuses that action value. A version bump would not add protection: + older loaders do not refuse a newer `schema_version`. + +## Limits + +- Native windows inside one desktop are screen states. The recorder does not + insert the focus step that reaches a hidden window; the author or the agent + records it as an ordinary click or hotkey. +- Dialogs: only the scripted recorder captures them (`expect_dialog`). The + interactive recorder does not yet record a dialog answer, so record that + step with the scripted recorder. Qualification does not yet treat a dialog + answer as its own evidence. A `prompt()` answer with text is not supported. +- A page that opens and closes itself before the step's check (a very fast + OAuth round trip) is seen as "did not open" and halts. In interactive + recording, a popup whose first action closes it is refused (above). +- Resume of a multi-surface program (graph) run, and of a linear checkpoint + without a surface observation, is refused (above). +- Re-binding a page is vision first, so two pages that look the same and share + every structured attribute halt as ambiguous. Two report tabs for the same + record are the example. This is by design. +- Attach mode follows only same-origin pages that a followed page opened. An + OAuth popup on another origin is refused by attach mode's origin contract. +- Qualification pinning and proposal do not yet propose per-surface + environment boundaries; `surface_environments` is written by the operator. +- A backend surface needs an explicit binding (`--surface` or `surfaces:`). + Nothing is guessed, and a managed run refuses to start without one. + +## Tests + +- `tests/test_multi_surface_e2e.py`: a local app that opens a popup + (`window.open`) and a `target=_blank` tab. It records, compiles, and replays + across both with zero model calls. It also halts on an undeclared popup, on a + declared popup that never opens, and on a popup at a different URL. Halt + messages quote no URL or title. +- `tests/test_multi_surface_backends.py`: web, then the MockWaa Windows + backend, in one bundle. It halts at the switch, before any Windows input, + when no backend is supplied, when the backend is the wrong kind, or when the + Windows screen is a different app. +- `tests/test_multi_surface_contract.py`: IR contract validation and the + replayer's refusals (a resume without a surface observation, a program + resume, a managed run with an unbound backend surface). +- `tests/test_multi_surface_identity.py`: a durable run pauses at "Check the + record" on the report tab. The checkpoint holds the surface name and + observation, with no handle. Resume needs approval, re-binds the tab by + identity, and finishes with zero model calls. A second identical tab makes + the re-bind ambiguous and it halts with nothing driven. A tab navigated + elsewhere fails page routing and halts. Router-level routing binds only the + page that passes. +- `tests/test_multi_surface_fail_closed.py`: a step on a closed page surface is + refused; a switch or rebind never takes a page this run did not open (the + person's tab in attach mode); a resume rebinds only on visual evidence; a + held page that fails its check halts; a page switch without identity + evidence halts; dialog errors carry a code, not the text; the sealed dialog + identity and URL pattern carry no record number or parameter value; the + layout-drift check follows the active surface. +- `tests/test_multi_surface_selfclose.py`: a popup that closes itself. Both the + scripted and the interactive recorder write an explicit switch back, and the + bundle replays with zero model calls. Without the switch back the workflow + is invalid. +- `tests/test_multi_surface_attach.py`: `record --cdp` against a separately + started Chromium follows a popup and a `noopener` tab and replays in the same + browser with zero model calls. A tab the person opens is refused. +- `tests/test_multi_surface_dialog.py`: a `confirm()` answer is compiled onto + the step that raises it and replays. A dialog with a different message is + dismissed and the run halts. A screen-driven backend refuses a dialog step. +- `tests/test_multi_surface_governed.py`: a governed, durable, Standard-profile + run from a web primary to a Windows backend surface reaches `VERIFIED` with + an independent-readback effect; a tampered surface record, or a persisted + re-read verifier, does not. Per-surface qualification boundaries: missing or + wrong-kind boundaries are refused, a switch is observed and moves the input + guard, a mismatch halts, and case evidence needs every surface's + observation. Single-surface reports and qualification projects serialize + byte-for-byte as before. +- `tests/test_surface_bindings.py`: `--surface` parsing, resolution against + the bundle and the config map, the factory, and a CLI `replay` that binds a + backend surface and finishes with zero model calls (and halts with a hint + when the binding is missing). +- `tests/test_multi_surface_interactive.py`: a scripted user drives a browser + that `record --url` launched through a popup and a tab; the recording + compiles to the same surfaces, `opens`, and switch steps as the scripted + `Recorder`'s, and replays with zero model calls. + +## Audit of origin/main (f21281fa) + +Where single-tab and single-backend were assumed or enforced: + +| Area | Location | What it assumed | +| --- | --- | --- | +| Interactive recorder | `openadapt_flow/interactive_recorder.py:2263`, `:2337`, `:2351` | installs `popup` and context `page` listeners only to refuse | +| | `interactive_recorder.py:2828-2847` | `_handle_popup` / `_handle_context_page`: "this recording is bound to one tab" | +| | `interactive_recorder.py:2913-2933` | `_assert_no_new_pages` refuses any page beyond the start baseline | +| Scripted recorder | `openadapt_flow/recorder.py:96` | one `self._backend` for the whole recording | +| | `recorder.py:209-224` | `meta.json` has one viewport and no surface list | +| | `recorder.py:495-511` | only a context-wide page count (`pages_before/after`) is captured | +| IR | `openadapt_flow/ir.py:84-94` | `ActionKind` has no way to change surface | +| | `ir.py:1030` (`Step`) | a step has no surface | +| | `ir.py:1997-2001` | `Workflow.surface` / `execution_mode`: one surface per bundle | +| | `ir.py:1812` (`BackendHints`) | one remote-display target window | +| | `ir.py:446-449` | `NEW_TAB_OPENED` only asserts that the page count grew | +| Compiler | `openadapt_flow/compiler/compile.py:1503-1550` | `_surface_binding_from_meta`: "a recording captures exactly one execution surface" | +| | `compile.py:1882-1897` | a viewport change inside an event is an error | +| | `compile.py:1085-1111` | a new tab only becomes a page-count postcondition | +| Playwright backend | `openadapt_flow/backends/playwright_backend.py:1036` | one backend wraps one `page` | +| | `playwright_backend.py:1146-1152` | `page_count` is the only view of other pages | +| | `playwright_backend.py:3142-3144` | `launch()` opens one page | +| Backend factory | `openadapt_flow/backends/factory.py:46`, `:83` | `build_backend` returns one backend for one `cfg.kind` | +| Replayer | `openadapt_flow/runtime/replayer.py:580`, `:649` | one `backend` for the whole run | +| | `replayer.py:1453` | the report records one `recorded_surface` | +| | `replayer.py:8402`, `:8431`, `:8507` | remote-surface safety rules read the bundle-level `workflow.surface` | +| | `replayer.py:7861-7872` | the on-screen read-back verifier is bound to the first backend and cached | +| | `replayer.py:13896-13909` | a new tab is only a count comparison | +| Identity gates | `replayer.py:12343` (`_verify_identity`) and `runtime/identity.py` | surface-agnostic: they read `self.backend`, so routing that attribute is enough | +| Effect gates | `replayer.py:7874` (`_verify_effects`) | surface-agnostic except the cached read-back verifier above | +| Qualification | `openadapt_flow/compiler/qualification_pins.py:199` | application identity is proposed from one `workflow.surface` | +| | `openadapt_flow/qualification.py:1201` | read-only action classes are WAIT and SCROLL only | +| CLI | `openadapt_flow/__main__.py:2118-2120`, `:3101`, `:3440` | replay launches one page | +| Docs | `README.md:267`, `docs/PRODUCTION.md:88-96`, `docs/LIMITS.md:406` | "one bundle drives one surface"; compose separate bundles instead | + +The identity ladder and the effect gates needed no change of their own. They +already act on `self.backend`, so routing the active surface through that +attribute, and resetting the per-backend caches on a switch, puts them on +every surface. diff --git a/packages/flow/docs/PRODUCTION.md b/packages/flow/docs/PRODUCTION.md index b024aac33..dc60893fa 100644 --- a/packages/flow/docs/PRODUCTION.md +++ b/packages/flow/docs/PRODUCTION.md @@ -88,10 +88,19 @@ windows belong to one synthetic fixture process, and the repository does not contain a completed campaign result. A real deployment needs its own qualification. -The boundary is one backend per bundle. Worklists repeat that bundle over input +A bundle that declares several surfaces ([MULTI_SURFACE.md](MULTI_SURFACE.md)) +runs governed, qualified, and durable under the same rules as one surface: +the surface map is sealed in the authorized bundle digest, each step result +records its surface, and qualification declares one environment boundary per +surface. Resume re-binds each surface by identity (never by a stored +handle) and still stops at "Check the record" first when a save may have gone +through. Managed Cloud dispatch needs only the deployment config's `surfaces:` +map, and it refuses to start while a backend surface is unbound. A program +(graph) resume of such a bundle is still refused. +Worklists repeat that bundle over input records, and subflows reuse steps inside it. Neither one switches backends. -Browser recording owns one tab and refuses popups or new tabs. Browser attach -mode stays on one origin. macOS and Linux bind one exact app and window, while a +Browser recording follows popups and new tabs in a browser Flow launched and, +in attach mode, only pages a followed page opened on the declared origin. macOS and Linux bind one exact app and window, while a governed Windows run binds its application identity. If a task crosses a browser and a native app, or otherwise changes backend, diff --git a/packages/flow/openadapt_flow/__main__.py b/packages/flow/openadapt_flow/__main__.py index 77040e094..6e6e37a79 100644 --- a/packages/flow/openadapt_flow/__main__.py +++ b/packages/flow/openadapt_flow/__main__.py @@ -718,6 +718,7 @@ def _configured_replayer( checkpoint_key: Optional[str] = None, approval_gate=None, qualification_fault_driver=None, + surface_backends=None, deployment_surface: Optional[str] = None, ): """Wire the grounding, verification, and actuation layers into a Replayer. @@ -741,7 +742,7 @@ def _configured_replayer( ): checkpoint_key = crypto.require_key(None) - return build_replayer( + replayer = build_replayer( backend, allow_egress=allow_egress, effect_verifier=effect_verifier, @@ -761,6 +762,11 @@ def _configured_replayer( qualification_fault_driver=qualification_fault_driver, deployment_surface=deployment_surface, ) + if surface_backends: + # Backend surfaces of a multi-surface bundle (docs/MULTI_SURFACE.md), + # bound by --surface / the config's surfaces: map. + replayer.surface_backends = dict(surface_backends) + return replayer def _build_and_run_replayer( @@ -794,6 +800,7 @@ def _build_and_run_replayer( run_id: Optional[str] = None, approval_gate=None, qualification_fault_driver=None, + surface_backends=None, deployment_surface: Optional[str] = None, ): """Build the shared Replayer configuration and execute one workflow.""" @@ -815,6 +822,7 @@ def _build_and_run_replayer( runtime_config=runtime_config, approval_gate=approval_gate, qualification_fault_driver=qualification_fault_driver, + surface_backends=surface_backends, deployment_surface=deployment_surface, ).run( workflow, @@ -979,6 +987,7 @@ def _replay_desktop( run_id: Optional[str] = None, approval_gate=None, qualification_fault_driver=None, + surface_cfgs=None, ) -> int: """Replay against a non-browser native/remote backend built by the factory. @@ -998,6 +1007,7 @@ def _replay_desktop( backend = build_backend(backend_cfg) except ValueError as e: raise SystemExit(str(e)) + surface_backends = _build_surface_backends(surface_cfgs or {}, page=None) try: report = _build_and_run_replayer( @@ -1028,12 +1038,14 @@ def _replay_desktop( run_id=run_id, approval_gate=approval_gate, qualification_fault_driver=qualification_fault_driver, + surface_backends=surface_backends, deployment_surface=backend_cfg.deployment_surface, ) finally: - close = getattr(backend, "close", None) - if callable(close): - close() # RDP transports hold a live socket; browsers/agents don't + for owned in (backend, *surface_backends.values()): + close = getattr(owned, "close", None) + if callable(close): + close() # RDP transports hold a live socket; browsers/agents don't return _finish_replay( run_dir, report, args, backend_kind=_report_backend_kind(backend_cfg.kind) ) @@ -2016,6 +2028,81 @@ def _default_run_dir() -> Path: return Path("runs") / f"replay-{stamp}" +def _resolve_surface_bindings(args: argparse.Namespace, cfg, workflow) -> dict: + """Resolve ``--surface`` flags and ``surfaces:`` config for this bundle. + + Refuses (before anything runs) a binding the bundle cannot use. Names + every backend surface left unbound; the run halts at the switch to it. + """ + + from openadapt_flow.surface_bindings import ( + SurfaceBindingError, + resolve_surface_configs, + unbound_backend_surfaces, + ) + + flags = list(getattr(args, "surface", None) or []) + configured = dict(getattr(cfg, "surfaces", {}) or {}) + if not getattr(workflow, "surfaces", None): + if flags or configured: + raise SystemExit( + "REFUSED: --surface / surfaces: were given, but this bundle " + "declares no surfaces. Nothing was executed." + ) + return {} + try: + resolved = resolve_surface_configs(workflow, flags, configured) + except SurfaceBindingError as exc: + raise SystemExit(f"REFUSED: {exc}. Nothing was executed.") + unbound = unbound_backend_surfaces(workflow, resolved) + if unbound: + print( + "NOTE: no backend is bound for surface(s) " + + ", ".join(unbound) + + "; the run halts at the switch to it. Bind one with " + "--surface NAME=KIND[:TARGET] or the config's surfaces: map.", + file=sys.stderr, + ) + return resolved + + +def _build_surface_backends(surface_cfgs, *, page) -> dict: + """Build bound backend surfaces; a web one opens in the run's browser.""" + + from openadapt_flow.surface_bindings import ( + SurfaceBindingError, + build_surface_backends, + ) + + new_page = None + if page is not None: + + def new_page(url: str): + # A second app is its own browser page in the run's browser: a new + # page in the run's context when the context allows one, else a + # page of its own context (Playwright's browser.new_page()). + try: + opened = page.context.new_page() + except Exception: + browser = page.context.browser + if browser is None: + raise + opened = browser.new_page(viewport=page.viewport_size) + opened.goto(url) + try: + opened.wait_for_load_state("load") + except Exception: + pass + # The run starts on the primary page. + page.bring_to_front() + return opened + + try: + return build_surface_backends(surface_cfgs or {}, new_page=new_page) + except (SurfaceBindingError, ValueError) as exc: + raise SystemExit(f"REFUSED: {exc}. Nothing was executed.") + + def _cmd_replay(args: argparse.Namespace) -> int: from openadapt_flow.backends.factory import _normalize_kind, build_backend from openadapt_flow.cli_compose import refuse_replay_composition @@ -2074,6 +2161,7 @@ def _cmd_replay(args: argparse.Namespace) -> int: refused = _enforce_surface_binding(args, workflow, backend_cfg, operation=operation) if refused is not None: return refused + surface_cfgs = _resolve_surface_bindings(args, cfg, workflow) # Opt-in attest pre-state snapshot: the earliest point where the run is # ADMITTED (every gate above passed) and the run directory is final, but # NO actuation has begun on either the desktop or the web path below. @@ -2114,6 +2202,7 @@ def _cmd_replay(args: argparse.Namespace) -> int: run_id=getattr(args, "_qualification_run_id", None), approval_gate=approval_gate, qualification_fault_driver=fault_driver, + surface_cfgs=surface_cfgs, ) headed = args.headed or cfg.backend.headed @@ -2136,6 +2225,7 @@ def run_on_page(page, *, entry_url: Optional[str]): # is exercised instead of the structural rung resolving it. backend = build_backend(backend_cfg, page=page) held_backend.append(backend) + surface_backends = _build_surface_backends(surface_cfgs, page=page) return _build_and_run_replayer( backend, workflow=workflow, @@ -2172,6 +2262,7 @@ def run_on_page(page, *, entry_url: Optional[str]): run_id=getattr(args, "_qualification_run_id", None), approval_gate=approval_gate, qualification_fault_driver=fault_driver, + surface_backends=surface_backends, deployment_surface=backend_cfg.deployment_surface, ) @@ -6307,6 +6398,26 @@ def _add_browser_session_flags(p: argparse.ArgumentParser) -> None: ) +def _add_surface_flags(p: argparse.ArgumentParser) -> None: + """Add ``--surface NAME=KIND[:TARGET]`` (multi-surface bundles).""" + + p.add_argument( + "--surface", + action="append", + default=None, + metavar="NAME=KIND[:TARGET]", + help=( + "Multi-surface bundles: bind a backend to the backend surface NAME " + "(repeatable). KIND defaults to the kind the bundle declares; " + "TARGET fills that kind's target (web: URL, windows: agent URL, " + "macos: app, linux: app, rdp: host, citrix: client window). " + "NAME=TARGET alone is accepted. Other fields go in the config's " + "surfaces: map. Browser tabs and popups need no binding. See " + "docs/MULTI_SURFACE.md." + ), + ) + + def _add_backend_flags(p: argparse.ArgumentParser) -> None: """Add the backend-selector flags (``--backend`` + targets) to a subparser. @@ -7432,6 +7543,7 @@ def build_parser() -> argparse.ArgumentParser: ), ) _add_backend_flags(p) + _add_surface_flags(p) _add_browser_session_flags(p) _add_deployment_flags(p, worklist=True) _add_attest_flags(p) @@ -7514,6 +7626,7 @@ def build_parser() -> argparse.ArgumentParser: ), ) _add_backend_flags(p) + _add_surface_flags(p) _add_browser_session_flags(p) _add_deployment_flags(p, worklist=True) _add_attest_flags(p) diff --git a/packages/flow/openadapt_flow/action_evidence.py b/packages/flow/openadapt_flow/action_evidence.py index 979876879..5a7239b9f 100644 --- a/packages/flow/openadapt_flow/action_evidence.py +++ b/packages/flow/openadapt_flow/action_evidence.py @@ -406,12 +406,25 @@ def action_evidence_error( receipt_error = _delivery_receipt_error(step, result, params=scoped_params) if receipt_error is not None: return receipt_error - elif step.action is ActionKind.WAIT and actuation is None: + elif ( + step.action in (ActionKind.WAIT, ActionKind.SWITCH_SURFACE) + and actuation is None + ): if result.delivery_attempted not in {None, False}: - return "wait action claims an input-delivery boundary" + return f"{step.action.value} action claims an input-delivery boundary" elif strict_production: return "production action lacks a closed runtime actuation path" + if step.action is ActionKind.SWITCH_SURFACE: + # A switch is armed by its SURFACE identity check (the recorded screen + # text and origin of its target, docs/MULTI_SURFACE.md), not by an + # entity identity row: it acts on no record. Its proof is the passed + # postconditions on the new surface. + if result.identity is not None: + return "surface switch contains an entity identity verdict" + if identity_required and step.expect and result.postconditions_ok is not True: + return "surface switch lacks a passed surface identity check" + return None if identity_required: return nonvacuous_identity_error(result.identity) # A step without an armed identity contract may continue when the optional diff --git a/packages/flow/openadapt_flow/backends/playwright_backend.py b/packages/flow/openadapt_flow/backends/playwright_backend.py index 9e4002030..774131039 100644 --- a/packages/flow/openadapt_flow/backends/playwright_backend.py +++ b/packages/flow/openadapt_flow/backends/playwright_backend.py @@ -1008,6 +1008,8 @@ def __init__( screenshot_mask_selectors: tuple[str, ...] = (), structural_state_reader: Optional[Callable[[], dict[str, Any]]] = None, screenshot_guard: Optional[Callable[[], None]] = None, + closed_page_frame: bool = False, + bring_to_front_before_capture: bool = False, ) -> None: """Wrap an existing Playwright page. @@ -1031,8 +1033,34 @@ def __init__( screenshot_guard: Optional fail-closed check that runs before Chromium creates screenshot bytes. Recording paths use it to bind or refuse closed-shadow secret boundaries. + closed_page_frame: Multi-surface PAGE surfaces only (a popup or + tab, docs/MULTI_SURFACE.md). Once the page has closed itself, + ``screenshot`` returns an all-black frame of the last size + instead of raising: a closed popup shows nothing. It never + shows another window, every screen postcondition fails on it, + and acting elsewhere still needs an explicit surface switch. + The primary page keeps the default (a closed page raises). + bring_to_front_before_capture: Make ``page`` the browser's + foreground tab before each screenshot. Chromium draws no new + frames for a background tab, so a capture of one can wait + forever (Playwright's own launch disables that throttling with + command-line flags; a person's browser reached over CDP has no + such flags). Recording in an attached browser sets this: a + popup or tab the recorded page opens takes the foreground, and + the opener's after-frame must still be a live, current frame. """ self.page = page + self._closed_page_frame = closed_page_frame + self._bring_to_front_before_capture = bring_to_front_before_capture + self._last_frame_size: Optional[tuple[int, int]] = None + # JavaScript dialogs (docs/MULTI_SURFACE.md, "Dialogs"): no listener + # is registered until a step arms one, so a bundle without dialog + # steps keeps Playwright's default (dismiss) exactly as before. + self._dialog_listening = False + self._dialog_armed: Optional[ + tuple[str, Optional[str], Optional[re.Pattern[str]]] + ] = None + self._dialogs_seen: list[dict[str, Any]] = [] self._screenshot_scale = screenshot_scale self._screenshot_mask_selectors = screenshot_mask_selectors self._structural_state_reader = structural_state_reader @@ -1147,10 +1175,38 @@ def page_count(self) -> Optional[int]: """Open pages in the browser context (new tabs are visible here even though the single-page screenshot never shows them).""" try: - return len(self.page.context.pages) + return sum(1 for page in self.page.context.pages if not page.is_closed()) except Exception: return None + def wait_for_page_count_above(self, count: int, timeout_s: float) -> bool: + """Wait on the context's ``page`` event until more than ``count`` + pages are open, or ``timeout_s`` passes. True when they are. + + Event-driven on purpose: with the sync API a sleep dispatches + nothing, so a popup a click opened can stay out of + ``context.pages`` until the next Playwright call. + """ + + import time as _time + + deadline = _time.monotonic() + max(0.0, float(timeout_s)) + while True: + current = self.page_count + if current is not None and current > count: + return True + remaining = deadline - _time.monotonic() + if remaining <= 0: + return False + try: + self.page.context.wait_for_event( + "page", timeout=max(1.0, remaining * 1000.0) + ) + except Exception: + # Timed out (or the context closed): one last look above. + current = self.page_count + return current is not None and current > count + # -- live execution-context identity ------------------------------- def _bind_context_identity(self, kind: str, value: str) -> bool: @@ -2885,6 +2941,137 @@ def _capture_across_navigation(self, options: dict[str, Any]) -> bytes: def screenshot(self) -> bytes: """Return a stable current full-viewport frame as PNG bytes.""" + if self._closed_page_frame and self._page_is_closed(): + return self._closed_frame() + try: + if self._bring_to_front_before_capture: + # A no-op for the tab already in front. + self.page.bring_to_front() + png = self._live_screenshot() + except Exception: + # The page closed itself while the frame was being captured. + if self._closed_page_frame and self._page_is_closed(): + return self._closed_frame() + raise + if self._closed_page_frame: + try: + from openadapt_flow.runtime.resolver import png_size + + self._last_frame_size = png_size(png) + except Exception: + pass + return png + + # -- JavaScript dialogs ------------------------------------------------- + + def arm_dialog( + self, + response: str, + message: Optional[str], + message_pattern: Optional[str] = None, + ) -> None: + """Answer the next dialog with ``response`` if it shows ``message``. + + ``message_pattern`` (a full-match regex) replaces ``message`` when + given. With neither (recording) any dialog is answered and its text + reported. A dialog whose text does not match is DISMISSED, never + accepted, and reported as a mismatch. Every dialog this page shows + while the listener exists is answered and reported, so an unexpected + one can never be accepted by default. + """ + + if response not in ("accept", "dismiss"): + raise ValueError("a dialog response is 'accept' or 'dismiss'") + if not self._dialog_listening: + self.page.on("dialog", self._on_dialog) + self._dialog_listening = True + pattern = ( + re.compile(message_pattern, re.DOTALL) + if message_pattern is not None + else None + ) + self._dialog_armed = (response, message, pattern) + self._dialogs_seen = [] + + def take_dialogs(self, timeout_s: float = 0.0) -> list[dict[str, Any]]: + """The dialogs answered since the last arm, then disarm. + + Waits up to ``timeout_s`` for the first one (a dialog can open a + moment after the input edge). Playwright delivers the dialog event + only while its loop runs, so the wait runs on the page. + """ + + deadline = time.monotonic() + max(0.0, float(timeout_s)) + while not self._dialogs_seen and time.monotonic() < deadline: + if self._page_is_closed(): + break + try: + self.page.wait_for_timeout(50) + except Exception: + break + seen, self._dialogs_seen = self._dialogs_seen, [] + self._dialog_armed = None + return seen + + def _on_dialog(self, dialog: Any) -> None: + try: + message = str(dialog.message) + except Exception: + message = "" + try: + kind = str(dialog.type) + except Exception: + kind = "" + armed, self._dialog_armed = self._dialog_armed, None + matched = False + if armed is not None: + _response, expected, pattern = armed + if pattern is not None: + matched = pattern.fullmatch(message) is not None + else: + matched = expected is None or expected == message + response = armed[0] if armed is not None and matched else "dismiss" + self._dialogs_seen.append( + { + "message": message, + "type": kind, + "response": response, + "matched": matched, + } + ) + try: + if response == "accept": + dialog.accept() + else: + dialog.dismiss() + except Exception: + pass + + def _page_is_closed(self) -> bool: + try: + return bool(self.page.is_closed()) + except Exception: + return True + + def _closed_frame(self) -> bytes: + """An all-black frame: what a closed popup or tab shows.""" + + from io import BytesIO + + from PIL import Image + + size = self._last_frame_size + if size is None: + try: + viewport = self.page.viewport_size or {} + size = (int(viewport["width"]), int(viewport["height"])) + except Exception: + size = (1280, 800) + buffer = BytesIO() + Image.new("RGB", size, (0, 0, 0)).save(buffer, format="PNG") + return buffer.getvalue() + + def _live_screenshot(self) -> bytes: base_options: dict[str, Any] = {} if self._screenshot_scale == "css": base_options["scale"] = "css" diff --git a/packages/flow/openadapt_flow/bundle_validation.py b/packages/flow/openadapt_flow/bundle_validation.py index 096637a35..e036e1037 100644 --- a/packages/flow/openadapt_flow/bundle_validation.py +++ b/packages/flow/openadapt_flow/bundle_validation.py @@ -202,6 +202,43 @@ class _SealedCanonicalOmission: "refusal reason remains sealed content" ), ), + _SealedCanonicalOmission( + owner=Step, + field_name="surface", + omitted_values=(None,), + reason=( + "a step with no named surface runs on the primary surface, " + "which is what every pre-multi-surface bundle did; a named " + "surface remains sealed content" + ), + ), + _SealedCanonicalOmission( + owner=Step, + field_name="opens", + omitted_values=([],), + reason=( + "an empty opened-page list predates the multi-surface fields; " + "a step that opens pages is sealed content" + ), + ), + _SealedCanonicalOmission( + owner=Step, + field_name="surface_origin", + omitted_values=(None,), + reason=( + "absent switch-target origin predates the multi-surface " + "fields; a recorded origin remains sealed content" + ), + ), + _SealedCanonicalOmission( + owner=Step, + field_name="dialog", + omitted_values=(None,), + reason=( + "a step that raises no browser dialog is what every earlier " + "bundle held; a recorded dialog answer remains sealed content" + ), + ), _SealedCanonicalOmission( owner=Effect, field_name="new_records", diff --git a/packages/flow/openadapt_flow/compiler/codegen.py b/packages/flow/openadapt_flow/compiler/codegen.py index 5e64dc429..3dfd218ca 100644 --- a/packages/flow/openadapt_flow/compiler/codegen.py +++ b/packages/flow/openadapt_flow/compiler/codegen.py @@ -108,6 +108,8 @@ def _step_call(step: Step) -> str: return f" flow.wait(timeout_s={step.timeout_s!r})" if step.action is ActionKind.SCROLL: return f" flow.scroll({step.scroll_dx or 0!r}, {step.scroll_dy or 0!r})" + if step.action is ActionKind.SWITCH_SURFACE: + return f" flow.switch_surface({step.surface!r})" raise ValueError(f"unknown action {step.action!r}") diff --git a/packages/flow/openadapt_flow/compiler/compile.py b/packages/flow/openadapt_flow/compiler/compile.py index 916085ffd..a7de255e5 100644 --- a/packages/flow/openadapt_flow/compiler/compile.py +++ b/packages/flow/openadapt_flow/compiler/compile.py @@ -31,6 +31,7 @@ BackendHints, BundleManifest, BundleProvenance, + DialogResponse, ExecutionMode, ExecutionTargetKind, Landmark, @@ -42,6 +43,7 @@ Region, Step, StructuralLocator, + SurfaceSpec, Workflow, structural_locator_is_locatable, ) @@ -1543,6 +1545,173 @@ def _surface_binding_from_meta( return surface, execution_mode_for_surface(surface) +# -- multi-surface recordings (docs/MULTI_SURFACE.md) -------------------------- + +#: Parameter values shorter than this are not generalized out of a URL/title +#: (a two-letter value would match half the alphabet). +_SURFACE_PARAM_MIN_CHARS = 3 + + +def _surface_url_pattern(url: str, exclude_texts: tuple[str, ...]) -> str: + """Full-match regex for a recorded page URL. + + Scheme, host, and port are exact. Each path segment is exact unless it + carries a demonstrated parameter value or looks like an identifier (a + digit, as in ``/patient/12345``, or a long opaque token); such a segment + becomes one path segment of any value, so the bundle never seals a + record identifier. The query and fragment are not asserted: they carry + tokens (OAuth ``state``) and record identifiers that change on every run. + """ + + from urllib.parse import unquote, urlsplit + + parts = urlsplit(url) + values = [v for v in exclude_texts if len(v.strip()) >= _SURFACE_PARAM_MIN_CHARS] + segments = [] + for segment in (parts.path or "/").split("/"): + decoded = unquote(segment) + if _segment_is_identifier(decoded) or any( + value in segment or value in decoded for value in values + ): + segments.append("[^/?#]+") + else: + segments.append(re.escape(segment)) + origin = re.escape(f"{parts.scheme}://{parts.netloc}".lower()) + return origin + "/".join(segments) + r"(?:[?#].*)?" + + +#: An API version marker (``v2``) is structure, not an identifier. +_VERSION_SEGMENT_RE = re.compile(r"[vV]\d{1,2}") +#: A long opaque token (a session id or hash) even without a digit. A slug +#: with separators (``order-history``) stays exact. +_OPAQUE_SEGMENT_RE = re.compile(r"[A-Za-z0-9]{24,}") + + +def _segment_is_identifier(segment: str) -> bool: + if not segment or _VERSION_SEGMENT_RE.fullmatch(segment): + return False + return bool(re.search(r"\d", segment) or _OPAQUE_SEGMENT_RE.fullmatch(segment)) + + +def _surface_title_pattern( + title: object, exclude_texts: tuple[str, ...] +) -> Optional[str]: + """Full-match regex for a recorded page title, or None to not assert it. + + A demonstrated parameter value becomes "any text". A title the privacy + pass flags as carrying an identifier is not asserted at all. + """ + + if not isinstance(title, str) or not title.strip(): + return None + if _text_carries_phi(title): + return None + values = sorted( + (v for v in exclude_texts if len(v.strip()) >= _SURFACE_PARAM_MIN_CHARS), + key=len, + reverse=True, + ) + if not values: + return re.escape(title) + splitter = re.compile("|".join(re.escape(v) for v in values)) + pieces = splitter.split(title) + return ".+".join(re.escape(piece) for piece in pieces) + + +def _dialog_identity(raw: dict, exclude_texts: tuple[str, ...]) -> DialogResponse: + """The sealed identity of a recorded browser dialog. + + A message with no demonstrated parameter value and no digits is kept + exactly. Otherwise it becomes a full-match pattern: each parameter value + matches any text and each digit run matches any digits, so a record + number, a date, or a typed value is never sealed into the bundle. A + message the privacy pass still flags after that is refused: the operator + must make the value a parameter. + """ + + response = raw.get("response") + message = str(raw.get("message") or "") + values = sorted( + (v for v in exclude_texts if len(v.strip()) >= _SURFACE_PARAM_MIN_CHARS), + key=len, + reverse=True, + ) + pieces = ( + re.compile("|".join(re.escape(v) for v in values)).split(message) + if values + else [message] + ) + literal = " ".join(re.sub(r"\d+", "0", piece) for piece in pieces) + if _text_carries_phi(literal): + raise ValueError( + "a recorded browser dialog message carries an identifier that is " + "not a demonstrated parameter; make that value a parameter and " + "compile again" + ) + if len(pieces) == 1 and not re.search(r"\d", message): + return DialogResponse.model_validate({"response": response, "message": message}) + pattern = ".+".join( + r"\d+".join(re.escape(part) for part in re.split(r"\d+", piece)) + for piece in pieces + ) + return DialogResponse.model_validate( + {"response": response, "message_pattern": pattern} + ) + + +def _surfaces_from_meta( + meta: dict, + exclude_texts: tuple[str, ...], + step_id_for_event: dict[int, str], +) -> list[SurfaceSpec]: + """``meta.json`` ``surfaces`` -> ``Workflow.surfaces`` (patterns derived).""" + + from openadapt_flow.runtime.surfaces import url_origin + + raw = meta.get("surfaces") + if raw is None: + return [] + if not isinstance(raw, list) or not raw: + raise ValueError("meta.json surfaces must be a non-empty list") + specs: list[SurfaceSpec] = [] + for entry in raw: + if not isinstance(entry, dict): + raise ValueError("meta.json surfaces entries must be objects") + sid = entry.get("id") + kind = entry.get("kind") + if "opened_by_event" not in entry: + specs.append(SurfaceSpec.model_validate({"id": sid, "kind": kind})) + continue + event_index = int(entry["opened_by_event"]) + if event_index not in step_id_for_event: + raise ValueError(f"surface {sid!r} is opened by an unknown event") + url = str(entry.get("url") or "") + origin = url_origin(url) + if origin is None: + raise ValueError( + f"surface {sid!r} was recorded without an http(s) URL; Flow " + "cannot identify that page at replay" + ) + specs.append( + SurfaceSpec.model_validate( + { + "id": sid, + "kind": kind, + "opened_by_step": step_id_for_event[event_index], + "opener": entry.get("opener"), + "open_order": int(entry.get("open_order", 0)), + "opener_link": entry.get("opener_link"), + "origin": origin, + "url_pattern": _surface_url_pattern(url, exclude_texts), + "title_pattern": _surface_title_pattern( + entry.get("title"), exclude_texts + ), + } + ) + ) + return specs + + # Parameter names must be usable as ValueExpr(param=...) / ApiBinding threads # and environment-variable stems -- the same shape ir.ApiBinding enforces. _PARAM_NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,127}$") @@ -1905,7 +2074,8 @@ def cached_lines(i: int, suffix: str, png: bytes) -> list[OcrLine]: event_index=i, ) if ( - before_viewport is not None + kind != "switch_surface" + and before_viewport is not None and after_viewport is not None and before_viewport != after_viewport ): @@ -2319,8 +2489,34 @@ def cached_lines(i: int, suffix: str, png: bytes) -> list[OcrLine]: event, ) ) + elif kind == "switch_surface": + target = str(event.get("surface") or "") + from openadapt_flow.runtime.surfaces import url_origin + + pending.append( + ( + Step( + id=step_id, + intent=f"switch to surface '{target}'", + action=ActionKind.SWITCH_SURFACE, + surface=target, + # Identity of a browser target: the origin the + # demonstration saw on arriving there. + surface_origin=url_origin(event.get("url_after")), + ), + before_png, + after_png, + event, + ) + ) else: raise ValueError(f"unknown event kind {kind!r} (event {i})") + if kind != "switch_surface": + stamped = pending[-1][0] + if event.get("surface") is not None: + stamped.surface = str(event["surface"]) + if event.get("opens"): + stamped.opens = [str(sid) for sid in event["opens"]] # Collapse a demonstrated native/pixel option selection into one explicit # SELECT_OPTION contract. This is deliberately evidence-gated and remote-only: @@ -2480,10 +2676,44 @@ def cached_lines(i: int, suffix: str, png: bytes) -> list[OcrLine]: binding_observations: dict[str, object] = {} for j, (step, step_before, step_after, event) in enumerate(pending): steps.append(step) + if isinstance(event.get("dialog"), dict): + # The browser dialog this action raised, answered by the backend + # at replay only when it shows the same message (values and + # digits generalized, never sealed). + step.dialog = _dialog_identity(event["dialog"], exclude_texts) if event.get("submit_binding") is not None: binding_observations[step.id] = event["submit_binding"] if step.action is ActionKind.SCROLL: continue # see the scroll branch in pass 1 + if step.action is ActionKind.SWITCH_SURFACE: + # The switch's screen identity: one stable text line from the + # FIRST frame of the new surface (every line there is "new"), with + # the same volatility, persistence, parameter, and PHI filters as + # any other step. Its before frame is another surface, so a diff + # would mean nothing. + if step_after is not None: + switch_next: Optional[list[OcrLine]] = None + switch_next_before = pending[j + 1][1] if j + 1 < len(pending) else None + if switch_next_before is not None: + switch_next = cached_lines( + int(pending[j + 1][3]["i"]), "before", switch_next_before + ) + text_pc = _new_text_postcondition( + [], + cached_lines(int(event["i"]), "after", step_after), + exclude_texts=exclude_texts, + avoid_labels=anchor_labels, + next_lines=switch_next, + reference_date=reference_date, + ) + step.expect = [text_pc] if text_pc is not None else [] + step.identity_armed = bool(step.expect) or step.surface_origin is not None + step.identity_unarmed_reason = ( + None + if step.identity_armed + else "no stable text on the new surface and no browser origin" + ) + continue step_region_drops: list[tuple[Region, str]] = [] next_before: Optional[bytes] = None next_lines: Optional[list[OcrLine]] = None @@ -2538,6 +2768,11 @@ def cached_lines(i: int, suffix: str, png: bytes) -> list[OcrLine]: # passes risk_overrides. Conservative: an unsure write-shaped step leans # irreversible (the safe direction). Explicit risk_overrides below win. for step in steps: + if step.action is ActionKind.SWITCH_SURFACE: + step.risk = "reversible" + step.risk_explanation = "a surface switch delivers no input" + step.risk_review_required = False + continue inference = infer_step_risk(step) step.risk = inference.risk step.risk_explanation = inference.explanation @@ -2705,6 +2940,12 @@ def cached_lines(i: int, suffix: str, png: bytes) -> list[OcrLine]: for note in compile_notes: logger.info("compile-pass %s: %s", note["step"], note["change"]) + for step in steps: + if step.action is ActionKind.SWITCH_SURFACE and step.risk != "reversible": + raise ValueError( + f"step {step.id!r} is a surface switch; it delivers no input " + "and cannot be marked irreversible" + ) workflow = Workflow( name=name, recording_id=meta.get("id"), @@ -2725,6 +2966,16 @@ def cached_lines(i: int, suffix: str, png: bytes) -> list[OcrLine]: }, secret_params=secret_params, steps=steps, + surfaces=_surfaces_from_meta( + meta, + exclude_texts, + { + int(event["i"]): absorbed_step_aliases.get( + f"step_{int(event['i']):03d}", f"step_{int(event['i']):03d}" + ) + for event in events + }, + ), ) source_recording_sha256: Optional[str] = None diff --git a/packages/flow/openadapt_flow/deployment.py b/packages/flow/openadapt_flow/deployment.py index fb63e760a..1655e64ff 100644 --- a/packages/flow/openadapt_flow/deployment.py +++ b/packages/flow/openadapt_flow/deployment.py @@ -27,7 +27,13 @@ from pathlib import Path from typing import Any, Callable, Literal, Mapping, Optional -from pydantic import BaseModel, Field, field_validator, model_validator +from pydantic import ( + BaseModel, + Field, + field_validator, + model_serializer, + model_validator, +) from openadapt_flow.lookup_capabilities import ( DeclaredSurface, @@ -793,6 +799,21 @@ class DeploymentConfig(BaseModel): runtime: RuntimeSection = Field(default_factory=RuntimeSection) policy: PolicySection = Field(default_factory=PolicySection) human_decisions: HumanDecisionsConfig = Field(default_factory=HumanDecisionsConfig) + #: Multi-surface bundles only (docs/MULTI_SURFACE.md): the backend for each + #: BACKEND surface the bundle switches to, keyed by surface id, with the + #: same fields as ``backend``. ``kind`` defaults to the kind the bundle + #: declares. ``--surface NAME=...`` flags apply over these entries. Empty + #: (and omitted when serialized) otherwise. + surfaces: dict[str, BackendConfig] = Field(default_factory=dict) + + @model_serializer(mode="wrap") + def _serialize_compatible(self, handler: Any) -> dict[str, Any]: + """Keep configs without surface bindings byte-identical.""" + + data: dict[str, Any] = handler(self) + if not self.surfaces: + data.pop("surfaces", None) + return data def incomplete_deployment_reasons(cfg: DeploymentConfig) -> list[str]: diff --git a/packages/flow/openadapt_flow/emit/skill.py b/packages/flow/openadapt_flow/emit/skill.py index 3d13e7b26..51edacbe5 100644 --- a/packages/flow/openadapt_flow/emit/skill.py +++ b/packages/flow/openadapt_flow/emit/skill.py @@ -117,6 +117,8 @@ def _step_line(step: Step, literals: list[str]) -> str: return "press " + "+".join([*step.modifiers, step.key or ""]) if action is ActionKind.SCROLL: return f"scroll by ({step.scroll_dx or 0}, {step.scroll_dy or 0})" + if action is ActionKind.SWITCH_SURFACE: + return f"switch to surface '{step.surface}'" return action.value diff --git a/packages/flow/openadapt_flow/execution_profiles.py b/packages/flow/openadapt_flow/execution_profiles.py index 2dbc5bd73..5ea275cd3 100644 --- a/packages/flow/openadapt_flow/execution_profiles.py +++ b/packages/flow/openadapt_flow/execution_profiles.py @@ -1664,6 +1664,17 @@ def classify_execution_outcome( if fault_prefix_review: paired_results = paired_results[:-1] identity_results = action_results[:-1] + if workflow.surfaces: + # Multi-surface (docs/MULTI_SURFACE.md): every executed step must + # carry the surface it ran on, and it must be the surface the sealed + # contract names (for a switch, its target). A result without that + # record, or on another surface, is not evidence for this contract. + primary_surface = workflow.surfaces[0].id + for result, step in paired_results: + if result.surface != (step.surface or primary_surface): + return ExecutionOutcome.COMPLETED_UNVERIFIED + elif any(result.surface is not None for result in report.results): + return ExecutionOutcome.COMPLETED_UNVERIFIED required_identity_ids = set(report.required_identity_step_ids) allowed_required_identity_sets = [expected_required_identity_ids] if fault_prefix_review: @@ -1832,7 +1843,10 @@ def _reported_parameter_predicate_value( is not None ): return ExecutionOutcome.COMPLETED_UNVERIFIED - if step.action is ActionKind.WAIT: + # WAIT and SWITCH_SURFACE deliver no input: no actuation path, no + # delivery boundary. A switch is verified by its own identity check + # and postconditions (docs/MULTI_SURFACE.md). + if step.action in (ActionKind.WAIT, ActionKind.SWITCH_SURFACE): if result.actuation is not None: return ExecutionOutcome.COMPLETED_UNVERIFIED elif result.actuation is None: @@ -1855,7 +1869,8 @@ def _reported_parameter_predicate_value( effects = effects_for_actuation(step, result.actuation) automated_gui = result.actuation in AUTOMATED_GUI_ACTUATIONS or ( - result.actuation is None and step.action is not ActionKind.WAIT + result.actuation is None + and step.action not in (ActionKind.WAIT, ActionKind.SWITCH_SURFACE) ) if api_actuation: # The API tier bypasses the GUI resolve/settle/act/postcondition diff --git a/packages/flow/openadapt_flow/interactive_recorder.py b/packages/flow/openadapt_flow/interactive_recorder.py index a8074c63e..333dc6524 100644 --- a/packages/flow/openadapt_flow/interactive_recorder.py +++ b/packages/flow/openadapt_flow/interactive_recorder.py @@ -81,7 +81,7 @@ import tempfile import uuid from pathlib import Path -from typing import Any, Callable, Optional +from typing import Any, Callable, Optional, cast from urllib.parse import urlsplit from PIL import Image @@ -420,7 +420,11 @@ def select_attached_page( const GLOBAL_KEY = '__oaflowRecorder'; const CLEANUP_KEY = '__oaflowCleanup_' + SESSION_ID; const previous = window[GLOBAL_KEY]; - if (previous && previous.sessionId === SESSION_ID) return; + // A Window can outlive its document: a popup's initial about:blank Window + // is REUSED by its first same-origin navigation (HTML spec), so the same + // session re-arms when the document differs, never only by session id. + if (previous && previous.sessionId === SESSION_ID + && previous.document === document) return; if (previous && typeof previous.cleanup === 'function') { try { previous.cleanup(); } catch (e) {} } @@ -589,6 +593,7 @@ def select_attached_page( } window[GLOBAL_KEY] = { sessionId: SESSION_ID, + document, stopEvents, cleanup, privacyStatus: () => ({ok: privacyBoundaryError === null, @@ -2268,6 +2273,18 @@ def __init__( self._context_page_baselines: list[tuple[Any, tuple[Any, ...]]] = [] self._context = None self._privacy_cdp = None + # Multi-surface (owned browser only, docs/MULTI_SURFACE.md): the tab + # the recording started in, pages the context opened that an action + # has not claimed yet, and one privacy CDP session per followed page. + self._primary_page: Any = None + self._init_js = "" + self._pages_final = False + self._pending_pages: list[Any] = [] + # Attached browser: followed pages that already carry this session's + # binding and init script (an owned browser binds the whole context). + self._bound_followed_pages: list[Any] = [] + self._attached_page_candidates: list[Any] = [] + self._privacy_cdp_by_page: list[tuple[Any, Any]] = [] self._context_pages_at_start: tuple[Any, ...] = () self._finalizing = False self.backend: Optional[PlaywrightBackend] = None @@ -2307,13 +2324,17 @@ def start(self) -> None: self._browser = launch_chromium( self._pw.chromium, headless=self._headless ) - self.page = self._browser.new_page( + # An explicit context, so popups and new tabs the app opens + # share it (and its init script and binding) and can be + # followed as surfaces. + owned_context = self._browser.new_context( viewport={ "width": self._viewport[0], "height": self._viewport[1], }, device_scale_factor=1, ) + self.page = owned_context.new_page() else: try: self._browser = self._pw.chromium.connect_over_cdp( @@ -2348,9 +2369,18 @@ def start(self) -> None: self.page.on("framenavigated", self._frame_navigation_listener) self.page.on("frameattached", self._frame_attached_listener) self.page.on("framedetached", self._frame_detached_listener) - self.page.on("popup", self._popup_listener) + if not self._owns_browser: + # An attached tab stays one surface; an owned browser follows + # its popups through the context "page" listener instead. + self.page.on("popup", self._popup_listener) self._page_lifecycle_listeners_installed = True - self.page.expose_binding( + self._primary_page = self.page + # Owned browser: bind the event channel and the listener script on + # the CONTEXT, so every popup or tab the app opens carries the same + # session, privacy masking, and secret boundaries from its first + # document. An attached browser keeps both on the one selected tab. + binding_target = self._context if self._owns_browser else self.page + binding_target.expose_binding( self._binding_name, lambda source, detail: self._enqueue_browser_event( detail, @@ -2371,7 +2401,8 @@ def start(self) -> None: ) .replace("__SPECIAL_KEYS__", json.dumps(list(_SPECIAL_KEYS))) ) - self.page.add_init_script(init_js) + binding_target.add_init_script(init_js) + self._init_js = init_js if self._owns_browser: self.page.goto(self._url) try: @@ -2410,6 +2441,7 @@ def start(self) -> None: ), structural_state_reader=self._read_scrubbed_page_state, screenshot_guard=self._guard_screenshot_privacy, + bring_to_front_before_capture=not self._owns_browser, ) assert self._recording_dir is not None self.recorder = Recorder( @@ -2417,6 +2449,7 @@ def start(self) -> None: self._recording_dir, app_url=self._url, system_of_record_reader=self._system_of_record_reader, + page_backend_factory=self._surface_page_backend, **self._settle, ) if self._owns_browser: @@ -2660,6 +2693,7 @@ def finish(self) -> Path: self._flush_scroll() assert self.recorder is not None out = self.recorder.finish() + self._pages_final = True if self._listener_error is not None: raise self._listener_error meta_path = out / "meta.json" @@ -2826,8 +2860,20 @@ def _handle_frame_tree_change(self, _frame: Any = None) -> None: self._retain_late_frame_error() def _handle_popup(self, _popup: Any = None) -> None: - """Refuse a second page that the selected recording tab opens.""" + """Follow a popup the selected tab opens; refuse one after the end. + + An attached tab's popup is followed like an owned browser's: it is + claimed as a surface by the recorded action that opened it. After the + final evidence is bound, any new page is refused. + """ + if not self._finalizing and _popup is not None: + if not any(_popup is pending for pending in self._pending_pages): + self._pending_pages.append(_popup) + return + self._refuse_attached_new_page() + + def _refuse_attached_new_page(self) -> None: self.done = True if self._listener_error is None: self._listener_error = BrowserAttachError( @@ -2837,8 +2883,24 @@ def _handle_popup(self, _popup: Any = None) -> None: ) def _handle_context_page(self, _page: Any = None) -> None: - """Irreversibly refuse any page created after context binding.""" + """Follow a page an owned browser opened; refuse it otherwise. + In a browser Flow launched, a popup or new tab is noted here and + claimed as a surface by the recorded action that opened it (see + ``Recorder.record_observed``). An attached browser, or any page that + appears after the final evidence was bound, is refused. + """ + + if self._owns_browser and not self._finalizing and _page is not None: + self._pending_pages.append(_page) + return + if not self._owns_browser and not self._finalizing: + # Judged on the next drain (_assert_no_new_pages): a page one of + # the followed pages opened is followed, any other is refused, + # even one that already closed again. + if _page is not None: + self._attached_page_candidates.append(_page) + return self.done = True if self._listener_error is None: self._listener_error = BrowserAttachError( @@ -2913,6 +2975,14 @@ def _bind_selected_context_page_latch(self) -> None: def _assert_no_new_pages(self) -> None: """Retain a refusal if this recording context gained another page.""" + if self._owns_browser: + # Popups and tabs are followed as surfaces. Once the last action + # is retained, every page must be one (_assert_pages_claimed). + if self._pages_final: + self._assert_pages_claimed() + elif self._listener_error is not None: + raise self._listener_error + return if self.page is None or not self._context_pages_at_start: return try: @@ -2922,15 +2992,320 @@ def _assert_no_new_pages(self) -> None: "the selected browser tab page inventory could not be read; " "recording was refused" ) from exc - for candidate in current_pages: - if not any( - candidate is existing for existing in self._context_pages_at_start - ): - self._handle_popup(candidate) + candidates = list(current_pages) + for late in self._attached_page_candidates: + if not any(late is known for known in candidates): + candidates.append(late) + for candidate in candidates: + if any(candidate is existing for existing in self._context_pages_at_start): + continue + try: + closed = bool(candidate.is_closed()) + except Exception: + closed = False # unknown: judge it as an open page + if closed: + # A page that opened and closed between two drains is + # accounted for only when a followed page opened it. + if self._is_followed_page(candidate): + continue + try: + opener = candidate.opener() + except Exception: + opener = None + if opener is not None and self._is_followed_page(opener): + continue + self._refuse_attached_new_page() break + if self._is_followed_page(candidate): + if ( + self._finalizing + and self.recorder is not None + and not self.recorder.knows_page(candidate) + ): + # Still open at the end, and no recorded action opened it. + self._refuse_attached_new_page() + break + continue + if not self._finalizing and self._opened_by_followed_page(candidate): + self._pending_pages.append(candidate) + continue + self._refuse_attached_new_page() + break if self._listener_error is not None: raise self._listener_error + def _opened_by_followed_page(self, candidate: Any) -> bool: + """Attached browser: did one of the followed pages open ``candidate``? + + ``window.opener`` answers for a popup. A ``noopener`` tab has none, but + the browser's target list still names the target that opened it. A + page with no such link (one the person opened by hand) is not + followed, so the recording is refused. + """ + + try: + opener = candidate.opener() + except Exception: + opener = None + if opener is not None: + return self._is_followed_page(opener) + opener_id = self._target_info(candidate).get("openerId") + if not opener_id: + return False + for page in self._followed_pages(): + if page is candidate: + continue + if self._target_info(page).get("targetId") == opener_id: + return True + return False + + @staticmethod + def _target_info(page: Any) -> dict[str, Any]: + """The DevTools target entry of ``page`` (empty when unreadable).""" + + try: + if page.is_closed(): + return {} + session = page.context.new_cdp_session(page) + except Exception: + return {} + try: + info = session.send("Target.getTargetInfo").get("targetInfo") + return dict(info) if isinstance(info, dict) else {} + except Exception: + return {} + finally: + try: + session.detach() + except Exception: + pass + + def _assert_pages_claimed(self) -> None: + """Owned browser, at finish: every page must be a recorded surface.""" + + if self._primary_page is None or self.recorder is None: + return + try: + current_pages = tuple(self._primary_page.context.pages) + except Exception: + return + for candidate in current_pages: + if any(candidate is start for start in self._context_pages_at_start): + continue + if self.recorder.knows_page(candidate): + continue + try: + closed = candidate.is_closed() + except Exception: + closed = True + if closed: + continue + if self._listener_error is None: + self._listener_error = BrowserAttachError( + "a popup or new tab opened that no recorded action accounts " + "for; Flow stopped before publishing incomplete metadata" + ) + self.done = True + break + if self._listener_error is not None: + raise self._listener_error + + def _is_followed_page(self, page: Any) -> bool: + """The start tab, a pending new page, or a recorded surface.""" + + if page is None: + return False + if page is self._primary_page: + return True + if any(page is pending for pending in self._pending_pages): + return True + return self.recorder is not None and ( + self.recorder.surface_for_page(page) is not None + ) + + def _surface_page_backend(self, page: Any) -> PlaywrightBackend: + """The backend for a followed page: same masking and scrubbed reads. + + Its screenshot guard and structural reader act on ``self.page``, which + is always the page of the ACTIVE surface: a backend only screenshots + while its surface is active (see ``_follow_event_page``). + """ + + self._arm_followed_page(page) + return PlaywrightBackend( + page, + screenshot_scale="device" if self._owns_browser else "css", + screenshot_mask_selectors=_secret_screenshot_selectors( + self._secret_fields, + marker_attribute=self._secret_marker_attribute, + ), + structural_state_reader=self._read_scrubbed_page_state, + screenshot_guard=self._guard_screenshot_privacy, + closed_page_frame=True, + bring_to_front_before_capture=not self._owns_browser, + ) + + def _arm_followed_page(self, page: Any) -> None: + """Make sure every document of a followed page runs the listener. + + Chromium runs a context init script in a popup's initial about:blank + document but not in the document its first same-origin navigation + commits into the SAME Window (HTML window reuse). Each poll tick + therefore checks the followed pages and evaluates the listener where + it is missing; the script itself arms once per document. A frame that + is mid-navigation is retried on the next tick. + """ + + try: + if page is None or page.is_closed(): + return + frames = list(page.frames) + except Exception: + return + for frame in frames: + try: + armed = frame.evaluate( + "(s) => { const r = window.__oaflowRecorder; return !!(r" + " && r.sessionId === s && r.document === document); }", + self._session_id, + ) + if not armed: + frame.evaluate(self._init_js) + except Exception: + continue + + def _bind_attached_page(self, page: Any) -> None: + """Attached browser: give a followed popup this session's channel. + + The selected tab carries the binding and init script on the PAGE (the + rest of the person's browser is left alone), so each popup or tab it + opens gets the same binding, script, and listener once. + """ + + if self._owns_browser or any( + page is bound for bound in self._bound_followed_pages + ): + return + self._bound_followed_pages.append(page) + try: + if page.is_closed(): + return + page.expose_binding( + self._binding_name, + lambda source, detail: self._enqueue_browser_event( + detail, + source=source, + ), + ) + page.add_init_script(self._init_js) + except Exception as exc: + try: + closed = page.is_closed() + except Exception: + closed = True + if closed: + return + raise BrowserAttachError( + "could not install the recording listener in a popup or tab " + "the selected tab opened; recording was refused" + ) from exc + + def _arm_followed_pages(self) -> None: + if not self._init_js: + return + pages = list(self._pending_pages) + if self.recorder is not None: + for surface_id in getattr(self.recorder, "surface_ids", ())[1:]: + pages.append( + getattr(self.recorder.surface_backend(surface_id), "page", None) + ) + for page in pages: + if page is None: + continue + self._bind_attached_page(page) + self._arm_followed_page(page) + + def _follow_event_page(self, event: dict[str, Any]) -> None: + """Record a surface switch when the user acts in another page. + + The pending action is flushed first (that is when a page it opened is + claimed as a surface). The switch event's before frame is the old + surface; its after frame is the new one, settled. + """ + + page = event.pop("_oaflow_page", None) + if page is None or page is self.page: + return + self._flush_click() + self._flush_type() + self._flush_scroll() + assert self.recorder is not None and self.backend is not None + surface_id = self.recorder.surface_for_page(page) + try: + target_closed = page.is_closed() + except Exception: + target_closed = True + if target_closed: + # The user's FIRST action in a popup closed it before Flow could + # see the popup: there is no frame of it to anchor that action + # or the switch to it. + raise BrowserAttachError( + "a popup or tab closed on the first action in it, before Flow " + "captured a frame of it; act in it once more before the action " + "that closes it (or record that step with the scripted " + "recorder), so recording stopped" + ) + if surface_id is None: + raise BrowserAttachError( + "an action happened in a browser page that no recorded action " + "opened; recording stopped before accepting the event" + ) + # When the popup or tab the user was in closed itself (an OAuth or + # approval window), acting on another page again is recorded as an + # explicit switch back with its own identity check; nothing ever + # returns on its own. A closed page shows nothing: its before frame is + # blank and it has no structural state. + before = self.backend.screenshot() + structural_before = self._structural_state() + self._swap_privacy_cdp(page) + self.page = page + self.backend = cast( + PlaywrightBackend, self.recorder.surface_backend(surface_id) + ) + self.recorder.switch_surface( + surface_id, before_png=before, structural_before=structural_before + ) + if self._owns_browser: + self._advance() + else: + # Each attached page has its own CSS viewport: bind the new page's + # exact coordinate space and frame before its first action. + self._rebaseline_attached_viewport() + + def _active_page_closed(self) -> bool: + """Only a followed (non-primary) page may close during recording.""" + + if self.page is self._primary_page: + return False + try: + return self.page is None or bool(self.page.is_closed()) + except Exception: + return True + + def _swap_privacy_cdp(self, page: Any) -> None: + """Keep one closed-shadow privacy CDP session per followed page.""" + + if self._privacy_cdp is not None and self.page is not None: + self._privacy_cdp_by_page = [ + (known, cdp) + for known, cdp in self._privacy_cdp_by_page + if known is not self.page + ] + self._privacy_cdp_by_page.append((self.page, self._privacy_cdp)) + self._privacy_cdp = next( + (cdp for known, cdp in self._privacy_cdp_by_page if known is page), None + ) + def _enqueue_browser_event( self, detail: Any, @@ -3005,11 +3380,16 @@ def _enqueue_browser_event( frame_composed = event.pop("__oaflow_frame_composed", None) is True source_page_matches = True source_is_selected_top_level = reported_top_level + source_page = None if source is not None: try: - source_page_matches = source.get("page") is self.page + source_page = source.get("page") + source_page_matches = source_page is self.page or ( + self._is_followed_page(source_page) + ) source_is_selected_top_level = ( - source_page_matches and source.get("frame") is self.page.main_frame + source_page_matches + and source.get("frame") is source_page.main_frame ) except Exception: source_page_matches = False @@ -3109,6 +3489,16 @@ def _enqueue_browser_event( return event["_oaflow_geometry"] = event_geometry if kind == "viewport": + # Only the ACTIVE page's coordinate space is bound. Another + # followed page is rebound when Flow switches to it, and its + # own actions carry their geometry. Chromium also fires + # "resize" when a background tab is shown again (Flow brings + # the active page to front before each capture) without + # changing its viewport: that is not a new coordinate space. + if source_page is not None and source_page is not self.page: + return + if event_geometry == self._attached_geometry: + return self._viewport_dirty = True return else: @@ -3125,6 +3515,9 @@ def _enqueue_browser_event( ) self.done = True return + if source_page is not None: + # Which page the user acted in; _follow_event_page reads it. + event["_oaflow_page"] = source_page self._pyq.append(event) def _origin_only_url(self) -> str: @@ -3221,13 +3614,7 @@ def _note_withheld_structural_text(self, reason: str) -> None: def _cleanup_page_listeners(self) -> None: """Remove this session's current-document listeners before detach.""" - if self.page is None: - return - try: - frames = list(self.page.frames) - except Exception: - frames = [] - for frame in frames: + for frame in self._followed_frames(): try: frame.evaluate( """sessionId => { @@ -3248,13 +3635,7 @@ def _cleanup_page_listeners(self) -> None: def _cleanup_secret_markers(self) -> None: """Remove this session's temporary secret-mask attributes.""" - if self.page is None: - return - try: - frames = list(self.page.frames) - except Exception: - frames = [] - for frame in frames: + for frame in self._followed_frames(): try: frame.evaluate( """([sessionId, marker]) => { @@ -3284,16 +3665,48 @@ def _cleanup_secret_markers(self) -> None: except Exception: continue + def _followed_pages(self) -> list[Any]: + """The active page, the start tab, and every followed popup or tab.""" + + pages: list[Any] = [self.page, self._primary_page, *self._pending_pages] + if self.recorder is not None: + for surface_id in getattr(self.recorder, "surface_ids", ()): + pages.append( + getattr(self.recorder.surface_backend(surface_id), "page", None) + ) + unique: list[Any] = [] + for page in pages: + if page is not None and not any(page is seen for seen in unique): + unique.append(page) + return unique + + def _followed_frames(self) -> list[Any]: + frames: list[Any] = [] + for page in self._followed_pages(): + try: + if page.is_closed(): + continue + frames.extend(page.frames) + except Exception: + continue + return frames + def _stop_browser_connection(self) -> None: """Close an owned browser or detach without closing an external one.""" self._cleanup_page_listeners() self._cleanup_secret_markers() - privacy_cdp, self._privacy_cdp = self._privacy_cdp, None + privacy_cdps = [self._privacy_cdp] + [ + cdp for _page, cdp in self._privacy_cdp_by_page + ] + self._privacy_cdp = None + self._privacy_cdp_by_page = [] browser, self._browser = self._browser, None playwright, self._pw = self._pw, None try: - if privacy_cdp is not None: + for privacy_cdp in privacy_cdps: + if privacy_cdp is None: + continue try: privacy_cdp.detach() except Exception: @@ -3308,8 +3721,25 @@ def _stop_browser_connection(self) -> None: # Keep all local lifecycle latches active until the external # Playwright connection has detached. They do not remain in # Chromium after the connection closes. - if self.backend is not None: - self.backend.stop_screenshot_mask_tracking() + tracked = [self.backend] + if self.recorder is not None: + # Popups and tabs are followed as surface backends. + tracked.extend( + self.recorder.surface_backend(surface_id) + for surface_id in getattr(self.recorder, "surface_ids", ()) + ) + seen: list[Any] = [] + for tracked_backend in tracked: + if tracked_backend is None or any( + tracked_backend is done for done in seen + ): + continue + seen.append(tracked_backend) + stop = getattr( + tracked_backend, "stop_screenshot_mask_tracking", None + ) + if callable(stop): + stop() self._page_lifecycle_listeners_installed = False self._context_page_listener_installed = False self._context_page_latches.clear() @@ -3326,7 +3756,7 @@ def _drain_event_queue(self) -> bool: del self._pyq[:] self._validate_event_batch(batch) rebased = False - if not self._owns_browser: + if not self._owns_browser and not self._active_page_closed(): current_geometry = self._read_attached_geometry() if self._viewport_dirty or current_geometry != self._attached_geometry: if batch: @@ -3338,6 +3768,9 @@ def _drain_event_queue(self) -> bool: self._rebaseline_attached_viewport() rebased = True for event in batch: + # A switch to another followed page comes first: the event's + # geometry belongs to the page it came from. + self._follow_event_page(event) if not self._owns_browser: event_geometry = event.pop("_oaflow_geometry", None) if event_geometry != self._attached_geometry: @@ -3353,9 +3786,13 @@ def _drain_event_queue(self) -> bool: # separate, falsely exact step. Same-field input and one scroll # batch remain safe to coalesce. self._validate_event_batch([event, *self._pyq]) - if not self._owns_browser and ( - self._viewport_dirty - or self._read_attached_geometry() != self._attached_geometry + if ( + not self._owns_browser + and not self._active_page_closed() + and ( + self._viewport_dirty + or self._read_attached_geometry() != self._attached_geometry + ) ): raise BrowserAttachError( "the browser resized or changed monitor scale while an " @@ -3418,10 +3855,12 @@ def _pump(self) -> bool: if self.done: return False try: - self.page.wait_for_timeout(self._poll_ms) + # Wait on the start tab: a followed popup may close by itself. + (self._primary_page or self.page).wait_for_timeout(self._poll_ms) except Exception: self.done = True return False + self._arm_followed_pages() if not self._drain_event_queue(): # Distinct scroll gestures are separated by pauses; flush a # completed scroll on idle so each becomes its own step. A type run @@ -3914,6 +4353,10 @@ def _read_scrubbed_page_state(self) -> dict[str, Any]: return state def _structural_state(self) -> dict[str, Any]: + if self._active_page_closed(): + # A followed popup that closed itself has no structural state; the + # next action elsewhere is recorded as an explicit switch back. + return {} state: dict[str, Any] = dict(self._read_scrubbed_page_state()) try: page_count = self.backend.page_count diff --git a/packages/flow/openadapt_flow/interop/types.py b/packages/flow/openadapt_flow/interop/types.py index f9c5cc9cd..920335f47 100644 --- a/packages/flow/openadapt_flow/interop/types.py +++ b/packages/flow/openadapt_flow/interop/types.py @@ -110,11 +110,14 @@ class _ActionSchemaDefaults(TypedDict, total=False): # The shared enum map. SELECT_OPTION deliberately stays outside it until the # cross-package schema can carry its commit key and field evidence without -# degrading it to TYPE; :func:`step_to_action` refuses that conversion loudly. +# degrading it to TYPE. SWITCH_SURFACE is a flow-only control step (it moves +# the replay between bound surfaces) with no canonical ActionType. +# :func:`step_to_action` refuses both conversions loudly. +_FLOW_ONLY_ACTION_KINDS = frozenset( + {ir.ActionKind.SELECT_OPTION, ir.ActionKind.SWITCH_SURFACE} +) ACTION_KIND_TO_ACTION_TYPE: dict[ir.ActionKind, str] = { - kind: kind.value - for kind in ir.ActionKind - if kind is not ir.ActionKind.SELECT_OPTION + kind: kind.value for kind in ir.ActionKind if kind not in _FLOW_ONLY_ACTION_KINDS } @@ -178,6 +181,11 @@ def step_to_action(step: ir.Step) -> "Action": "SELECT_OPTION is not representable in openadapt-types without " "losing its atomic commit contract" ) + if step.action is ir.ActionKind.SWITCH_SURFACE: + raise ValueError( + "SWITCH_SURFACE is a flow-only control step with no openadapt-types " + "ActionType" + ) action_type = ActionType(ACTION_KIND_TO_ACTION_TYPE[step.action]) # Target: canonical coordinates come from the anchor's click point. The diff --git a/packages/flow/openadapt_flow/ir.py b/packages/flow/openadapt_flow/ir.py index e4117021a..7ae47ffe3 100644 --- a/packages/flow/openadapt_flow/ir.py +++ b/packages/flow/openadapt_flow/ir.py @@ -92,6 +92,12 @@ class ActionKind(str, Enum): HOTKEY = "hotkey" WAIT = "wait" SCROLL = "scroll" + # Move execution to another declared surface (a browser tab or popup this + # run opened, or another backend such as a native Windows app or an RDP + # window). It delivers no input. It is typed so an older runtime, which + # does not know the value, refuses the whole bundle instead of driving the + # wrong window (docs/MULTI_SURFACE.md). + SWITCH_SURFACE = "switch_surface" class Landmark(BaseModel): @@ -1027,6 +1033,288 @@ class SubmitBinding(BaseModel): ) +# -- multi-surface workflows (docs/MULTI_SURFACE.md) --------------------------- +# +# A workflow can span browser tabs, popups, and backends. Every surface it uses +# is declared once on ``Workflow.surfaces``; every step names the surface it +# runs on (``Step.surface``); a step whose action opens tabs or popups lists +# them in order (``Step.opens``); and execution only ever moves between +# surfaces through an explicit SWITCH_SURFACE step with its own identity check. +# A bundle with no declared surfaces is a single-surface bundle and replays +# exactly as before. + +#: A surface id: short, lower-case, closed alphabet (it appears in run reports). +SURFACE_ID_PATTERN = r"^[a-z][a-z0-9_]{0,31}$" +#: The id of the surface a run starts on when a recording names none. +PRIMARY_SURFACE_ID = "main" +#: Most pages one step may open (a guard against a popup storm being recorded). +MAX_PAGES_OPENED_PER_STEP = 8 + + +def _compiled_full_match(pattern: Optional[str], label: str) -> Optional[str]: + if pattern is None: + return None + try: + re.compile(pattern) + except re.error as exc: + raise ValueError(f"{label} is not a valid regular expression") from exc + return pattern + + +class SurfaceSpec(BaseModel): + """One surface a multi-surface workflow runs on. + + Three shapes, distinguished by their fields: + + - the PRIMARY surface: the first entry, the backend the run is started + with (``opened_by_step`` is None); + - a PAGE surface: a browser tab or popup that a recorded step opened + (``opened_by_step`` names that step). At replay it is identified by the + step that opened it, its position among the pages that step opened, the + opener relation, its origin, and full-match URL/title patterns; + - a BACKEND surface: another backend the caller supplies by id (a native + app, an RDP/Citrix window, another browser). ``opened_by_step`` is None + and the runtime refuses to switch to it unless the supplied backend is of + the declared ``kind``. + + The URL and title patterns can echo application content, so they live only + inside ``workflow.json`` (sealed with it when the bundle is encrypted) and + are never copied into a manifest, run report, or refusal message. + """ + + model_config = ConfigDict( + extra="forbid", + frozen=True, + hide_input_in_errors=True, + ) + + id: str = Field(pattern=SURFACE_ID_PATTERN) + kind: ExecutionTargetKind + opened_by_step: Optional[str] = Field(default=None, min_length=1, max_length=128) + opener: Optional[str] = Field(default=None, pattern=SURFACE_ID_PATTERN) + open_order: int = Field(default=0, ge=0, lt=MAX_PAGES_OPENED_PER_STEP) + opener_link: Optional[Literal["window_opener", "none"]] = Field( + default=None, + description=( + "PAGE surfaces: 'window_opener' when the recorded page's " + "window.opener was the opener surface's page (window.open, or a " + "link without noopener); 'none' when it had no opener (a " + "target=_blank link, which implies noopener). Replay requires the " + "same relation." + ), + ) + origin: Optional[str] = Field( + default=None, + max_length=512, + description="PAGE surfaces: exact scheme://host[:port] at open time.", + ) + url_pattern: Optional[str] = Field( + default=None, + max_length=2048, + description=( + "PAGE surfaces: regular expression the page URL must FULLY match " + "when it is claimed. Recorded parameter values are generalized." + ), + ) + title_pattern: Optional[str] = Field( + default=None, + max_length=1024, + description="PAGE surfaces: optional full-match regex for the title.", + ) + + @field_validator("url_pattern") + @classmethod + def _valid_url_pattern(cls, value: Optional[str]) -> Optional[str]: + return _compiled_full_match(value, "url_pattern") + + @field_validator("title_pattern") + @classmethod + def _valid_title_pattern(cls, value: Optional[str]) -> Optional[str]: + return _compiled_full_match(value, "title_pattern") + + @property + def is_page(self) -> bool: + """True for a tab/popup that a recorded step opened.""" + + return self.opened_by_step is not None + + @model_validator(mode="after") + def _validate_shape(self) -> "SurfaceSpec": + page_only = ( + self.opener, + self.opener_link, + self.origin, + self.url_pattern, + self.title_pattern, + ) + if self.opened_by_step is None: + if any(value is not None for value in page_only[:2]): + raise ValueError( + "only a page surface (opened_by_step set) has an opener" + ) + if self.open_order != 0: + raise ValueError("only a page surface has an open_order") + return self + if self.kind != "web": + raise ValueError("a page surface opened by a step must be kind 'web'") + if self.opener is None or self.opener_link is None: + raise ValueError("a page surface needs its opener and opener_link") + if self.opener == self.id: + raise ValueError("a page surface cannot open itself") + if self.origin is None or self.url_pattern is None: + raise ValueError("a page surface needs its origin and url_pattern") + return self + + +def surface_contract_errors(workflow: "Workflow") -> list[str]: + """Structural errors in a workflow's multi-surface contract. + + Empty for a valid contract, and always empty for a single-surface bundle + that uses none of the multi-surface fields. Checked when a workflow is + built or loaded, so a bundle whose surfaces do not line up never runs. + Linear bundles are also checked for continuity: a step runs on the surface + the previous SWITCH_SURFACE selected, and a page can be switched to only + after the step that opens it. + """ + + errors: list[str] = [] + linear = list(workflow.steps) + graph_steps: list[Step] = [] + for graph in [ + *([workflow.program] if workflow.program is not None else []), + *workflow.subflows.values(), + ]: + for state in graph.states.values(): + if state.step is not None: + graph_steps.append(state.step) + all_steps = [*linear, *graph_steps] + uses_fields = any( + step.surface is not None + or step.opens + or step.action is ActionKind.SWITCH_SURFACE + for step in all_steps + ) + surfaces = workflow.surfaces + if not surfaces: + if uses_fields: + errors.append( + "steps name surfaces, open pages, or switch surface, but the " + "workflow declares no surfaces" + ) + return errors + + by_id: dict[str, SurfaceSpec] = {} + for spec in surfaces: + if spec.id in by_id: + errors.append(f"surface id {spec.id!r} is declared twice") + by_id[spec.id] = spec + primary = surfaces[0] + if primary.is_page: + errors.append("the first (primary) surface cannot be a page a step opened") + if workflow.surface is not None and primary.kind != workflow.surface: + errors.append("the primary surface kind contradicts Workflow.surface") + + steps_by_id = {step.id: step for step in all_steps} + expected_opens: dict[str, list[tuple[int, str]]] = {} + for spec in surfaces: + if not spec.is_page: + continue + assert spec.opened_by_step is not None + if spec.opened_by_step not in steps_by_id: + errors.append(f"surface {spec.id!r} is opened by an unknown step") + if spec.opener not in by_id: + errors.append(f"surface {spec.id!r} names an undeclared opener") + elif by_id[spec.opener].kind != "web": + errors.append(f"surface {spec.id!r} has a non-browser opener") + expected_opens.setdefault(spec.opened_by_step, []).append( + (spec.open_order, spec.id) + ) + for step in all_steps: + if step.surface is not None and step.surface not in by_id: + errors.append(f"step {step.id!r} names undeclared surface") + declared = [sid for _, sid in sorted(expected_opens.get(step.id, []))] + if list(step.opens) != declared: + errors.append( + f"step {step.id!r} opens {list(step.opens)!r} but the surfaces " + f"opened by it are {declared!r} (in open_order)" + ) + for order, sid in enumerate(step.opens): + opened = by_id.get(sid) + if opened is not None and opened.open_order != order: + errors.append(f"surface {sid!r} open_order does not match step") + if opened is not None and step.action is not ActionKind.SWITCH_SURFACE: + if opened.opener != (step.surface or primary.id): + errors.append( + f"surface {sid!r} opener is not the surface step " + f"{step.id!r} runs on" + ) + + # Continuity of the linear path (the program interpreter enforces the + # same rule at run time, where the path is known). + active = primary.id + available = {spec.id for spec in surfaces if not spec.is_page} + for step in linear: + if step.action is ActionKind.SWITCH_SURFACE: + target = step.surface + if target not in available: + errors.append( + f"step {step.id!r} switches to {target!r} before that " + "surface exists" + ) + elif target == active: + errors.append(f"step {step.id!r} switches to the active surface") + if target is not None: + active = target + continue + if (step.surface or primary.id) != active: + errors.append( + f"step {step.id!r} runs on {step.surface or primary.id!r} but " + f"the active surface is {active!r}; a surface change must be " + "an explicit switch_surface step" + ) + available.update(step.opens) + return errors + + +class DialogResponse(BaseModel): + """How a step answers the JavaScript dialog its action raises. + + A browser ``alert``/``confirm``/``prompt``/``beforeunload`` dialog is not a + surface: it is a screen state of the page that raised it, and it blocks + that page until it is answered. On the browser backend the step's action + arms a one-shot answer before delivering input; the dialog's message is + its identity. A dialog that does not show exactly ``message`` is + dismissed, never accepted, and the step halts. On a screen-driven backend + (desktop, RDP, Citrix) a dialog is just pixels and is recorded as the + clicks that answer it, so this field is browser-only + (docs/MULTI_SURFACE.md, "Dialogs"). + + The identity is either the exact ``message`` or ``message_pattern``, a + full-match regex. The compiler writes the pattern whenever the recorded + message carried a demonstrated parameter value or digits (a record + number, a date), so the bundle never seals that value. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + response: Literal["accept", "dismiss"] + message: Optional[str] = Field(default=None, min_length=1, max_length=2000) + message_pattern: Optional[str] = Field(default=None, min_length=1, max_length=4000) + + @model_validator(mode="after") + def _one_identity(self) -> "DialogResponse": + if (self.message is None) == (self.message_pattern is None): + raise ValueError( + "a dialog response carries exactly one of message and message_pattern" + ) + if self.message_pattern is not None: + try: + re.compile(self.message_pattern) + except re.error as exc: + raise ValueError(f"message_pattern is not a valid regex: {exc}") + return self + + class Step(BaseModel): id: str intent: str = Field(description="Human-readable purpose of the step") @@ -1146,6 +1434,79 @@ class Step(BaseModel): # ``guard``: a deterministic precondition evaluated on the entry frame. # Unmet => HALT (default) or SKIP the step (see Guard.on_unmet). guard: Optional[Guard] = None + # Multi-surface workflows (docs/MULTI_SURFACE.md). All three are None/empty + # on a single-surface bundle, which replays exactly as before. + surface: Optional[str] = Field( + default=None, + pattern=SURFACE_ID_PATTERN, + description=( + "Id of the declared surface (Workflow.surfaces) this step runs on;" + " for SWITCH_SURFACE, the surface it switches TO. None means the" + " primary surface." + ), + ) + opens: list[str] = Field( + default_factory=list, + max_length=MAX_PAGES_OPENED_PER_STEP, + description=( + "Ids of the page surfaces this step's action opens (new tabs," + " popups), in the order the browser created them. At replay the" + " step must open exactly these pages; any other new page halts." + ), + ) + surface_origin: Optional[str] = Field( + default=None, + max_length=512, + description=( + "SWITCH_SURFACE to a browser surface: the exact scheme://host[:port]" + " the target page showed when the demonstration switched to it." + ), + ) + dialog: Optional[DialogResponse] = Field( + default=None, + description=( + "Browser backend: the JavaScript dialog this step's action raises," + " answered by the backend when its message matches exactly." + ), + ) + + @model_validator(mode="after") + def _validate_surface_contract(self) -> "Step": + if len(set(self.opens)) != len(self.opens): + raise ValueError("opens lists a surface twice") + for sid in self.opens: + if not re.fullmatch(SURFACE_ID_PATTERN, sid): + raise ValueError("opens holds an invalid surface id") + if self.action is not ActionKind.SWITCH_SURFACE: + if self.surface_origin is not None: + raise ValueError("surface_origin is valid only on SWITCH_SURFACE") + if self.dialog is not None and self.action is ActionKind.WAIT: + raise ValueError("a WAIT step delivers no input and raises no dialog") + return self + if self.surface is None: + raise ValueError("SWITCH_SURFACE names the surface it switches to") + carried = { + "anchor": self.anchor is not None, + "text": self.text is not None, + "param": self.param is not None, + "secret": self.secret, + "key": self.key is not None, + "drag_end_anchor": self.drag_end_anchor is not None, + "scroll": self.scroll_dx is not None or self.scroll_dy is not None, + "effects": bool(self.effects), + "api_binding": self.api_binding is not None, + "opens": bool(self.opens), + "dialog": self.dialog is not None, + "guard": self.guard is not None, + "wait_until": self.wait_until is not None, + "irreversible risk": self.risk == "irreversible", + } + present = sorted(name for name, value in carried.items() if value) + if present: + raise ValueError( + "SWITCH_SURFACE delivers no input and carries no " + ", ".join(present) + ) + return self @model_validator(mode="after") def _validate_rich_action_contract(self) -> "Step": @@ -2042,6 +2403,11 @@ class Workflow(BaseModel): # None for every pre-qualification bundle and omitted from its serialized # bytes by ``_serialize_compatible`` below. qualification: Optional["QualificationProject"] = None + # Multi-surface workflows (docs/MULTI_SURFACE.md): every browser tab, + # popup, and backend the workflow runs on, primary first. Empty (the + # default) for a single-surface bundle, and then omitted from serialized + # bytes by ``_serialize_compatible`` below so legacy bundles round-trip. + surfaces: list[SurfaceSpec] = Field(default_factory=list, max_length=32) # Where the program came from, when that limits how it may run. # ``"agent_run"``: compiled from a computer-use agent's run # (:mod:`openadapt_flow.jit`). Such a bundle is a draft and runs only with @@ -2081,10 +2447,25 @@ def _serialize_compatible(self, handler: Any) -> dict[str, Any]: data.pop("execution_mode", None) if self.qualification is None: data.pop("qualification", None) + if not self.surfaces: + data.pop("surfaces", None) if self.origin is None: data.pop("origin", None) return data + @model_validator(mode="after") + def _validate_surfaces(self) -> "Workflow": + errors = surface_contract_errors(self) + if errors: + raise ValueError("invalid multi-surface contract: " + "; ".join(errors)) + return self + + @property + def is_multi_surface(self) -> bool: + """True when this workflow declares more than its primary surface.""" + + return len(self.surfaces) > 1 + # -- bundle I/O --------------------------------------------------------- def decrypted_template(self, rel: str) -> Optional[bytes]: @@ -3258,15 +3639,66 @@ class StepResult(BaseModel): before_png: Optional[str] = None # run-dir-relative paths after_png: Optional[str] = None elapsed_ms: float = 0.0 + #: Multi-surface runs only (docs/MULTI_SURFACE.md): the id of the surface + #: this step ran on (for a SWITCH_SURFACE step, the surface it made + #: active; for a refused switch, the surface the run stayed on). None on + #: every single-surface run, and then omitted from the serialized result. + surface: Optional[str] = None @model_serializer(mode="wrap") - def _omit_empty_field_checks(self, handler: Any) -> Any: - data = handler(self) + def _serialize_compatible(self, handler: Any) -> dict[str, Any]: + """Keep single-surface results byte-identical to older releases.""" + + data: dict[str, Any] = handler(self) + if self.surface is None: + data.pop("surface", None) if isinstance(data, dict) and not self.effect_field_checks: data.pop("effect_field_checks", None) return data +class ObservedSurfaceEnvironment(BaseModel): + """PHI-free environment observation of one non-primary surface. + + A multi-surface qualification case observes every surface it switches to + against that surface's qualified boundary + (``QualificationProject.surface_environments``). The same digests a + single-surface report keeps in its ``observed_*`` fields are kept here, + one entry per surface id. + """ + + model_config = ConfigDict(extra="forbid") + + target_kind: str + application_sha256: str = Field(pattern="^[a-f0-9]{64}$") + application_version_sha256: str = Field(pattern="^[a-f0-9]{64}$") + session_sha256: str = Field(pattern="^[a-f0-9]{64}$") + environment_digest: str = Field(pattern="^[a-f0-9]{64}$") + observer_id: str + observer_contract_sha256: str = Field(pattern="^[a-f0-9]{64}$") + binding_sha256: str = Field(pattern="^[a-f0-9]{64}$") + + +class SurfaceObservation(BaseModel): + """The last-seen identity observation of one surface (never a handle). + + A surface is a name plus an identity check, not a live window. Durable + checkpoints keep this observation per surface so a resume can re-bind each + surface by identity (docs/MULTI_SURFACE.md, "Binding by identity"). It + holds no URL path, title, or screen text: the origin is the structured + attribute already declared in the bundle, and the screen is kept only as a + perceptual hash of the last frame Flow saw on that surface. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + surface: str = Field(pattern=SURFACE_ID_PATTERN) + kind: str = Field(min_length=1, max_length=32) + observed_after_step: Optional[str] = Field(default=None, max_length=128) + origin: Optional[str] = Field(default=None, max_length=512) + screen_phash: Optional[str] = Field(default=None, pattern="^[a-f0-9]{4,64}$") + + class UnarmedStep(BaseModel): """A click step that will proceed with NO identity verification.""" @@ -4072,6 +4504,13 @@ class RunReport(BaseModel): qualification_environment_observer_contract_sha256: Optional[str] = Field( default=None, pattern="^[a-f0-9]{64}$" ) + #: Multi-surface qualification cases only: the environment observed on + #: each non-primary surface the run switched to, keyed by surface id. The + #: primary surface stays in the ``observed_*`` fields above. Empty (and + #: omitted from the serialized report) otherwise. + observed_surface_environments: dict[str, ObservedSurfaceEnvironment] = Field( + default_factory=dict + ) governed_authorized_effect_contracts: dict[str, list[str]] = Field( default_factory=dict ) @@ -4159,6 +4598,15 @@ class RunReport(BaseModel): ), ) + @model_serializer(mode="wrap") + def _serialize_compatible(self, handler: Any) -> dict[str, Any]: + """Omit additive multi-surface evidence from single-surface reports.""" + + data: dict[str, Any] = handler(self) + if not self.observed_surface_environments: + data.pop("observed_surface_environments", None) + return data + @model_validator(mode="after") def _validate_outcome_envelope_binding(self) -> "RunReport": """Keep the transported envelope bound to this exact report. diff --git a/packages/flow/openadapt_flow/qualification.py b/packages/flow/openadapt_flow/qualification.py index a9774166f..31c3a3817 100644 --- a/packages/flow/openadapt_flow/qualification.py +++ b/packages/flow/openadapt_flow/qualification.py @@ -912,6 +912,13 @@ class QualificationProject(BaseModel): created_at: str = Field(default_factory=_now) updated_at: str = Field(default_factory=_now) environment: EnvironmentBoundary + #: Multi-surface workflows only (docs/MULTI_SURFACE.md): the qualified + #: environment of each NON-primary surface, keyed by surface id. The + #: primary surface is ``environment``. A qualification case run of a + #: multi-surface workflow observes every surface it switches to against + #: its boundary here and refuses a surface that has none. Empty (and + #: omitted from the contract bytes) for every single-surface project. + surface_environments: dict[str, EnvironmentBoundary] = Field(default_factory=dict) minimum_effect_tier: VerificationTier = ( VerificationTier.PERSISTED_STATE_REACQUISITION ) @@ -987,6 +994,11 @@ def _consistent_keys(self) -> "QualificationProject": raise ValueError( f"trusted {key_kind} public key must be 32-byte Ed25519" ) + from openadapt_flow.ir import SURFACE_ID_PATTERN + + for surface_id in self.surface_environments: + if not re.fullmatch(SURFACE_ID_PATTERN, surface_id): + raise ValueError("surface environment key is not a surface id") case_ids = [case.id for case in self.cases] if len(case_ids) != len(set(case_ids)): raise ValueError("qualification case ids must be unique") @@ -1018,6 +1030,8 @@ def _serialize_compatible(self, handler: Any) -> dict[str, Any]: """Omit the additive trust store until a fault driver is bound.""" data: dict[str, Any] = handler(self) + if not self.surface_environments: + data.pop("surface_environments", None) if not self.trusted_fault_driver_keys: data.pop("trusted_fault_driver_keys", None) if not self.judgment_fact_schemas: @@ -1221,7 +1235,11 @@ def _inferred_action_classification(step: "Step") -> ActionRiskClassification: elif effects: classification = ActionRiskClass.STATE_CHANGING explanation = "step declares a business-effect contract" - elif step.action in {ActionKind.WAIT, ActionKind.SCROLL}: + elif step.action in { + ActionKind.WAIT, + ActionKind.SCROLL, + ActionKind.SWITCH_SURFACE, + }: classification = ActionRiskClass.READ_ONLY explanation = f"{step.action.value} does not actuate business state" else: @@ -2511,6 +2529,72 @@ def _one_evidence( return matches[0], None +def _surface_environment_evidence_refusal( + report: Any, workflow: "Workflow", project: "QualificationProject" +) -> Optional[str]: + """Check a multi-surface case's per-surface environment observations. + + Every non-primary surface a step ran on must carry an observation that + matches that surface's qualified boundary exactly (the same fields the + primary check compares), and no observation may name a surface the run + never used. Single-surface reports carry none and pass unchanged. + """ + + from openadapt_flow.qualification_environment import ( + qualification_environment_binding_sha256, + ) + + observed = dict(getattr(report, "observed_surface_environments", {}) or {}) + if not workflow.surfaces: + return ( + "case report carries surface environments for a single-surface workflow" + if observed + else None + ) + primary = workflow.surfaces[0].id + used = { + result.surface + for result in report.results + if result.surface is not None and result.surface != primary + } + if set(observed) != used: + return "case report surface environments do not match the surfaces it ran on" + for surface_id in sorted(used): + boundary = project.surface_environments.get(surface_id) + item = observed[surface_id] + expected_identity = ( + None if boundary is None else boundary.expected_application_identity + ) + if boundary is None or expected_identity is None: + return f"surface '{surface_id}' has no qualified environment boundary" + binding = qualification_environment_binding_sha256( + target_kind=item.target_kind, + observer_id=item.observer_id, + observer_contract_sha256=item.observer_contract_sha256, + application_identity_sha256=item.application_sha256, + application_version_sha256=item.application_version_sha256, + environment_digest=item.environment_digest, + session_identity_sha256=item.session_sha256, + ) + if ( + item.target_kind != boundary.target_kind + or item.application_sha256 + != hashlib.sha256(expected_identity.encode("utf-8")).hexdigest() + or item.application_version_sha256 + != hashlib.sha256(boundary.application_version.encode("utf-8")).hexdigest() + or item.environment_digest != boundary.environment_digest + or item.observer_id != boundary.environment_observer_id + or item.observer_contract_sha256 + != boundary.environment_observer_contract_sha256 + or item.binding_sha256 != binding + ): + return ( + f"case report environment for surface '{surface_id}' does not " + "match the project" + ) + return None + + def _case_run_report_integrity_error( *, workflow: "Workflow", @@ -3880,6 +3964,9 @@ def exact_confirmed_effect_error( QualificationRefusalCode.CASE_ENVIRONMENT_CHANGED, "case report environment binding does not match the project", ) + surface_refusal = _surface_environment_evidence_refusal(report, workflow, project) + if surface_refusal is not None: + return (QualificationRefusalCode.CASE_ENVIRONMENT_CHANGED, surface_refusal) if ( not report.qualification_evidence_only diff --git a/packages/flow/openadapt_flow/qualification_faults.py b/packages/flow/openadapt_flow/qualification_faults.py index 76039e866..0762adc49 100644 --- a/packages/flow/openadapt_flow/qualification_faults.py +++ b/packages/flow/openadapt_flow/qualification_faults.py @@ -19,7 +19,7 @@ from collections.abc import Sequence from dataclasses import dataclass from importlib.metadata import entry_points -from typing import Any, Literal, Protocol, runtime_checkable +from typing import Any, Literal, Optional, Protocol, runtime_checkable from cryptography.exceptions import InvalidSignature from cryptography.hazmat.primitives.asymmetric.ed25519 import ( @@ -412,6 +412,9 @@ class QualificationFaultContext: #: with. A driver that signs a frame hash should hash this capture, since #: two capture paths can encode identical pixels as different bytes. capture_frame: Any = None + #: Multi-surface runs only: the surface the targeted step runs on (and + #: that ``backend`` drives). None on a single-surface run. + surface_id: Optional[str] = None @dataclass(frozen=True) diff --git a/packages/flow/openadapt_flow/recorder.py b/packages/flow/openadapt_flow/recorder.py index 43f5f3f77..2d30ae60f 100644 --- a/packages/flow/openadapt_flow/recorder.py +++ b/packages/flow/openadapt_flow/recorder.py @@ -35,6 +35,14 @@ frames/{i:04d}_before.png frames/{i:04d}_after.png # captured after the action settled +Multi-surface recordings (docs/MULTI_SURFACE.md) add, only once a second +surface exists: ``"opens": [...]`` on an event whose action opened browser +pages, a ``{"kind": "switch_surface", "surface": ""}`` event for every +surface change, ``"surface"`` on every later event, and ``"surfaces"`` in +``meta.json`` (each page's opening event, opener relation, URL, and title, from +which the compiler derives the replay-time patterns). A recording that never +opens a page or switches is written exactly as before. + The settle wait is implemented inline here (perceptual-hash polling) on purpose: this module must not depend on `openadapt_flow.vision`. """ @@ -53,7 +61,39 @@ from openadapt_flow.backend import Backend, RichPointerActionBackend from openadapt_flow.image_hash import perceptual_hash -from openadapt_flow.ir import StructuralLocator, structural_locator_is_locatable +from openadapt_flow.ir import ( + MAX_PAGES_OPENED_PER_STEP, + PRIMARY_SURFACE_ID, + StructuralLocator, + structural_locator_is_locatable, +) + +#: Bound on waiting for a newly opened page's first navigation. +_NEW_PAGE_LOAD_TIMEOUT_S = 10.0 +# How long a recorded action that raises a browser dialog waits for it. +_DIALOG_WAIT_S = 5.0 + + +def _has_window_opener(page: Any) -> bool: + """Whether the page's own ``window.opener`` is set (False for noopener).""" + + from openadapt_flow.runtime.surfaces import page_has_window_opener + + return bool(page_has_window_opener(page)) + + +def _url_without_query(url: Any) -> str: + """``url`` with its query and fragment removed.""" + + from urllib.parse import urlsplit, urlunsplit + + if not isinstance(url, str) or not url: + return "" + try: + parts = urlsplit(url) + except ValueError: + return "" + return urlunsplit((parts.scheme, parts.netloc, parts.path, "", "")) def _phash(png: bytes) -> str: @@ -92,8 +132,30 @@ def __init__( settle_interval_s: float = 0.1, settle_stable_frames: int = 2, settle_timeout_s: float = 3.0, + surface_id: str = PRIMARY_SURFACE_ID, + surface_kind: Optional[str] = None, + page_backend_factory: Optional[Callable[[Any], Backend]] = None, ) -> None: + # ``_backend`` is always the ACTIVE surface's backend; on a + # single-surface recording it never changes. self._backend = backend + # Builds the backend for a browser page an action opened. The + # interactive recorder passes one that applies its privacy masking + # and scrubbed structural reads to every page it follows. + self._page_backend_factory = page_backend_factory + self._primary_backend = backend + self._primary_surface_id = surface_id + self._primary_surface_kind = surface_kind + self._active_surface = surface_id + self._surface_backends: dict[str, Backend] = {surface_id: backend} + self._surface_meta: list[dict[str, Any]] = [] + self._multi_surface = False + # Set by expect_dialog(): how the NEXT action answers the dialog it + # raises (docs/MULTI_SURFACE.md, "Dialogs"). + self._expected_dialog: Optional[str] = None + self._page_counters = {"popup": 0, "tab": 0} + self._known_pages: list[Any] = [] + self._remember_pages(backend) self._dir = Path(out_dir) self._frames_dir = self._dir / "frames" self._frames_dir.mkdir(parents=True, exist_ok=True) @@ -206,8 +268,8 @@ def scroll(self, dx: int, dy: int) -> None: def finish(self) -> Path: """Write ``meta.json`` and return the recording directory.""" - viewport = self._backend.viewport - meta = { + viewport = self._primary_backend.viewport + meta: dict[str, Any] = { "id": uuid.uuid4().hex, "created_at": datetime.now(timezone.utc).isoformat(), "viewport": [int(viewport[0]), int(viewport[1])], @@ -222,9 +284,248 @@ def finish(self) -> Path: "params": dict(self._params), "secret_params": sorted(self._secret_params), } + if self._multi_surface: + meta["surfaces"] = [ + { + "id": self._primary_surface_id, + "kind": self._primary_kind(), + }, + *self._surface_meta, + ] (self._dir / "meta.json").write_text(json.dumps(meta, indent=2)) return self._dir + # -- multi-surface (docs/MULTI_SURFACE.md) --------------------------------- + + @property + def active_surface(self) -> str: + """Id of the surface the next action is recorded on.""" + + return self._active_surface + + @property + def surface_ids(self) -> tuple[str, ...]: + """Every surface recorded so far, primary first.""" + + return tuple(self._surface_backends) + + def surface_backend(self, surface_id: str) -> Backend: + """The backend recorded for ``surface_id`` (a page opened by a step).""" + + return self._surface_backends[surface_id] + + def surface_for_page(self, page: Any) -> Optional[str]: + """The surface id recorded for browser ``page``, or None.""" + + for surface_id, backend in self._surface_backends.items(): + if self._browser_page(backend) is page: + return surface_id + return None + + def knows_page(self, page: Any) -> bool: + """Whether ``page`` was open at the start or claimed by an action.""" + + return any(page is known for known in self._known_pages) + + def expect_dialog(self, response: str = "accept") -> None: + """The next action raises a browser dialog; answer it with ``response``. + + Browser backend only. The dialog's message is recorded as its identity + and the compiled step answers only a dialog that shows that message. + """ + + if response not in ("accept", "dismiss"): + raise ValueError("a dialog response is 'accept' or 'dismiss'") + self._expected_dialog = response + + def switch_surface( + self, + surface_id: str, + *, + backend: Optional[Backend] = None, + kind: Optional[str] = None, + before_png: Optional[bytes] = None, + structural_before: Optional[dict[str, Any]] = None, + ) -> None: + """Record moving to another surface (a tab/popup, or another backend). + + A page some earlier action opened is switched to by its id (see + ``surface_ids``). Another backend (a native app, an RDP/Citrix window) + is passed in with ``backend``; its kind is read from the backend class + or given as ``kind``. The event's before frame is the old surface and + its after frame the new one, settled. + """ + + import re as _re + + from openadapt_flow.ir import SURFACE_ID_PATTERN + + if surface_id == self._active_surface: + raise ValueError(f"surface {surface_id!r} is already active") + if backend is None: + if surface_id not in self._surface_backends: + raise ValueError( + f"surface {surface_id!r} was not opened in this recording; " + "pass backend= to record a switch to another backend" + ) + target = self._surface_backends[surface_id] + else: + if surface_id in self._surface_backends: + if self._surface_backends[surface_id] is not backend: + raise ValueError(f"surface {surface_id!r} is already recorded") + elif not _re.fullmatch(SURFACE_ID_PATTERN, surface_id): + raise ValueError(f"invalid surface id {surface_id!r}") + else: + self._surface_meta.append( + {"id": surface_id, "kind": self._kind_of(backend, kind)} + ) + self._surface_backends[surface_id] = backend + self._remember_pages(backend) + target = backend + # A caller that observed the switch (the interactive recorder) captured + # the old surface already; a driven recording captures it now. + before = before_png if before_png is not None else self._backend.screenshot() + if structural_before is None: + structural_before = self._structural_state() + self._multi_surface = True + self._backend = target + self._active_surface = surface_id + page = self._browser_page(target) + if page is not None: + try: + page.bring_to_front() + except Exception: + pass + after = self._wait_settled() + self._commit( + {"kind": "switch_surface", "surface": surface_id}, + before, + after, + structural_before, + ) + + def _primary_kind(self) -> str: + return self._kind_of(self._primary_backend, self._primary_surface_kind) + + @staticmethod + def _kind_of(backend: Backend, kind: Optional[str]) -> str: + from openadapt_flow.runtime.surfaces import backend_surface_kinds + + if kind is not None: + return kind + kinds = backend_surface_kinds(backend) + if len(kinds) != 1: + raise ValueError( + "cannot tell this backend's surface kind; pass kind= " + "(web/windows/macos/linux/rdp/citrix)" + ) + return next(iter(kinds)) + + @staticmethod + def _browser_page(backend: Any) -> Any: + page = getattr(backend, "page", None) + if page is None or not hasattr(page, "context"): + return None + return page + + def _context_pages(self, backend: Any) -> list[Any]: + page = self._browser_page(backend) + if page is None: + return [] + try: + return list(page.context.pages) + except Exception: + return [page] + + def _remember_pages(self, backend: Any) -> None: + for page in self._context_pages(backend): + if not any(page is known for known in self._known_pages): + self._known_pages.append(page) + + def _claim_new_pages(self, event_index: int) -> list[str]: + """Record every page opened since the last check, in creation order.""" + + new_pages = [ + page + for page in self._context_pages(self._backend) + if not any(page is known for known in self._known_pages) + ] + opener_page = self._browser_page(self._backend) + + def _opened_by_another_known_page(page: Any) -> bool: + # A page another recorded page opened belongs to THAT page's + # action; it is claimed when that action is recorded. (The + # interactive recorder can see a new page before it has written + # the pending action of the surface the user just left.) + try: + live_opener = page.opener() + except Exception: + return False + return ( + live_opener is not None + and live_opener is not opener_page + and any(live_opener is known for known in self._known_pages) + ) + + new_pages = [p for p in new_pages if not _opened_by_another_known_page(p)] + if not new_pages: + return [] + if len(new_pages) > MAX_PAGES_OPENED_PER_STEP: + raise RuntimeError( + f"one action opened {len(new_pages)} browser pages; at most " + f"{MAX_PAGES_OPENED_PER_STEP} can be recorded per step" + ) + from openadapt_flow.backends.playwright_backend import PlaywrightBackend + + ids: list[str] = [] + for order, page in enumerate(new_pages): + self._known_pages.append(page) + try: + page.wait_for_load_state( + "domcontentloaded", timeout=_NEW_PAGE_LOAD_TIMEOUT_S * 1000 + ) + except Exception: + pass + try: + live_opener = page.opener() + except Exception: + live_opener = None + link = "window_opener" if _has_window_opener(page) else "none" + if live_opener is not None and live_opener is not opener_page: + raise RuntimeError( + "a recorded page was opened by a page that is not the " + "active surface; refusing to guess its opener" + ) + prefix = "popup" if link == "window_opener" else "tab" + self._page_counters[prefix] += 1 + surface_id = f"{prefix}_{self._page_counters[prefix]}" + try: + url = _url_without_query(page.url) + except Exception: + url = "" + # meta.json keeps only what identifies the page at replay: the + # URL without its query or fragment (OAuth code/state, search + # terms), and no title (titles carry record names). + self._surface_meta.append( + { + "id": surface_id, + "kind": "web", + "opened_by_event": event_index, + "opener": self._active_surface, + "open_order": order, + "opener_link": link, + "url": url, + } + ) + self._surface_backends[surface_id] = ( + self._page_backend_factory(page) + if self._page_backend_factory is not None + else PlaywrightBackend(page, closed_page_frame=True) + ) + ids.append(surface_id) + self._multi_surface = True + return ids + @property def event_count(self) -> int: """Number of complete events already persisted.""" @@ -274,8 +575,34 @@ def _record(self, event: dict[str, Any], act: Callable[[], None]) -> None: binding = self._submit_binding_at(int(event["x"]), int(event["y"])) if binding: event = {**event, "submit_binding": binding} + dialog_response, self._expected_dialog = self._expected_dialog, None + if dialog_response is not None: + arm = getattr(self._backend, "arm_dialog", None) + if not callable(arm): + raise RuntimeError( + "this backend cannot see browser dialogs; on a screen-driven " + "backend record the clicks that answer the dialog instead" + ) + arm(dialog_response, None) act() + if dialog_response is not None: + seen = self._backend.take_dialogs(_DIALOG_WAIT_S) # type: ignore[attr-defined] + if len(seen) != 1: + raise RuntimeError( + f"the action raised {len(seen)} browser dialogs; exactly one " + "was expected" + ) + event = { + **event, + "dialog": { + "response": dialog_response, + "message": str(seen[0]["message"]), + }, + } after = self._wait_settled() + opened = self._claim_new_pages(self._i) + if opened: + event = {**event, "opens": opened} self._commit(event, before, after, structural_before) def record_observed( @@ -346,6 +673,12 @@ def record_observed( # click) passes it in; otherwise we settle and capture now. if after_png is None: after_png = self._wait_settled() + # The observed action may have opened browser pages (a popup, a + # target=_blank tab): claim them for this event, exactly as a driven + # action does. A non-browser backend has no pages, so nothing changes. + opened = self._claim_new_pages(self._i) + if opened: + event = {**event, "opens": opened} self._commit( event, before_png, @@ -390,6 +723,8 @@ def _commit( "viewport_before": before_viewport, "viewport_after": after_viewport, } + if self._multi_surface and event.get("kind") != "switch_surface": + line["surface"] = self._active_surface for key, value in structural_before.items(): line[f"{key}_before"] = value if structural_after is None: diff --git a/packages/flow/openadapt_flow/runtime/durable/checkpoint.py b/packages/flow/openadapt_flow/runtime/durable/checkpoint.py index aa69f853b..df298c879 100644 --- a/packages/flow/openadapt_flow/runtime/durable/checkpoint.py +++ b/packages/flow/openadapt_flow/runtime/durable/checkpoint.py @@ -47,7 +47,7 @@ from pathlib import Path from typing import Iterator, Literal, Optional -from pydantic import BaseModel, Field, model_validator +from pydantic import BaseModel, Field, model_serializer, model_validator from openadapt_flow.ir import ( ActionDeliveryReceipt, @@ -61,6 +61,7 @@ ProgramExceptionEvidence, ProgramTransitionEvidence, Resolution, + SurfaceObservation, ) from openadapt_flow.runtime.authorization import ( GovernedRunAuthorization, @@ -251,8 +252,27 @@ class RunCheckpoint(BaseModel): default_factory=list ) attended_reconciliation_at: Optional[str] = None + #: Multi-surface runs only (docs/MULTI_SURFACE.md): the surface this step + #: ran on, the surface active after it, and each surface's last-seen + #: identity observation. Never a live handle: a resume re-binds every + #: surface by identity. Omitted when unset, so a single-surface checkpoint + #: serializes exactly as before. + surface: Optional[str] = None + active_surface: Optional[str] = None + surface_observations: dict[str, SurfaceObservation] = Field(default_factory=dict) created_at: str = Field(default_factory=_now) + @model_serializer(mode="wrap") + def _serialize_compatible(self, handler): # type: ignore[no-untyped-def] + data = handler(self) + if isinstance(data, dict): + for key in ("surface", "active_surface"): + if data.get(key) is None: + data.pop(key, None) + if not data.get("surface_observations"): + data.pop("surface_observations", None) + return data + class PendingEscalation(BaseModel): """A durable pause: the run HALTED for an operator instead of dying. diff --git a/packages/flow/openadapt_flow/runtime/durable/controller.py b/packages/flow/openadapt_flow/runtime/durable/controller.py index b3aed3700..36f412b99 100644 --- a/packages/flow/openadapt_flow/runtime/durable/controller.py +++ b/packages/flow/openadapt_flow/runtime/durable/controller.py @@ -31,7 +31,7 @@ import re import secrets from pathlib import Path -from typing import TYPE_CHECKING, Literal, Optional +from typing import TYPE_CHECKING, Any, Literal, Optional from openadapt_flow.bundle_validation import BundleIntegrityError from openadapt_flow.ir import ( @@ -482,6 +482,8 @@ def record( *, workflow: Optional[Workflow] = None, transition_observation: Optional["TransitionObservation"] = None, + active_surface: Optional[str] = None, + surface_observations: Optional[dict[str, Any]] = None, ) -> None: """Persist the durable artifact for one completed step. @@ -563,6 +565,9 @@ def record( resolution=result.resolution, drift_oracle_calls=result.drift_oracle_calls, heal=result.heal, + surface=result.surface, + active_surface=active_surface, + surface_observations=dict(surface_observations or {}), ) ) self._sync_authority() @@ -850,6 +855,7 @@ def resumed_step_results( resolution=checkpoint.resolution, drift_oracle_calls=checkpoint.drift_oracle_calls, heal=checkpoint.heal, + surface=checkpoint.surface, error=None, ) ) diff --git a/packages/flow/openadapt_flow/runtime/durable/resume.py b/packages/flow/openadapt_flow/runtime/durable/resume.py index 35ee8bb25..3dbb8f7b8 100644 --- a/packages/flow/openadapt_flow/runtime/durable/resume.py +++ b/packages/flow/openadapt_flow/runtime/durable/resume.py @@ -1042,6 +1042,7 @@ def _resume_under_lease( replayer.revalidate_linear_checkpoint_state( last_linear, bundle_dir=resolved_bundle, + workflow=workflow, ) if ( bundle_version(resolved_bundle) != live_bundle_version diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py index 1412238e0..7d9e12e6c 100644 --- a/packages/flow/openadapt_flow/runtime/replayer.py +++ b/packages/flow/openadapt_flow/runtime/replayer.py @@ -101,6 +101,7 @@ InterstitialActionResult, LoopIterationBinding, LoopSpec, + ObservedSurfaceEnvironment, PixelIdentityEvidence, Point, Postcondition, @@ -196,6 +197,7 @@ visual_resolution_anchor_contract_sha256, visual_resolution_evaluator_contract_sha256, ) +from openadapt_flow.runtime.surfaces import SurfaceRouter from openadapt_flow.verification import ( EffectContractMutationError, verifier_effect_tier, @@ -205,6 +207,8 @@ _DeliveryResultT = TypeVar("_DeliveryResultT") _MAX_FRESH_ACTUATION_REACQUISITIONS = 2 +# How long a step that raises a browser dialog waits for it to open. +_DIALOG_WAIT_S = 5.0 # Private module capability. The public ``run`` API cannot mint a durable # continuation. ``runtime.durable.resume`` imports this object only after it # has validated the pause, approval, bundle, checkpoint, and live state. @@ -683,6 +687,14 @@ class Replayer: the save so a person handles it. Its post-run decision is stamped on the report as ``approval``. It can only add refusals; it never changes an outcome word. None (default) leaves replay unchanged. + surface_backends: Multi-surface workflows only + (``Workflow.surfaces``, docs/MULTI_SURFACE.md): the live backend for + each BACKEND surface the workflow switches to (a native app, an + RDP/Citrix window), keyed by surface id. The primary surface is + ``backend``; browser tabs and popups the run opens need nothing + here. A switch to a surface with no supplied backend, or to one + whose backend is of a different kind, halts. Ignored (and must be + empty) for a single-surface workflow. attended_draft_by: Who attends this run of a JIT draft (the canary, :mod:`openadapt_flow.jit`): a person's name, or a registered ``openadapt_flow.jit.authority.Principal`` (an admitter key, or an @@ -732,6 +744,7 @@ def __init__( QualificationEnvironmentObserver ] = None, approval_gate: Optional["ApprovalGate"] = None, + surface_backends: Optional[Mapping[str, Backend]] = None, attended_draft_by: Optional[Any] = None, draft_key_dir: Optional[Path] = None, deployment_surface: Optional[str] = None, @@ -776,6 +789,14 @@ def __init__( ) self.backend = backend + # Multi-surface routing (docs/MULTI_SURFACE.md). ``_surface_router`` is + # set only while a multi-surface workflow runs; ``self.backend`` is then + # the ACTIVE surface's backend, so every gate acts on that surface. + self.surface_backends: dict[str, Backend] = dict(surface_backends or {}) + self._surface_router: Optional[SurfaceRouter] = None + self._surface_environment_observations: dict[ + str, ObservedSurfaceEnvironment + ] = {} # Font/layout drift check (runtime.layout_drift) on the frame each # step acts on, against the fingerprint the compiler stored in the # bundle. None = on for remote-desktop replays (rdp/citrix targets or @@ -1956,6 +1977,27 @@ def acknowledge_delivery(self_nonlocal) -> None: report.total_ms = (time.monotonic() - t_run) * 1000.0 return self._finalize_report(report, workflow, run_dir) + surface_refusal = self._begin_surfaces( + workflow, + resume=resume_from is not None or resume_program is not None, + resume_from=resume_from, + resume_program=resume_program is not None, + bundle_dir=bundle_dir, + ) + if surface_refusal is not None: + report.results.append( + StepResult( + step_id="", + intent="bind the workflow's declared surfaces", + ok=False, + failure_category="governed_refusal", + error=surface_refusal, + ) + ) + report.success = False + report.total_ms = (time.monotonic() - t_run) * 1000.0 + return self._finalize_report(report, workflow, run_dir) + if resume_from is not None: from openadapt_flow.runtime.durable import resumed_step_results @@ -2034,6 +2076,9 @@ def acknowledge_delivery(self_nonlocal) -> None: # Tier-3: verified step -> checkpoint; halt -> pending # escalation (resumable from the last checkpoint). if post_action_refusal is None: + router = self._surface_router + if router is not None and result.ok: + router.observe_active(step.id) durable_run.record( step_index, step, @@ -2045,6 +2090,12 @@ def acknowledge_delivery(self_nonlocal) -> None: if not result.ok else None ), + active_surface=( + router.active_id if router is not None else None + ), + surface_observations=( + router.observations if router is not None else None + ), ) if ( self._durable_continuation_guard is not None @@ -2355,6 +2406,11 @@ def _finalize_report( ) -> RunReport: """Persist exact evidence first, then project its terminal outcome.""" + self._end_surfaces() + if self._surface_environment_observations: + report.observed_surface_environments = dict( + self._surface_environment_observations + ) report.qualification_fault_mutations = list(self._qualification_fault_mutations) self._stamp_execution_outcome(report, workflow, run_dir) self._stamp_approval(report) @@ -4581,13 +4637,87 @@ def revalidate_linear_checkpoint_state( checkpoint: Any, *, bundle_dir: Path, + workflow: Optional[Workflow] = None, ) -> None: - """Prove that the live app still satisfies a linear resume cursor.""" + """Prove that the live app still satisfies a linear resume cursor. - self._revalidate_retained_postconditions( - checkpoint.expected_postconditions, - bundle_dir=bundle_dir, + In a multi-surface run the retained contract belongs to the surface + that was active at the checkpoint. That surface is first bound by + identity (the one live target whose screen shows the retained + contract, plus its declared structured attributes); zero or several + matches refuse the resume. + """ + + active = getattr(checkpoint, "active_surface", None) + if ( + workflow is None + or not workflow.surfaces + or active is None + or active == workflow.surfaces[0].id + ): + self._revalidate_retained_postconditions( + checkpoint.expected_postconditions, + bundle_dir=bundle_dir, + ) + return + backend = self._bind_checkpoint_surface(workflow, checkpoint, bundle_dir) + primary = self.backend + self._activate_surface_backend(backend) + try: + self._revalidate_retained_postconditions( + checkpoint.expected_postconditions, + bundle_dir=bundle_dir, + ) + finally: + self._activate_surface_backend(primary) + + def _bind_checkpoint_surface( + self, workflow: Workflow, checkpoint: Any, bundle_dir: Path + ) -> Any: + """Bind the checkpoint's active surface by identity, or refuse.""" + + active = checkpoint.active_surface + self._surface_identity_context: tuple[Workflow, Optional[Path]] = ( + workflow, + bundle_dir, ) + try: + router = SurfaceRouter( + workflow, + self.backend, + self.surface_backends, + poll_interval_s=self.poll_interval_s, + identity_check=self._surface_identity_check, + resume_active=active, + observations=dict(checkpoint.surface_observations), + ) + except ValueError as exc: + raise StateDiverged(f"{exc}; refusing to resume") from exc + probe = Step( + id=checkpoint.step_id, + intent=f"re-bind surface '{active}' for resume", + action=ActionKind.SWITCH_SURFACE, + surface=active, + expect=[ + condition.model_copy(deep=True) + for condition in checkpoint.expected_postconditions + ], + ) + prefix = f"Resume after step '{checkpoint.step_id}'" + if router.specs[active].is_page: + page, error = router._bind_page(active, probe, prefix) + if error is not None or page is None: + raise StateDiverged(error or f"{prefix}: surface not bound") + return router.backends[active] + backend, error = router._bind_backend(active, probe, prefix) + if error is not None or backend is None: + raise StateDiverged(error or f"{prefix}: surface not bound") + if self._surface_identity_check(backend, probe) is not True: + raise StateDiverged( + f"{prefix}: surface '{active}' does not show its retained " + "identity; refusing to resume" + ) + return backend def _revalidate_retained_postconditions( self, @@ -5806,6 +5936,11 @@ def _qualification_fault_context( effect_verifier=effect_verifier, effects=effects, capture_frame=capture_frame, + surface_id=( + self._surface_router.active_id + if self._surface_router is not None + else None + ), ) def _require_current_fault_driver_binding(self) -> QualificationFaultDriver: @@ -5957,6 +6092,295 @@ def _apply_effect_fault_mutation( ) return mutation.effect_verifier + # -- multi-surface routing (docs/MULTI_SURFACE.md) ----------------------- + + def _begin_surfaces( + self, + workflow: Workflow, + *, + resume: bool, + resume_from: Optional[int] = None, + resume_program: bool = False, + bundle_dir: Optional[Path] = None, + ) -> Optional[str]: + """Bind the workflow's surfaces for this run, or say why not. + + A single-surface workflow binds nothing and runs exactly as before. + A surface is a name plus an identity check, never a live handle, so a + linear resume re-binds every surface by identity from the last + checkpoint's surface observations (docs/MULTI_SURFACE.md). + """ + + self._surface_router = None + if not workflow.surfaces: + if self.surface_backends: + return ( + "surface_backends were supplied, but this workflow declares " + "no surfaces; refusing to run" + ) + return None + # Governed, qualified, durable, resumed, and managed (Cloud) runs are + # admitted: the governed authorization binds the sealed bundle, whose + # workflow.json carries the declared surface map; every step result + # records the surface it ran on; the outcome classifier requires that + # record to match the contract; a qualification case observes each + # surface against its own qualified boundary; and checkpoints carry + # each surface's last-seen identity observation, never a handle. + if resume_program: + return ( + "a multi-surface workflow program cannot be resumed yet: program " + "checkpoints do not carry surface observations; refusing to run" + ) + if self._managed_dispatch_snapshot is not None: + # An unattended run must not start when a surface it will switch + # to has no target: it would stop half-way after earlier writes. + unbound = [ + spec.id + for spec in workflow.surfaces[1:] + if not spec.is_page and spec.id not in (self.surface_backends or {}) + ] + if unbound: + return ( + "a managed dispatch of a multi-surface workflow needs a " + "backend for every backend surface in the deployment " + "profile's surfaces: map; unbound: " + + ", ".join(unbound) + + "; refusing to run" + ) + resume_active: Optional[str] = None + resume_observations: dict[str, Any] = {} + if resume and resume_from: + last = None + for checkpoint in self._durable_linear_snapshot: + if getattr(checkpoint, "step_index", None) == resume_from - 1: + last = checkpoint + if last is None or getattr(last, "active_surface", None) is None: + return ( + "the checkpoint this multi-surface run resumes from carries " + "no surface observation, so its surfaces cannot be re-bound " + "by identity; refusing to run" + ) + resume_active = last.active_surface + resume_observations = dict(last.surface_observations) + authorization = self.governed_authorization + if ( + authorization is not None + and authorization.qualification_case_id is not None + ): + project = workflow.qualification + if project is None: + return ( + "a multi-surface qualification case needs the bundle's " + "qualification project; refusing to run" + ) + for spec in workflow.surfaces[1:]: + boundary = project.surface_environments.get(spec.id) + if boundary is None: + return ( + f"the qualification project has no environment boundary " + f"for surface '{spec.id}'; every surface of a " + "multi-surface workflow must be qualified; refusing to run" + ) + if boundary.target_kind != spec.kind: + return ( + f"the qualification boundary for surface '{spec.id}' is " + f"{boundary.target_kind}, but the surface is declared " + f"{spec.kind}; refusing to run" + ) + extra = sorted( + set(project.surface_environments) + - {spec.id for spec in workflow.surfaces[1:]} + ) + if extra: + return ( + "the qualification project bounds surfaces the workflow " + "does not declare: " + ", ".join(extra) + "; refusing to run" + ) + self._surface_environment_observations = {} + try: + self._surface_identity_context = (workflow, bundle_dir) + self._surface_router = SurfaceRouter( + workflow, + self.backend, + self.surface_backends, + poll_interval_s=self.poll_interval_s, + identity_check=self._surface_identity_check, + resume_active=resume_active, + observations=resume_observations, + ) + except ValueError as exc: + return f"{exc}; refusing to run" + return None + + def _surface_identity_check(self, backend: Any, step: Step) -> Optional[bool]: + """Vision-first identity check of one live target for ``step``. + + Pixels and screen text only, on the candidate's own screenshot: the + step's anchor template for an action step, the recorded screen text + (TEXT_PRESENT) of a SWITCH_SURFACE step, which was mined from the + first frame of its target. None when the step carries neither; the + router then relies on the declared structured attributes alone. + """ + + workflow, bundle_dir = getattr(self, "_surface_identity_context", (None, None)) + try: + png = backend.screenshot() + except Exception: + return False + evidence = False + if step.action is ActionKind.SWITCH_SURFACE: + for condition in step.expect: + text = condition.text + if ( + condition.kind is not PostconditionKind.TEXT_PRESENT + or not text + or "{" in text + ): + continue + evidence = True + if self.vision.find_text(png, text) is None: + return False + elif step.anchor is not None and step.anchor.template and bundle_dir: + template = self._asset_bytes( + Path(bundle_dir), step.anchor.template, workflow=workflow + ) + if template: + evidence = True + if self.vision.find_template(png, template) is None: + return False + return True if evidence else None + + def _end_surfaces(self) -> None: + """Restore the primary backend after a multi-surface run.""" + + router = self._surface_router + if router is None: + return + self._surface_router = None + if self.backend is not router.primary_backend: + self._activate_surface_backend(router.primary_backend) + # Drop the qualification input-edge guard from every other surface + # backend; the primary's guard follows the single-surface lifecycle. + for backend in router.backends.values(): + if backend is router.primary_backend: + continue + clear_guard = getattr(backend, "set_qualification_input_guard", None) + if callable(clear_guard): + try: + clear_guard(None) + except Exception: + pass + + def _activate_surface_backend(self, backend: Backend) -> None: + """Make ``backend`` current and drop every cache bound to the old one. + + Nothing observed on one surface may be reused on another: the settle + seed, the on-screen read-back verifier, the last click point, and any + armed pointer/keyboard lease all belong to the surface they came from. + """ + + self._cancel_guarded_coordinate() + self._cancel_guarded_keyboard() + self.backend = backend + self._settle_seed = None + self._actuation_epoch += 1 + # The on-screen read-back verifier is bound to one backend; drop it so + # the next step builds one for the new surface. + vars(self).pop("_readback_verifier", None) + self._last_click_point = None + self._last_click_region = None + self._control_overlay_last_observation_png = None + + def _bind_surface_qualification_environment( + self, workflow: Workflow, surface_id: str + ) -> Optional[str]: + """Observe the newly active surface against its qualified boundary. + + Only a qualification case run observes environments. The primary + surface is checked against the project's ``environment`` (as at run + start); every other surface against ``surface_environments[id]`` + (``_begin_surfaces`` already refused a case whose project lacks one). + On a match the input-edge guard moves to the new surface's backend, + so every later input on that surface rechecks that environment. + """ + + authorization = self.governed_authorization + if authorization is None or authorization.qualification_case_id is None: + return None + router = self._surface_router + project = workflow.qualification + if router is None or project is None: + return "qualification environment evidence is unavailable" + self._qualification_environment_call = None + self._qualification_environment_bound = None + boundary: Any + if surface_id == router.primary_id: + boundary = project.environment + else: + boundary = project.surface_environments.get(surface_id) + if boundary is None: + return "the qualification project does not bound this surface" + refusal, observed, ids = self._bind_qualification_boundary( + boundary, surface_scoped=surface_id != router.primary_id + ) + if surface_id != router.primary_id and observed is not None and ids is not None: + self._surface_environment_observations[surface_id] = ( + ObservedSurfaceEnvironment( + target_kind=observed.target_kind, + application_sha256=hashlib.sha256( + observed.application_identity.encode("utf-8") + ).hexdigest(), + application_version_sha256=hashlib.sha256( + observed.application_version.encode("utf-8") + ).hexdigest(), + session_sha256=observed.session_identity_sha256, + environment_digest=observed.environment_digest, + observer_id=ids[0], + observer_contract_sha256=ids[1], + binding_sha256=observed.binding_sha256( + observer_id=ids[0], observer_contract_sha256=ids[1] + ), + ) + ) + return refusal + + def _active_surface_binding( + self, workflow: Workflow + ) -> tuple[Optional[ExecutionTargetKind], Optional[str]]: + """(kind, execution mode) of the surface the run is on right now. + + Single-surface runs use the bundle-level binding, unchanged. In a + multi-surface run the remote-surface safety rules follow the ACTIVE + surface, so an RDP segment of a web-first bundle keeps them. + """ + + router = self._surface_router + if router is None: + return workflow.surface, workflow.execution_mode + from openadapt_flow.surface_selection import execution_mode_for_surface + + kind = cast(ExecutionTargetKind, router.active_kind()) + return kind, execution_mode_for_surface(kind) + + def _active_surface_is_external_remote(self, workflow: Workflow) -> bool: + kind, mode = self._active_surface_binding(workflow) + return kind in ("rdp", "citrix") and mode == "external" + + @staticmethod + def _surface_refusal_result(step: Step, error: str) -> StepResult: + return StepResult( + step_id=step.id, + intent=step.intent, + ok=False, + risk=step.risk, + risk_explanation=step.risk_explanation, + risk_review_required=step.risk_review_required, + delivery_attempted=False, + safety_halt=True, + failure_category="safety_halt", + error=error, + ) + def _run_step( self, step: Step, @@ -5968,6 +6392,180 @@ def _run_step( run_dir: Path, new_crops: dict[str, bytes], graph_ctx: Optional["_GraphStepContext"] = None, + ) -> StepResult: + """Execute one step on the surface it names; never raises. + + A single-surface run goes straight to the step pipeline. In a + multi-surface run the step is first checked against the live surface + map (no undeclared page, the active page still open, the step on the + active surface), a SWITCH_SURFACE step runs its identity check, and + after the action the pages the step declares are claimed and any other + new page halts. + """ + + router = self._surface_router + if router is None: + if step.action is ActionKind.SWITCH_SURFACE: + return self._surface_refusal_result( + step, + f"Step '{step.id}' switches surface, but this run has no " + "declared surfaces; run halted", + ) + return self._run_step_on_active_surface( + step, + workflow=workflow, + step_index=step_index, + params=params, + bundle_dir=bundle_dir, + run_dir=run_dir, + new_crops=new_crops, + graph_ctx=graph_ctx, + ) + refusal = router.entry_refusal(step) + if refusal is None and step.action is not ActionKind.SWITCH_SURFACE: + # Route the step to its surface's live target; after a resume or + # a closed page this re-binds it by the step's identity check. + refusal = router.route(step) + if refusal is None and router.active_backend is not self.backend: + self._activate_surface_backend(router.active_backend) + refusal = self._bind_surface_qualification_environment( + workflow, router.active_id + ) + if refusal is not None: + refused = self._surface_refusal_result(step, refusal) + refused.surface = router.active_id + return refused + if step.action is ActionKind.SWITCH_SURFACE: + switched = self._run_switch_surface_step( + step, workflow=workflow, bundle_dir=bundle_dir, run_dir=run_dir + ) + # The surface the run is on after the switch: the target on + # success, the unchanged surface on a refusal. + switched.surface = router.active_id + return switched + result = self._run_step_on_active_surface( + step, + workflow=workflow, + step_index=step_index, + params=params, + bundle_dir=bundle_dir, + run_dir=run_dir, + new_crops=new_crops, + graph_ctx=graph_ctx, + ) + # Receipt: the surface this step ran on. The entry check above proved + # it is the step's declared surface; the outcome classifier re-checks + # the record against the contract. + result.surface = router.active_id + if not result.ok: + return result + # A guard-skipped step delivered nothing, so it cannot have opened its + # pages; any page that did appear is still undeclared. + checked = step.model_copy(update={"opens": []}) if result.skipped else step + error = router.after_action_refusal(checked, timeout_s=step.timeout_s) + if error is not None: + result.ok = False + result.safety_halt = True + result.failure_category = "safety_halt" + result.error = error + return result + + def _run_switch_surface_step( + self, step: Step, *, workflow: Workflow, bundle_dir: Path, run_dir: Path + ) -> StepResult: + """Switch to ``step.surface`` after its identity check; never raises. + + Delivers no input. The router checks the target (declared, opened on + this run, the exact claimed page or a supplied backend of the declared + kind, the recorded origin); then the step's postconditions, mined from + the first frame of the new surface, must hold on it. No model may + rescue that screen check: it is an identity check, not drift. + """ + + router = self._surface_router + assert router is not None + t0 = time.monotonic() + evidence_step_id = self._new_step_evidence_key(run_dir, step.id) + result = StepResult( + step_id=step.id, + intent=step.intent, + ok=False, + risk=step.risk, + risk_explanation=step.risk_explanation, + risk_review_required=step.risk_review_required, + delivery_attempted=False, + ) + try: + result.before_png = self._save_step_png( + run_dir, evidence_step_id, "before", self.backend.screenshot() + ) + except Exception: + pass + backend, refusal = router.switch(step) + if refusal is not None or backend is None: + result.safety_halt = True + result.failure_category = "safety_halt" + result.error = refusal or "surface switch refused" + result.elapsed_ms = (time.monotonic() - t0) * 1000.0 + return result + self._activate_surface_backend(backend) + env_refusal = self._bind_surface_qualification_environment( + workflow, router.active_id + ) + if env_refusal is not None: + result.safety_halt = True + result.failure_category = "governed_refusal" + result.error = ( + f"Step '{step.id}' ({step.intent}): {env_refusal} on surface " + f"'{router.active_id}'; run halted" + ) + result.elapsed_ms = (time.monotonic() - t0) * 1000.0 + return result + failed: list[str] = [] + try: + start_state = self._structural_state() + frame = self._settle_seeded() + ok, frame = self._poll_postconditions(step, frame, bundle_dir, start_state) + if not ok: + frame = self.vision.wait_settled(self.backend) + failed = [ + self._describe_postcondition(pc) + for pc in step.expect + if not self._postcondition_passes( + pc, frame, bundle_dir, start_state + ) + ] + result.after_png = self._save_step_png( + run_dir, evidence_step_id, "after", frame + ) + except Exception as exc: + failed = [f"the new surface could not be observed ({type(exc).__name__})"] + result.postconditions_ok = (not failed) if step.expect else None + if failed: + result.safety_halt = True + result.failure_category = "safety_halt" + result.error = ( + f"Surface identity check failed for step '{step.id}' " + f"({step.intent}): surface '{step.surface}' does not show the " + f"recorded screen state — failed: {'; '.join(failed)} — run " + "halted" + ) + else: + result.ok = True + result.elapsed_ms = (time.monotonic() - t0) * 1000.0 + return result + + def _run_step_on_active_surface( + self, + step: Step, + *, + workflow: Workflow, + step_index: int, + params: Mapping[str, RuntimeParamScalar], + bundle_dir: Path, + run_dir: Path, + new_crops: dict[str, bytes], + graph_ctx: Optional["_GraphStepContext"] = None, ) -> StepResult: """Execute a single step; never raises (failures land in the result). @@ -8660,8 +9258,7 @@ def _step_requires_remote_identity_binding( return ( step.identity_armed is True - and workflow.surface in ("rdp", "citrix") - and workflow.execution_mode == "external" + and self._active_surface_is_external_remote(workflow) and isinstance(self.backend, RemoteActuationBackend) ) @@ -8688,10 +9285,8 @@ def _selection_contract_error( f"Step '{step.id}' ({step.intent}) has an incomplete exact " "option-selection contract; refusing input" ) - if ( - workflow.surface not in ("rdp", "citrix") - or workflow.execution_mode != "external" - or not isinstance(self.backend, RemoteActuationBackend) + if not self._active_surface_is_external_remote(workflow) or not isinstance( + self.backend, RemoteActuationBackend ): return ( f"Step '{step.id}' ({step.intent}) carries a remote " @@ -8765,8 +9360,7 @@ def _step_needs_consequential_revalidation( """ if ( step.action is ActionKind.SELECT_OPTION - and workflow.surface in ("rdp", "citrix") - and workflow.execution_mode == "external" + and self._active_surface_is_external_remote(workflow) and isinstance(self.backend, RemoteActuationBackend) ): # A selection is a composite focus -> typeahead -> commit gesture. @@ -10364,6 +10958,75 @@ def _stamp_approval(self, report: RunReport) -> None: ) def _act( + self, + step: Step, + resolution: Optional[Resolution], + params: Mapping[str, RuntimeParamScalar], + **kwargs: Any, + ) -> Optional[str]: + """Perform the step's action, answering the dialog it raises, if any. + + A step with a recorded ``dialog`` arms a one-shot answer on the + browser backend before its input edge. The dialog's message is its + identity: a different message is dismissed, never accepted, and the + step halts; so does a dialog that never appears. A backend that cannot + see browser dialogs refuses the step before any input + (docs/MULTI_SURFACE.md, "Dialogs"). Errors name a code + (``dialog_missing``, ``dialog_mismatch``), never the dialog's text: + the error lands in report.json and REPORT.md, and a live dialog can + show application data. + """ + + dialog = step.dialog + if dialog is None: + return self._act_without_dialog(step, resolution, params, **kwargs) + arm = getattr(self.backend, "arm_dialog", None) + take = getattr(self.backend, "take_dialogs", None) + if not callable(arm) or not callable(take): + return ( + f"Step '{step.id}' ({step.intent}) answers a browser dialog, but " + "the active backend cannot see browser dialogs; run aborted" + ) + self._arm_dialog(arm, dialog) + try: + error = self._act_without_dialog(step, resolution, params, **kwargs) + finally: + seen = take(0.0) + if error is not None: + return error + if not seen: + seen = self._wait_for_dialog(take, dialog) + if not seen: + return ( + f"Step '{step.id}' ({step.intent}): the recorded browser dialog " + "did not appear (dialog_missing); run aborted" + ) + mismatched = [entry for entry in seen if not entry.get("matched")] + if mismatched: + return ( + f"Step '{step.id}' ({step.intent}): a browser dialog did not " + "show the recorded message (dialog_mismatch); it was " + "dismissed, not answered; run aborted" + ) + return None + + @staticmethod + def _arm_dialog(arm: Callable[..., Any], dialog: Any) -> None: + if dialog.message_pattern is not None: + arm(dialog.response, None, message_pattern=dialog.message_pattern) + else: + arm(dialog.response, dialog.message) + + def _wait_for_dialog(self, take: Callable[..., Any], dialog: Any) -> list[Any]: + """Wait briefly for an armed dialog that opens after the input edge.""" + + arm = getattr(self.backend, "arm_dialog", None) + if callable(arm): + self._arm_dialog(arm, dialog) + seen = take(_DIALOG_WAIT_S) + return list(seen) if isinstance(seen, list) else [] + + def _act_without_dialog( self, step: Step, resolution: Optional[Resolution], @@ -11469,56 +12132,120 @@ def _observe_qualification_environment( return "qualification run identity does not match its authorization" if execution_target_kind != project.environment.target_kind: return "qualification target kind does not match the project" + refusal, observed, observer_ids = self._bind_qualification_boundary( + project.environment + ) + if observed is not None and observer_ids is not None: + report.observed_application_sha256 = hashlib.sha256( + observed.application_identity.encode("utf-8") + ).hexdigest() + report.observed_application_version_sha256 = hashlib.sha256( + observed.application_version.encode("utf-8") + ).hexdigest() + report.observed_session_sha256 = observed.session_identity_sha256 + report.observed_environment_digest = observed.environment_digest + report.qualification_environment_observer_id = observer_ids[0] + report.qualification_environment_observer_contract_sha256 = observer_ids[1] + report.observed_environment_binding_sha256 = observed.binding_sha256( + observer_id=observer_ids[0], + observer_contract_sha256=observer_ids[1], + ) + return refusal + + def _bind_qualification_boundary( + self, boundary: Any, *, surface_scoped: bool = False + ) -> tuple[ + Optional[str], + Optional[QualificationEnvironmentObservation], + Optional[tuple[str, str]], + ]: + """Observe ``self.backend`` against one qualified environment boundary. + + Returns ``(refusal, observation, (observer_id, observer_contract))``. + The observation is returned whenever one was taken, even when it does + not match, so the caller can retain it. On a match the input-edge + environment guard is bound to ``self.backend``. A multi-surface + qualification case calls this once for the primary surface (the + project's ``environment``) and again on every surface switch, with that + surface's boundary (``QualificationProject.surface_environments``). + """ configured_observer = self.qualification_environment_observer + if ( + surface_scoped + and configured_observer is not None + and boundary.environment_observer_id is not None + ): + try: + configured_id = configured_observer.observer_id + except Exception: + configured_id = None + if configured_id != boundary.environment_observer_id: + # A run has one configured observer. A surface whose boundary + # names another observer uses that observer's own route below. + configured_observer = None if configured_observer is None: from openadapt_flow.qualification_web_session import ( SIGNED_IN_WEB_SESSION_OBSERVER_ID, SignedInWebSessionObserver, ) - if ( - project.environment.environment_observer_id - == SIGNED_IN_WEB_SESSION_OBSERVER_ID - ): + if boundary.environment_observer_id == SIGNED_IN_WEB_SESSION_OBSERVER_ID: # Real web apps carry no OpenAdapt markers: identify them by # origin and signed-in session, with the version the operator # confirmed in Check it. The contract and environment digests # below still have to match the project exactly. try: configured_observer = SignedInWebSessionObserver( - project.environment.application_version + boundary.application_version ) except ValueError: - return "qualification confirmed application version is invalid" + return ( + "qualification confirmed application version is invalid", + None, + None, + ) if configured_observer is None: if not callable( getattr(self.backend, "qualification_environment_identity", None) ): return ( - "qualification backend cannot observe the complete environment; " - "configure the project's environment observer" + "qualification backend cannot observe the complete " + "environment; configure the project's environment observer", + None, + None, ) configured_observer = BackendQualificationEnvironmentObserver(self.backend) + observer = configured_observer expected_observer = ( - project.environment.environment_observer_id, - project.environment.environment_observer_contract_sha256, + boundary.environment_observer_id, + boundary.environment_observer_contract_sha256, ) try: observed_observer = ( - configured_observer.observer_id, - configured_observer.contract_sha256, + observer.observer_id, + observer.contract_sha256, ) except Exception: - return "qualification environment observer identity is unavailable" + return ( + "qualification environment observer identity is unavailable", + None, + None, + ) if expected_observer != observed_observer: - return "qualification environment observer does not match the project" + return ( + "qualification environment observer does not match the project", + None, + None, + ) + backend = self.backend + target_kind = boundary.target_kind def environment_call() -> QualificationEnvironmentObservation: try: before_observer = ( - configured_observer.observer_id, - configured_observer.contract_sha256, + observer.observer_id, + observer.contract_sha256, ) except Exception as exc: raise StructuralResolutionRefused( @@ -11528,14 +12255,11 @@ def environment_call() -> QualificationEnvironmentObservation: raise StructuralResolutionRefused( "qualification environment observer binding changed before input" ) - current = configured_observer.observe( - self.backend, - project.environment.target_kind, - ) + current = observer.observe(backend, target_kind) try: after_observer = ( - configured_observer.observer_id, - configured_observer.contract_sha256, + observer.observer_id, + observer.contract_sha256, ) except Exception as exc: raise StructuralResolutionRefused( @@ -11550,40 +12274,38 @@ def environment_call() -> QualificationEnvironmentObservation: try: observed = environment_call() except Exception: - return "qualification environment observation failed" - application = observed.application_identity - application_version = observed.application_version - if observed.target_kind != project.environment.target_kind: - return "qualification environment observer returned the wrong surface" - session = observed.session_identity_sha256 - report.observed_application_sha256 = hashlib.sha256( - application.encode("utf-8") - ).hexdigest() - report.observed_application_version_sha256 = hashlib.sha256( - application_version.encode("utf-8") - ).hexdigest() - report.observed_session_sha256 = session - report.observed_environment_digest = observed.environment_digest - report.qualification_environment_observer_id = observed_observer[0] - report.qualification_environment_observer_contract_sha256 = observed_observer[1] - report.observed_environment_binding_sha256 = observed.binding_sha256( - observer_id=observed_observer[0], - observer_contract_sha256=observed_observer[1], - ) - if project.environment.expected_application_identity is None: + return "qualification environment observation failed", None, None + if observed.target_kind != boundary.target_kind: + return ( + "qualification environment observer returned the wrong surface", + None, + None, + ) + ids = (observed_observer[0], observed_observer[1]) + if boundary.expected_application_identity is None: return ( "qualification project has no executable application identity; " - "set application_identity separately from the display name" + "set application_identity separately from the display name", + observed, + ids, ) - if application != project.environment.expected_application_identity: - return "observed qualification application does not match the project" - if application_version != project.environment.application_version: + if observed.application_identity != boundary.expected_application_identity: return ( - "observed qualification application version does not match the project" + "observed qualification application does not match the project", + observed, + ids, ) - if observed.environment_digest != project.environment.environment_digest: + if observed.application_version != boundary.application_version: return ( - "observed qualification environment digest does not match the project" + "observed qualification application version does not match the project", + observed, + ids, + ) + if observed.environment_digest != boundary.environment_digest: + return ( + "observed qualification environment digest does not match the project", + observed, + ids, ) self._qualification_environment_call = environment_call # The observer owns ``observed`` and can legally reuse or mutate that @@ -11595,9 +12317,13 @@ def environment_call() -> QualificationEnvironmentObservation: if not callable(set_guard): self._qualification_environment_call = None self._qualification_environment_bound = None - return "qualification backend cannot bind an input-edge environment guard" + return ( + "qualification backend cannot bind an input-edge environment guard", + observed, + ids, + ) set_guard(self._require_qualification_environment_current) - return None + return None, observed, ids def _require_qualification_environment_current(self) -> None: """Recheck the bound environment immediately before an input edge.""" @@ -14241,12 +14967,23 @@ def _poll_postconditions( """First pass: poll each postcondition until pass or timeout.""" for pc in step.expect: deadline = time.monotonic() + pc.timeout_s + kind = pc.kind.value if hasattr(pc.kind, "value") else pc.kind while True: if self._postcondition_passes(pc, frame_png, bundle_dir, start_state): break if time.monotonic() >= deadline: return False, frame_png - time.sleep(self.poll_interval_s) + wait_for_page = getattr(self.backend, "wait_for_page_count_above", None) + start_pages = (start_state or {}).get("page_count") + if ( + kind == "new_tab_opened" + and callable(wait_for_page) + and isinstance(start_pages, int) + ): + # Wait on the browser's page event, not a poll. + wait_for_page(start_pages, deadline - time.monotonic()) + else: + time.sleep(self.poll_interval_s) frame_png = self.backend.screenshot() return True, frame_png @@ -14362,7 +15099,13 @@ def _postcondition_template(self, pc: Any, bundle_dir: Path) -> Optional[bytes]: def _layout_drift_on(self, workflow: Workflow) -> bool: if self.layout_drift is not None: return self.layout_drift - kind = self._layout_target_kind or workflow.surface + kind: Optional[str] + if self._surface_router is not None: + # Multi-surface run: follow the surface the step acts on, so an + # RDP segment of a web-first bundle keeps the check. + kind, _ = self._active_surface_binding(workflow) + else: + kind = self._layout_target_kind or workflow.surface if kind in ("rdp", "citrix"): return True return type(self.backend).__name__ == "FreeRDPBackend" diff --git a/packages/flow/openadapt_flow/runtime/surfaces.py b/packages/flow/openadapt_flow/runtime/surfaces.py new file mode 100644 index 000000000..68f1d3089 --- /dev/null +++ b/packages/flow/openadapt_flow/runtime/surfaces.py @@ -0,0 +1,754 @@ +"""Run-time surface routing for multi-surface workflows. + +A multi-surface workflow (``Workflow.surfaces``, docs/MULTI_SURFACE.md) runs on +several surfaces: browser tabs and popups its own steps open, and other +backends the caller supplies. :class:`SurfaceRouter` owns the map from surface +id to live backend and answers three questions for the replayer, all without a +model call: + +- may this step run now (``entry_refusal``)? It refuses when the browser holds + a page no step declared, when the active page has closed, or when the step + names a surface other than the active one; +- did the step open exactly the pages it declared (``after_action_refusal``)? + Each new page is claimed in creation order and must match its recorded + opener relation, origin, URL pattern, and title pattern; +- may execution switch to this surface (``switch``)? The target must exist on + this run, be the exact page claimed earlier (or a supplied backend of the + declared kind), and show the origin the demonstration saw. + +A surface is a name plus an identity check, never a live handle. The router +caches the page or backend it last bound for each surface. When that binding +is missing it binds again by identity: the one live target that passes the +check. Zero or several matches halt. A page surface rebinds only to a page +claimed on this run; only a resumed run, which claimed nothing yet, may bind a +page that was already open, and every rebind needs visual evidence. A step +never runs on a page that closed under it. The check is vision-first (the +step's anchor template and recorded screen text on the candidate's own +screenshot), with structured attributes (origin, URL and title patterns, +opener relation, the last-seen origin) added only where the target exposes +them. + +Every refusal is a halt. The router never picks a window for the run, and its +messages never quote a URL or a title (they can carry application content). +""" + +from __future__ import annotations + +import re +import time +from typing import Any, Callable, Mapping, Optional +from urllib.parse import urlsplit + +from openadapt_flow.ir import ( + ActionKind, + Step, + SurfaceObservation, + SurfaceSpec, + Workflow, +) + +#: ``identity_check(backend, step)``: True when ``backend``'s live screen shows +#: the step's recorded visual identity, False when it does not, None when the +#: step carries no visual identity evidence at all. +IdentityCheck = Callable[[Any, Step], Optional[bool]] + +#: Backend class name -> the surface kinds it can serve. Read through the MRO, +#: most specific class first. A backend may instead declare ``surface_kind``. +_CLASS_SURFACE_KINDS: dict[str, frozenset[str]] = { + "PlaywrightBackend": frozenset({"web"}), + "WindowsBackend": frozenset({"windows"}), + "MacOSBackend": frozenset({"macos"}), + "LinuxBackend": frozenset({"linux"}), + "FreeRDPBackend": frozenset({"rdp"}), + "CitrixWorkspaceBackend": frozenset({"citrix"}), + # The generic local client-window backend serves both RDP and Citrix. + "RemoteDisplayBackend": frozenset({"rdp", "citrix"}), +} + + +def backend_surface_kinds(backend: Any) -> frozenset[str]: + """Surface kinds ``backend`` can serve; empty when it cannot be told.""" + + declared = getattr(backend, "surface_kind", None) + if isinstance(declared, str) and declared: + return frozenset({declared}) + for cls in type(backend).__mro__: + kinds = _CLASS_SURFACE_KINDS.get(cls.__name__) + if kinds: + return kinds + return frozenset() + + +def url_origin(url: Optional[str]) -> Optional[str]: + """``scheme://host[:port]`` of an http(s) URL, lower-cased; else None.""" + + if not isinstance(url, str) or not url: + return None + try: + parts = urlsplit(url) + except ValueError: + return None + if parts.scheme not in ("http", "https") or not parts.netloc: + return None + return f"{parts.scheme}://{parts.netloc}".lower() + + +def _browser_page(backend: Any) -> Any: + """The Playwright page a browser backend wraps, or None.""" + + page = getattr(backend, "page", None) + if page is None or not hasattr(page, "context"): + return None + return page + + +def page_has_window_opener(page: Any) -> Optional[bool]: + """The page's own ``window.opener`` relation; None if unreadable. + + ``window.open`` keeps the opener; a ``target=_blank`` link (implicit + ``noopener``) or ``rel=noopener`` clears it. The browser still reports the + page that created it (``page.opener()``), which is checked separately. + """ + + try: + return bool(page.evaluate("() => window.opener !== null")) + except Exception: + return None + + +def _page_closed(page: Any) -> bool: + try: + return bool(page.is_closed()) + except Exception: + return True + + +def _default_page_backend(page: Any) -> Any: + from openadapt_flow.backends.playwright_backend import PlaywrightBackend + + return PlaywrightBackend(page, closed_page_frame=True) + + +def _pump_wait(page: Any, seconds: float) -> None: + """Wait ``seconds`` while Playwright keeps dispatching browser events. + + ``page.wait_for_timeout`` runs the sync API's event loop; ``time.sleep`` + would freeze it. Objects without it (test doubles, closed pages) sleep. + """ + + wait = getattr(page, "wait_for_timeout", None) if page is not None else None + if callable(wait) and not _page_closed(page): + try: + wait(max(1.0, seconds * 1000.0)) + return + except Exception: + pass + time.sleep(seconds) + + +class SurfaceRouter: + """Live surface map for one run of a multi-surface workflow.""" + + def __init__( + self, + workflow: Workflow, + primary_backend: Any, + supplied_backends: Optional[Mapping[str, Any]] = None, + *, + page_backend_factory: Callable[[Any], Any] = _default_page_backend, + poll_interval_s: float = 0.05, + identity_check: Optional[IdentityCheck] = None, + resume_active: Optional[str] = None, + observations: Optional[Mapping[str, SurfaceObservation]] = None, + ) -> None: + if not workflow.surfaces: + raise ValueError("SurfaceRouter needs a workflow that declares surfaces") + self.specs: dict[str, SurfaceSpec] = {s.id: s for s in workflow.surfaces} + self.primary_id = workflow.surfaces[0].id + self.primary_backend = primary_backend + self.active_id = self.primary_id + self._supplied = dict(supplied_backends or {}) + unknown = sorted(set(self._supplied) - set(self.specs)) + if unknown: + raise ValueError( + "surface_backends names surfaces the workflow does not declare: " + + ", ".join(unknown) + ) + for sid in self._supplied: + if self.specs[sid].is_page or sid == self.primary_id: + raise ValueError( + f"surface {sid!r} is not a backend surface; only backend " + "surfaces are supplied by the caller" + ) + self._page_backend_factory = page_backend_factory + self._poll_interval_s = max(0.01, float(poll_interval_s)) + self._identity_check = identity_check + # Last-seen identity observation per surface (checkpointed; a resume + # seeds it from the last checkpoint). + self._observations: dict[str, SurfaceObservation] = { + sid: obs for sid, obs in (observations or {}).items() if sid in self.specs + } + if resume_active is not None: + if resume_active not in self.specs: + raise ValueError( + f"the checkpoint's active surface {resume_active!r} is not " + "declared by this workflow" + ) + self.active_id = resume_active + self.backends: dict[str, Any] = {self.primary_id: primary_backend} + # Pages claimed for page surfaces, by surface id. + self._claimed: dict[str, Any] = {} + # Every page claimed or bound on this run, and the surfaces bound so + # far. A page surface rebinds only to one of these pages, except on a + # resume before that surface is first bound: the pages the earlier + # run opened are then only in the starting inventory. + self._claimed_on_run: list[Any] = [] + self._bound_on_run: set[str] = set() + self._resuming = resume_active is not None + # Every page that is accounted for: the starting inventory of each + # watched browser context, plus every claimed page. + self._known_pages: list[Any] = [] + self._contexts: list[Any] = [] + self._watch(primary_backend) + + # -- inventory -------------------------------------------------------- + + def _watch(self, backend: Any) -> None: + """Start watching the browser context of a browser backend.""" + + page = _browser_page(backend) + if page is None: + return + try: + context = page.context + except Exception: + return + if any(context is known for known in self._contexts): + self._remember(page) + return + self._contexts.append(context) + try: + pages = list(context.pages) + except Exception: + pages = [page] + for existing in pages: + self._remember(existing) + self._remember(page) + + def _wait_for_new_page(self, remaining_s: float) -> None: + """Block until a watched context opens a page, or ``remaining_s``.""" + + remaining_s = max(0.0, remaining_s) + if remaining_s <= 0: + return + contexts = list(self._contexts) + if len(contexts) == 1: + wait = getattr(contexts[0], "wait_for_event", None) + if callable(wait): + try: + wait("page", timeout=max(1.0, remaining_s * 1000.0)) + except Exception: + # A timeout (or a closed context): the caller re-reads + # the inventory and decides. + pass + return + # Several contexts, or a test double: wait one interval through a + # live page so Playwright still dispatches, then let the caller look. + live = next( + (p for p in self._known_pages if not _page_closed(p)), + None, + ) + _pump_wait(live, min(self._poll_interval_s, remaining_s)) + + def _remember(self, page: Any) -> None: + if not any(page is known for known in self._known_pages): + self._known_pages.append(page) + + def undeclared_pages(self) -> list[Any]: + """Open pages, in creation order, that no surface accounts for.""" + + found: list[Any] = [] + for context in self._contexts: + try: + pages = list(context.pages) + except Exception: + continue + for page in pages: + if _page_closed(page): + continue + if any(page is known for known in self._known_pages): + continue + if any(page is seen for seen in found): + continue + found.append(page) + return found + + @property + def active_backend(self) -> Any: + return self.backends.get(self.active_id) + + @property + def observations(self) -> dict[str, SurfaceObservation]: + """The last-seen identity observation of every surface seen so far.""" + + return dict(self._observations) + + def observe_active(self, after_step: Optional[str] = None) -> None: + """Record the active surface's identity observation (no handle).""" + + backend = self.active_backend + if backend is None: + return + page = _browser_page(backend) + origin = None + if page is not None and not _page_closed(page): + try: + origin = url_origin(page.url) + except Exception: + origin = None + phash = None + try: + from io import BytesIO + + from PIL import Image + + from openadapt_flow.image_hash import perceptual_hash + + with Image.open(BytesIO(backend.screenshot())) as image: + phash = perceptual_hash(image.convert("RGB")) + except Exception: + phash = None + self._observations[self.active_id] = SurfaceObservation( + surface=self.active_id, + kind=self.specs[self.active_id].kind, + observed_after_step=after_step, + origin=origin, + screen_phash=phash, + ) + + def active_kind(self) -> str: + return self.specs[self.active_id].kind + + # -- per-step checks ---------------------------------------------------- + + def _undeclared_refusal(self, step: Step, pages: list[Any], when: str) -> str: + return ( + f"Step '{step.id}' ({step.intent}): {len(pages)} browser page(s) " + f"opened {when} that no step of this workflow declares (an " + "unexpected popup, tab, or window); Flow does not choose a window " + f"by itself, so the run halted on surface '{self.active_id}'" + ) + + def entry_refusal(self, step: Step) -> Optional[str]: + """Why ``step`` may not start now, or None.""" + + pages = self.undeclared_pages() + if pages: + return self._undeclared_refusal(step, pages, "before this step") + if step.action is ActionKind.SWITCH_SURFACE: + # Leaving a page that closed itself (an OAuth or approval popup) + # is allowed: the switch binds its target by identity and never + # returns anywhere on its own. + return None + active_page = _browser_page(self.active_backend) + if active_page is not None and _page_closed(active_page): + # Whatever its kind: a page surface is bound to the page this run + # claimed, and only an explicit switch_surface step moves on. + return ( + f"Step '{step.id}' ({step.intent}): the active surface " + f"'{self.active_id}' was closed; refusing to act on another " + "window" + ) + wanted = step.surface or self.primary_id + if wanted != self.active_id: + return ( + f"Step '{step.id}' ({step.intent}) runs on surface '{wanted}', " + f"but the active surface is '{self.active_id}'; a surface " + "change must be an explicit switch_surface step" + ) + return None + + def after_action_refusal(self, step: Step, *, timeout_s: float) -> Optional[str]: + """Claim the pages ``step`` declared; refuse anything else.""" + + expected = list(step.opens) + deadline = time.monotonic() + max(0.0, float(timeout_s)) + pages = self.undeclared_pages() + while len(pages) < len(expected) and time.monotonic() < deadline: + # Wait on the browser's own "page" event, not a sleep: with the + # sync Playwright API a sleep dispatches nothing, so a popup the + # click opened stays invisible to ``context.pages`` until the + # next Playwright call, and a loaded machine misses it. + self._wait_for_new_page(deadline - time.monotonic()) + pages = self.undeclared_pages() + if not expected: + if pages: + return self._undeclared_refusal(step, pages, "during this step") + return None + if len(pages) > len(expected): + return ( + f"Step '{step.id}' ({step.intent}) opened {len(pages)} browser " + f"page(s) but the workflow declares {len(expected)}; Flow does " + "not choose a window by itself, so the run halted" + ) + if len(pages) < len(expected): + return ( + f"Step '{step.id}' ({step.intent}) was expected to open " + f"{len(expected)} browser page(s) ({', '.join(expected)}) but " + f"{len(pages)} opened within {timeout_s:g}s; run halted" + ) + for page, sid in zip(pages, expected): + error = self._claim(step, page, self.specs[sid], deadline) + if error is not None: + return error + return None + + def _claim( + self, step: Step, page: Any, spec: SurfaceSpec, deadline: float + ) -> Optional[str]: + """Bind ``page`` to ``spec`` after checking that it is that page.""" + + prefix = ( + f"Step '{step.id}' ({step.intent}) opened a page for surface '{spec.id}'" + ) + opener_page = _browser_page(self.backends.get(spec.opener or "", None)) + try: + creator = page.opener() + except Exception: + creator = None + if creator is not None and creator is not opener_page: + return ( + f"{prefix}, but the browser reports it was opened by a page " + f"other than surface '{spec.opener}'; run halted" + ) + assert spec.url_pattern is not None + url_re = re.compile(spec.url_pattern) + title_re = ( + re.compile(spec.title_pattern) if spec.title_pattern is not None else None + ) + url_ok = origin_ok = title_ok = False + while True: + try: + url = page.url + except Exception: + url = "" + origin_ok = url_origin(url) == spec.origin + url_ok = bool(url_re.fullmatch(url or "")) + if title_re is None: + title_ok = True + else: + try: + title = page.title() + except Exception: + title = "" + title_ok = bool(title_re.fullmatch(title or "")) + if (origin_ok and url_ok and title_ok) or time.monotonic() >= deadline: + break + # The page's URL and title update only as Playwright dispatches + # its events; wait through the page so they do. + _pump_wait(page, self._poll_interval_s) + if not origin_ok: + return ( + f"{prefix}, but it is on a different origin than recorded; run halted" + ) + if not url_ok: + return ( + f"{prefix}, but its URL does not match the recorded pattern; run halted" + ) + if not title_ok: + return ( + f"{prefix}, but its title does not match the recorded pattern; " + "run halted" + ) + link = page_has_window_opener(page) + expected_link = spec.opener_link == "window_opener" + if link is None or link != expected_link: + return ( + f"{prefix}, but its window.opener relation differs from the " + "recording; run halted" + ) + self._bind(spec.id, page) + return None + + def _bind(self, sid: str, page: Any) -> None: + self._claimed[sid] = page + self._bound_on_run.add(sid) + if not any(page is known for known in self._claimed_on_run): + self._claimed_on_run.append(page) + self._remember(page) + self.backends[sid] = self._page_backend_factory(page) + + # -- binding by identity ------------------------------------------------- + + def _bound_pages(self, *, except_id: Optional[str] = None) -> list[Any]: + bound = [_browser_page(self.primary_backend)] + bound.extend(page for sid, page in self._claimed.items() if sid != except_id) + return [page for page in bound if page is not None] + + def _unbound_pages(self, sid: str) -> list[Any]: + """Live candidate pages for ``sid`` that no other surface holds. + + Pages claimed on this run only. On a resume, before ``sid`` is first + bound, every page Flow knows (the starting inventory) instead. + """ + + held = self._bound_pages(except_id=sid) + cached = self._claimed.get(sid) + if self._resuming and sid not in self._bound_on_run: + pool = self._known_pages + else: + pool = self._claimed_on_run + found: list[Any] = [] + for context in self._contexts: + try: + pages = list(context.pages) + except Exception: + continue + for page in pages: + if _page_closed(page) or page is cached: + continue + if not any(page is known for known in pool): + # Undeclared (the entry check halts on it), or a page + # this run never claimed, such as the person's own tab. + continue + if any(page is other for other in held): + continue + if any(page is seen for seen in found): + continue + found.append(page) + return found + + def _structured_ok(self, page: Any, spec: SurfaceSpec) -> bool: + """Structured attributes, used only where the page exposes them.""" + + try: + url = page.url or "" + except Exception: + url = "" + if spec.origin is not None and url_origin(url) != spec.origin: + return False + if spec.url_pattern is not None and not re.fullmatch(spec.url_pattern, url): + return False + if spec.title_pattern is not None: + try: + title = page.title() or "" + except Exception: + title = "" + if not re.fullmatch(spec.title_pattern, title): + return False + if spec.opener_link is not None: + link = page_has_window_opener(page) + if link is not None and link != (spec.opener_link == "window_opener"): + return False + seen = self._observations.get(spec.id) + if seen is not None and seen.origin is not None: + live = url_origin(url) + if live is not None and live != seen.origin: + return False + return True + + def _page_passes( + self, page: Any, spec: SurfaceSpec, step: Step, *, require_visual: bool + ) -> bool: + """Vision-first identity check of one candidate page for ``step``. + + The structured attributes are evaluated first only because they are + cheap and need no screenshot; a candidate must pass both. With + ``require_visual`` (binding a page other than the one this surface + holds) the step must carry visual evidence and the page must show it; + otherwise a step without visual evidence relies on the structured + attributes alone. + """ + + if not self._structured_ok(page, spec): + return False + if self._identity_check is None: + return not require_visual + try: + visual = self._identity_check(self._page_backend_factory(page), step) + except Exception: + return False + if require_visual: + return visual is True + return visual is not False + + def _bind_page( + self, sid: str, step: Step, prefix: str + ) -> tuple[Optional[Any], Optional[str]]: + """Bind page surface ``sid`` to the one live page that passes ``step``. + + The page this surface last used is kept while it is open and passes + (or when no other candidate exists: the step's own identity ladder + then checks it, as on any single page). Otherwise exactly one other + candidate (:meth:`_unbound_pages`) must show the step's visual + evidence; zero or several halt, and so does a held page that fails + its check. + """ + + spec = self.specs[sid] + cached = self._claimed.get(sid) + cached_live = cached is not None and not _page_closed(cached) + others = self._unbound_pages(sid) + if cached_live and not others: + return cached, None + if cached_live and self._page_passes(cached, spec, step, require_visual=False): + return cached, None + passing = [ + page + for page in others + if self._page_passes(page, spec, step, require_visual=True) + ] + if len(passing) == 1: + page = passing[0] + self._bind(sid, page) + return page, None + if len(passing) > 1: + return None, ( + f"{prefix}: {len(passing)} open browser pages pass the identity " + f"check of surface '{sid}'; Flow does not choose between " + "them, so the run halted" + ) + if cached_live: + return None, ( + f"{prefix}: the page surface '{sid}' holds no longer passes its " + "identity check, and no other page this run opened does; run " + "halted" + ) + return None, ( + f"{prefix}: no open browser page passes the identity check of " + f"surface '{sid}' (rebinding needs this step's visual evidence on " + "a page this run opened); run halted" + ) + + def _bind_backend( + self, sid: str, step: Step, prefix: str + ) -> tuple[Optional[Any], Optional[str]]: + """Bind backend surface ``sid``: the supplied target must pass.""" + + spec = self.specs[sid] + backend = self.backends.get(sid) or self._supplied.get(sid) + if backend is None: + return None, ( + f"{prefix}: no backend was supplied for surface '{sid}' " + f"(kind {spec.kind}); bind one with --surface {sid}=KIND[:TARGET] or " + "the config's surfaces: map; run halted" + ) + kinds = backend_surface_kinds(backend) + if spec.kind not in kinds: + told = ", ".join(sorted(kinds)) or "unknown" + return None, ( + f"{prefix}: surface '{sid}' is declared {spec.kind}, but the " + f"supplied backend serves {told}; run halted" + ) + return backend, None + + def route(self, step: Step) -> Optional[str]: + """Route a non-switch step to the live target of its surface. + + Called after :meth:`entry_refusal`. In steady state this is the page + or backend the surface already uses. After a resume (no binding yet) + or when the surface's page closed, the target is bound again by the + step's own identity check: exactly one match, or a halt. + """ + + sid = self.active_id + spec = self.specs[sid] + prefix = f"Step '{step.id}' ({step.intent})" + if sid == self.primary_id: + return None + if spec.is_page: + page, error = self._bind_page(sid, step, prefix) + return error + if sid in self.backends: + return None + backend, error = self._bind_backend(sid, step, prefix) + if error is not None: + return error + visual = None + if self._identity_check is not None: + try: + visual = self._identity_check(backend, step) + except Exception: + visual = False + if visual is not True: + return ( + f"{prefix}: surface '{sid}' could not be re-bound: its live " + "target does not show this step's recorded identity" + + ( + " (the step carries no visual identity evidence)" + if visual is None + else "" + ) + + "; run halted" + ) + self.backends[sid] = backend + self._watch(backend) + return None + + # -- switching ----------------------------------------------------------- + + def switch(self, step: Step) -> tuple[Optional[Any], Optional[str]]: + """Make ``step.surface`` active after its identity check. + + Returns ``(backend, None)`` on success and ``(None, reason)`` on a + refusal; the active surface is unchanged on a refusal. + """ + + sid = step.surface + prefix = f"Step '{step.id}' ({step.intent})" + if sid is None or sid not in self.specs: + return None, f"{prefix} switches to an undeclared surface; run halted" + if sid == self.active_id: + return None, f"{prefix} switches to the surface that is already active" + if not step.expect and step.surface_origin is None: + # Every target kind: without recorded screen text or an origin, + # nothing would show the right page or application is in front. + return None, ( + f"{prefix}: the switch to surface '{sid}' has no identity " + "evidence (no recorded screen text or origin), so nothing " + "would show the right application is in front; run halted" + ) + spec = self.specs[sid] + backend: Any + if sid == self.primary_id: + backend = self.primary_backend + primary_page = _browser_page(backend) + if primary_page is not None and _page_closed(primary_page): + return None, f"{prefix}: surface '{sid}' was closed; run halted" + elif spec.is_page: + cached = self._claimed.get(sid) + if cached is None and not self._unbound_pages(sid): + return None, ( + f"{prefix}: surface '{sid}' was not opened on this run; run halted" + ) + page, error = self._bind_page(sid, step, prefix) + if error is not None: + return None, error + if page is None or _page_closed(page): + return None, f"{prefix}: surface '{sid}' was closed; run halted" + backend = self.backends[sid] + else: + bound, error = self._bind_backend(sid, step, prefix) + if error is not None: + return None, error + backend = bound + self.backends[sid] = backend + self._watch(backend) + page = _browser_page(backend) + if step.surface_origin is not None: + live_origin = url_origin(getattr(page, "url", None)) if page else None + if live_origin != step.surface_origin.lower(): + return None, ( + f"{prefix}: surface '{sid}' is not on the origin the " + "demonstration switched to; run halted" + ) + if page is not None: + try: + page.bring_to_front() + except Exception as exc: + return None, ( + f"{prefix}: surface '{sid}' could not be brought to the " + f"front ({type(exc).__name__}); run halted" + ) + self.active_id = sid + return backend, None diff --git a/packages/flow/openadapt_flow/surface_bindings.py b/packages/flow/openadapt_flow/surface_bindings.py new file mode 100644 index 000000000..2f914f529 --- /dev/null +++ b/packages/flow/openadapt_flow/surface_bindings.py @@ -0,0 +1,177 @@ +"""Bind live backends to a bundle's backend surfaces at replay time. + +A multi-surface bundle (docs/MULTI_SURFACE.md) declares its surfaces. Browser +tabs and popups its own steps open need nothing from the caller. A BACKEND +surface (a native app, an RDP/Citrix window, a second web app) is supplied by +the caller, keyed by surface id. This module is the one generic way the CLI +does that, with no per-workflow code: + +- ``--surface NAME=SPEC`` (repeatable), where ``SPEC`` is ``KIND``, + ``KIND:TARGET``, or just ``TARGET``. The kind defaults to the kind the + bundle declares for ``NAME``; a different kind is refused. ``TARGET`` fills + the kind's one primary target field (see :data:`TARGET_FIELD`); +- ``surfaces:`` in a deployment config: a map from surface id to a full + ``backend`` section, for any field the one-token target cannot carry + (tokens, window titles, readiness text, markers). + +A flag is applied over the config entry for the same surface. Nothing is +inferred: a backend surface with neither stays unbound, and the run halts at +the switch to it with a reason that names the flag. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any, Callable, Mapping, Optional, Sequence + +if TYPE_CHECKING: # pragma: no cover + from openadapt_flow.deployment import BackendConfig + from openadapt_flow.ir import Workflow + +#: Surface kind -> the ``BackendConfig`` field one ``TARGET`` token fills. +TARGET_FIELD: dict[str, str] = { + "web": "url", + "windows": "agent_url", + "macos": "macos_app", + "linux": "linux_app", + "rdp": "rdp_host", + "citrix": "rdp_window", +} + +_KINDS = frozenset(TARGET_FIELD) + + +class SurfaceBindingError(ValueError): + """A ``--surface`` flag or ``surfaces:`` entry cannot be bound.""" + + +def parse_surface_flag(text: str) -> tuple[str, Optional[str], Optional[str]]: + """Split ``NAME=SPEC`` into ``(name, kind or None, target or None)``. + + ``SPEC`` is ``KIND``, ``KIND:TARGET``, or ``TARGET`` (kind from the + bundle). A target that itself contains a colon (``http://host:5001``) is + read as a target unless the text before its first colon is a surface + kind. + """ + + name, sep, spec = text.partition("=") + name = name.strip() + spec = spec.strip() + if not sep or not name or not spec: + raise SurfaceBindingError( + f"--surface expects NAME=KIND[:TARGET] or NAME=TARGET, got {text!r}" + ) + if spec.lower() in _KINDS: + return name, spec.lower(), None + head, colon, rest = spec.partition(":") + if colon and head.lower() in _KINDS: + return name, head.lower(), (rest.strip() or None) + return name, None, spec + + +def resolve_surface_configs( + workflow: "Workflow", + flags: Sequence[str] = (), + configured: Optional[Mapping[str, "BackendConfig"]] = None, +) -> dict[str, "BackendConfig"]: + """Resolve one ``BackendConfig`` per bound backend surface. + + Refuses a name the bundle does not declare, a primary or page surface (the + primary is the run's own ``--backend``; pages are opened by steps), and a + kind other than the declared one. + """ + + from openadapt_flow.deployment import BackendConfig + + specs = {spec.id: spec for spec in workflow.surfaces} + primary = workflow.surfaces[0].id if workflow.surfaces else None + + def _check(name: str) -> Any: + spec = specs.get(name) + if spec is None: + declared = ", ".join(s.id for s in workflow.surfaces[1:] if not s.is_page) + raise SurfaceBindingError( + f"surface {name!r} is not declared by this bundle" + + (f" (backend surfaces: {declared})" if declared else "") + ) + if name == primary: + raise SurfaceBindingError( + f"surface {name!r} is the primary surface; select its backend " + "with --backend / backend: instead" + ) + if spec.is_page: + raise SurfaceBindingError( + f"surface {name!r} is a browser page a step opens; it needs no binding" + ) + return spec + + resolved: dict[str, BackendConfig] = {} + for name, cfg in (configured or {}).items(): + spec = _check(name) + cfg = BackendConfig.model_validate(cfg) + if "kind" not in cfg.model_fields_set: + cfg = cfg.model_copy(update={"kind": spec.kind}) + resolved[name] = cfg + for text in flags: + name, kind, target = parse_surface_flag(text) + spec = _check(name) + base = resolved.get(name) or BackendConfig(kind=spec.kind) + updates: dict[str, Any] = {"kind": kind or base.kind} + if target is not None: + updates[TARGET_FIELD[str(updates["kind"]).lower()]] = target + resolved[name] = base.model_copy(update=updates) + for name, cfg in resolved.items(): + kind = str(cfg.kind).strip().lower() + if kind != specs[name].kind: + raise SurfaceBindingError( + f"surface {name!r} is declared {specs[name].kind}, but it was " + f"bound to a {kind} backend" + ) + return resolved + + +def unbound_backend_surfaces( + workflow: "Workflow", bound: Mapping[str, Any] +) -> list[str]: + """Backend surfaces of ``workflow`` with no binding, in declared order.""" + + return [ + spec.id + for spec in workflow.surfaces[1:] + if not spec.is_page and spec.id not in bound + ] + + +def build_surface_backends( + configs: Mapping[str, "BackendConfig"], + *, + new_page: Optional[Callable[[str], Any]] = None, + build: Optional[Callable[..., Any]] = None, +) -> dict[str, Any]: + """Construct each bound backend through the ordinary backend factory. + + A ``web`` backend surface is a second app in the run's own browser: + ``new_page(url)`` opens and navigates that page (the CLI passes a function + that opens it in the launched browser context, so the surface router + watches it like every other page). + """ + + if build is None: + from openadapt_flow.backends.factory import build_backend + + build = build_backend + backends: dict[str, Any] = {} + for name, cfg in configs.items(): + if str(cfg.kind).strip().lower() == "web": + if not cfg.url: + raise SurfaceBindingError( + f"web surface {name!r} needs a URL (--surface {name}=web:URL)" + ) + if new_page is None: + raise SurfaceBindingError( + f"web surface {name!r} needs a browser; bind it on a run " + "whose primary surface is the browser" + ) + backends[name] = build(cfg, page=new_page(cfg.url)) + else: + backends[name] = build(cfg) + return backends diff --git a/packages/flow/tests/test_attached_background_tab_capture.py b/packages/flow/tests/test_attached_background_tab_capture.py new file mode 100644 index 000000000..185c0e2cf --- /dev/null +++ b/packages/flow/tests/test_attached_background_tab_capture.py @@ -0,0 +1,123 @@ +"""A capture of an attached tab that a popup pushed to the background. + +In a person's Chromium (reached over CDP, so without Playwright's launch flags +that keep background tabs rendering), ``window.open`` opens a new foreground +tab. Chromium then draws no new frames for the opener, so a secret-masked +capture of it can wait forever (``Page.screenshot: Timeout 30000ms``). Attach +recording captures the opener's after-frame exactly then. The backend brings +the captured page to front first, so every capture is a live frame. +""" + +from __future__ import annotations + +import subprocess +import threading +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Iterator + +import pytest + +pytest.importorskip("playwright.sync_api") + +from openadapt_flow.backends.playwright_backend import PlaywrightBackend # noqa: E402 +from tests.test_browser_attach import _chromium_executable # noqa: E402 + +_PAGE = ( + b"" + b"" +) + + +@pytest.fixture +def app_url() -> Iterator[str]: + class Handler(BaseHTTPRequestHandler): + def do_GET(self) -> None: + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(_PAGE))) + self.end_headers() + self.wfile.write(_PAGE) + + def log_message(self, *_args: object) -> None: + return + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + yield f"http://127.0.0.1:{server.server_address[1]}/" + finally: + server.shutdown() + server.server_close() + thread.join(timeout=5) + + +@pytest.mark.timeout(120) +def test_masked_capture_of_a_backgrounded_attached_tab_does_not_stall( + tmp_path: Path, app_url: str +) -> None: + executable = _chromium_executable() + if executable is None: + pytest.skip("no Chromium executable is installed") + from playwright.sync_api import sync_playwright + + profile = tmp_path / "profile" + profile.mkdir() + process = subprocess.Popen( + [ + str(executable), + "--headless=new", + "--no-sandbox", + "--no-first-run", + "--no-default-browser-check", + "--remote-debugging-address=127.0.0.1", + "--remote-debugging-port=0", + f"--user-data-dir={profile}", + "--window-size=1280,800", + "--force-device-scale-factor=1", + app_url, + ], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + try: + port_file = profile / "DevToolsActivePort" + deadline = time.monotonic() + 20 + port_line = "" + while time.monotonic() < deadline and not port_line: + if process.poll() is not None: + pytest.fail("Chromium exited before its CDP endpoint was ready") + try: + port_line = port_file.read_text().splitlines()[0].strip() + except (OSError, IndexError): + time.sleep(0.05) + assert port_line, "Chromium CDP endpoint did not become ready" + with sync_playwright() as playwright: + browser = playwright.chromium.connect_over_cdp( + f"http://127.0.0.1:{int(port_line)}" + ) + page = browser.contexts[0].pages[0] + page.wait_for_load_state("load") + backend = PlaywrightBackend( + page, + screenshot_scale="css", + screenshot_mask_selectors=("#secret",), + bring_to_front_before_capture=True, + ) + with page.expect_popup() as popup_info: + page.click("#open") + popup = popup_info.value + popup.set_content("") + popup.fill("#note", "the popup now holds the foreground") + popup.click("#save") + page.set_default_timeout(10_000) + for _ in range(12): + assert backend.screenshot().startswith(b"\x89PNG") + finally: + process.terminate() + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() diff --git a/packages/flow/tests/test_bundle_schema_v2.py b/packages/flow/tests/test_bundle_schema_v2.py index 988b4adb8..12bdba193 100644 --- a/packages/flow/tests/test_bundle_schema_v2.py +++ b/packages/flow/tests/test_bundle_schema_v2.py @@ -192,6 +192,11 @@ def _pre_frame_path_content(wf: Workflow, shape: str) -> dict: # Reviewed v2 omission: submit-binding evidence postdates these seals. assert step.pop("submit_binding") is None assert step.pop("submit_binding_unbound_reason") is None + # Reviewed v2 omission: multi-surface fields postdate these seals. + assert step.pop("surface") is None + assert step.pop("opens") == [] + assert step.pop("surface_origin") is None + assert step.pop("dialog") is None for effect in _effect_payloads(content, shape): # Reviewed v2 omission: the exact_new_set over-write guard postdates # these seals. Navigated to the exact owning Effect, never stripped by @@ -246,6 +251,10 @@ def _write_synthetic_pre_field_bundle( assert step.pop("selection_region") is None assert step.pop("submit_binding") is None assert step.pop("submit_binding_unbound_reason") is None + assert step.pop("surface") is None + assert step.pop("opens") == [] + assert step.pop("surface_origin") is None + assert step.pop("dialog") is None for effect in _effect_payloads(raw, shape): assert effect.pop("new_records") == [] assert effect.pop("identity_field") == "id" diff --git a/packages/flow/tests/test_interop_types.py b/packages/flow/tests/test_interop_types.py index e199a30ec..8928f4a9a 100644 --- a/packages/flow/tests/test_interop_types.py +++ b/packages/flow/tests/test_interop_types.py @@ -46,7 +46,8 @@ def test_action_kind_map_covers_losslessly_shared_values() -> None: ir.ActionKind.SCROLL: "scroll", } assert set(interop.ACTION_KIND_TO_ACTION_TYPE) == set(ir.ActionKind) - { - ir.ActionKind.SELECT_OPTION + ir.ActionKind.SELECT_OPTION, + ir.ActionKind.SWITCH_SURFACE, } assert interop.ACTION_KIND_TO_ACTION_TYPE == expected @@ -79,6 +80,17 @@ def test_select_option_refuses_lossy_shared_type_conversion() -> None: interop.step_to_action(step) +def test_switch_surface_refuses_shared_type_conversion() -> None: + step = ir.Step( + id="switch", + intent="switch to the approval window", + action=ir.ActionKind.SWITCH_SURFACE, + surface="approval", + ) + with pytest.raises(ValueError, match="flow-only control step"): + interop.step_to_action(step) + + # --- Step -> Action -------------------------------------------------------- diff --git a/packages/flow/tests/test_multi_surface_attach.py b/packages/flow/tests/test_multi_surface_attach.py new file mode 100644 index 000000000..8e8c0929b --- /dev/null +++ b/packages/flow/tests/test_multi_surface_attach.py @@ -0,0 +1,204 @@ +"""Attach mode (``record --cdp``) follows popups and tabs (docs/MULTI_SURFACE.md). + +The person's own Chromium (started here with remote debugging) shows the +fixture app of test_multi_surface_e2e.py. Flow attaches to its one tab. The +app's ``window.open`` popup and its ``noopener`` report tab come from the +browser's target list: a page is followed only when one of the followed pages +opened it (``window.opener``, or the target's opener id for a noopener tab). +Each action in another page is recorded as an explicit switch, and the bundle +replays in the same attached browser with zero model calls. A tab the person +opens by hand is still refused. +""" + +from __future__ import annotations + +import subprocess +import time +from pathlib import Path +from typing import Iterator + +import pytest + +pytest.importorskip("playwright.sync_api") + +import tests.test_multi_surface_e2e as _e2e # noqa: E402 +from openadapt_flow.backends.playwright_backend import PlaywrightBackend # noqa: E402 +from openadapt_flow.compiler import compile_recording # noqa: E402 +from openadapt_flow.interactive_recorder import ( # noqa: E402 + BrowserAttachError, + record_interactive, +) +from openadapt_flow.ir import ActionKind, Workflow # noqa: E402 +from openadapt_flow.runtime.replayer import Replayer # noqa: E402 +from tests._attached_chromium import resize_tab_viewport # noqa: E402 +from tests.test_browser_attach import _chromium_executable # noqa: E402 + +pytestmark = pytest.mark.timeout(600) + +app_url = _e2e.app_url + +SHAPE = [ + (ActionKind.CLICK, "main"), + (ActionKind.SWITCH_SURFACE, "popup_1"), + (ActionKind.CLICK, "popup_1"), + (ActionKind.SWITCH_SURFACE, "main"), + (ActionKind.CLICK, "main"), + (ActionKind.SWITCH_SURFACE, "tab_1"), + (ActionKind.CLICK, "tab_1"), + (ActionKind.SWITCH_SURFACE, "main"), + (ActionKind.CLICK, "main"), +] + + +@pytest.fixture(scope="module") +def endpoint(app_url, tmp_path_factory) -> Iterator[str]: + executable = _chromium_executable() + if executable is None: + pytest.skip("no Chromium executable is installed") + profile = tmp_path_factory.mktemp("attach-profile") + process = subprocess.Popen( + [ + str(executable), + "--headless=new", + "--no-sandbox", + "--no-first-run", + "--no-default-browser-check", + "--disable-background-networking", + "--disable-popup-blocking", + "--remote-debugging-address=127.0.0.1", + "--remote-debugging-port=0", + f"--user-data-dir={profile}", + "--window-size=1280,800", + "--force-device-scale-factor=1", + "about:blank", + ], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + try: + port_file = profile / "DevToolsActivePort" + deadline = time.monotonic() + 20 + port_line = "" + while time.monotonic() < deadline and not port_line: + if process.poll() is not None: + pytest.fail("Chromium exited before its CDP endpoint was ready") + try: + port_line = port_file.read_text().splitlines()[0].strip() + except (OSError, IndexError): + time.sleep(0.05) + assert port_line, "Chromium CDP endpoint did not become ready" + yield f"http://127.0.0.1:{int(port_line)}" + finally: + process.terminate() + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + + +def _reset_person_browser(endpoint: str, url: str) -> None: + """One tab at ``url`` with the replay viewport, as a person would have it.""" + + from playwright.sync_api import sync_playwright + + with sync_playwright() as pw: + browser = pw.chromium.connect_over_cdp(endpoint) + context = browser.contexts[0] + # A fresh tab, never one an earlier test left behind: a leftover + # popup keeps its window.name, so the app's window.open(..., name) + # would navigate that tab instead of opening a popup, and the + # replay's new_tab_opened check would (rightly) fail. CDP lists + # pages in no fixed order, so "keep the first" could keep the popup. + keep = context.new_page() + for extra in list(context.pages): + if extra is not keep: + extra.close() + assert context.pages == [keep] + keep.goto(url) + keep.wait_for_load_state("load") + assert keep.evaluate("window.name") == "" + resize_tab_viewport(keep, *_e2e.VIEWPORT.values()) + + +def _drive(page, pump) -> None: + def settle(n: int = 3) -> None: + for _ in range(n): + pump() + + with page.expect_popup() as popup_info: + page.click("#open-popup") + popup = popup_info.value + popup.wait_for_load_state("load") + settle() + popup.click("#approve") + settle() + with page.expect_popup() as tab_info: + page.click("#open-report") + tab = tab_info.value + tab.wait_for_load_state("load") + settle() + tab.click("#reviewed") + settle() + page.click("#finish") + settle(5) + + +@pytest.fixture(scope="module") +def attached_bundle(endpoint, app_url, tmp_path_factory) -> tuple[Path, Workflow]: + root = tmp_path_factory.mktemp("attach_multi_surface") + url = app_url + "/" + _reset_person_browser(endpoint, url) + recording = record_interactive( + url, root / "recording", cdp_endpoint=endpoint, script=_drive + ) + workflow = compile_recording(recording, root / "bundle", name="attached-popup") + return root / "bundle", workflow + + +def test_cdp_attach_records_a_popup_and_a_tab_as_surfaces(attached_bundle) -> None: + _bundle_dir, workflow = attached_bundle + assert [s.id for s in workflow.surfaces] == ["main", "popup_1", "tab_1"] + assert workflow.surfaces[1].opener_link == "window_opener" + assert workflow.surfaces[2].opener_link == "none" + assert [(s.action, s.surface) for s in workflow.steps] == SHAPE + + +def test_cdp_attached_bundle_replays_in_the_same_browser( + attached_bundle, endpoint, app_url, tmp_path +) -> None: + from playwright.sync_api import sync_playwright + + bundle_dir, _workflow = attached_bundle + url = app_url + "/" + _reset_person_browser(endpoint, url) + with sync_playwright() as pw: + browser = pw.chromium.connect_over_cdp(endpoint) + page = browser.contexts[0].pages[0] + report = Replayer(PlaywrightBackend(page), poll_interval_s=0.05).run( + Workflow.load(bundle_dir), bundle_dir=bundle_dir, run_dir=tmp_path / "run" + ) + assert report.success, _e2e._summary(report) + assert report.model_calls == 0 + assert [r.surface for r in report.results] == [s for _, s in SHAPE] + assert page.locator("#popup-status").inner_text() == "Approval Granted" + assert page.locator("#done").inner_text() == "Referral Finished" + + +def test_cdp_attach_refuses_a_tab_the_person_opened( + endpoint, app_url, tmp_path +) -> None: + url = app_url + "/" + _reset_person_browser(endpoint, url) + + def drive(page, pump) -> None: + pump() + # A tab no followed page opened: Flow cannot tell what it belongs to. + stray = page.context.new_page() + stray.goto(app_url + "/other.html") + for _ in range(3): + pump() + + with pytest.raises(BrowserAttachError, match="popup or new tab"): + record_interactive( + url, tmp_path / "recording", cdp_endpoint=endpoint, script=drive + ) diff --git a/packages/flow/tests/test_multi_surface_backends.py b/packages/flow/tests/test_multi_surface_backends.py new file mode 100644 index 000000000..56c0da3d6 --- /dev/null +++ b/packages/flow/tests/test_multi_surface_backends.py @@ -0,0 +1,260 @@ +"""One bundle that switches backends: a web app, then a native Windows app. + +The Windows half runs against the stateful MockWaa server that +``tests/test_windows_backend.py`` already uses. The demonstration starts in a +browser, switches to the Windows backend with an explicit SWITCH_SURFACE step, +and finishes there; replay gets the Windows backend by surface id. Refusals: +no backend supplied, a backend of the wrong kind, and a Windows backend that +shows a different application all halt at the switch, before any input +reaches the Windows app. +""" + +from __future__ import annotations + +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Iterator + +import cv2 +import pytest + +pytest.importorskip("playwright.sync_api") + +from openadapt_flow.backends.playwright_backend import PlaywrightBackend # noqa: E402 +from openadapt_flow.backends.windows_backend import WindowsBackend # noqa: E402 +from openadapt_flow.compiler import compile_recording # noqa: E402 +from openadapt_flow.ir import ActionKind, PostconditionKind, Workflow # noqa: E402 +from openadapt_flow.recorder import Recorder # noqa: E402 +from openadapt_flow.runtime.replayer import Replayer # noqa: E402 +from tests.test_windows_backend import ( # noqa: E402 + BUTTON_CENTER, + NOTE_VALUE, + MockWaa, + app_screens, + blank, + draw_text, + png_bytes, +) + +pytestmark = [ + pytest.mark.timeout(900), + pytest.mark.filterwarnings("ignore:win_agent channel is plaintext HTTP"), +] + +_PAGE = """Intake Queue +

Intake Queue Overview

+ +

""" + + +class _Handler(BaseHTTPRequestHandler): + def log_message(self, *args: object) -> None: + pass + + def do_GET(self) -> None: # noqa: N802 - http.server API + data = _PAGE.encode() + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + +@pytest.fixture(scope="module") +def app_url() -> Iterator[str]: + server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) + threading.Thread(target=server.serve_forever, daemon=True).start() + yield f"http://127.0.0.1:{server.server_address[1]}" + server.shutdown() + server.server_close() + + +@pytest.fixture(scope="module") +def waa() -> Iterator[MockWaa]: + server = MockWaa(app_screens()) + yield server + server.close() + + +@pytest.fixture(scope="module") +def browser() -> Iterator[object]: + from openadapt_flow._browser_setup import ensure_chromium_installed + + ensure_chromium_installed() + from playwright.sync_api import sync_playwright + + with sync_playwright() as pw: + chromium = pw.chromium.launch(headless=True) + yield chromium + chromium.close() + + +def _page(browser: object, url: str): + page = browser.new_page( # type: ignore[attr-defined] + viewport={"width": 1280, "height": 800}, device_scale_factor=1 + ) + page.goto(url) + page.wait_for_load_state("load") + return page + + +def _windows(waa: MockWaa) -> WindowsBackend: + return WindowsBackend( + waa.url, screenshot_retry_delay_s=0.01, allow_legacy_exec=True + ) + + +@pytest.fixture(scope="module") +def bundle(app_url: str, browser: object, waa: MockWaa, tmp_path_factory): + root = tmp_path_factory.mktemp("multi_backend") + page = _page(browser, app_url) + waa.reset() + try: + recorder = Recorder( + PlaywrightBackend(page), + root / "recording", + app_url=app_url, + settle_interval_s=0.02, + settle_timeout_s=2.0, + ) + box = page.locator("#claim").bounding_box() + assert box is not None + recorder.click(int(box["x"] + box["width"] / 2), int(box["y"] + 20)) + recorder.switch_surface("windows_1", backend=_windows(waa)) + recorder.click(*BUTTON_CENTER) + recorder.type_text(NOTE_VALUE, param="note") + recorder.press("Enter") + recording = recorder.finish() + finally: + page.context.close() + assert waa.state == 3 + workflow = compile_recording(recording, root / "bundle", name="web-then-windows") + return root / "bundle", workflow + + +def test_compiles_a_typed_backend_switch(bundle) -> None: + _dir, workflow = bundle + assert [(s.id, s.kind) for s in workflow.surfaces] == [ + ("main", "web"), + ("windows_1", "windows"), + ] + assert [(s.action, s.surface) for s in workflow.steps] == [ + (ActionKind.CLICK, None), # before any switch: primary + (ActionKind.SWITCH_SURFACE, "windows_1"), + (ActionKind.CLICK, "windows_1"), + (ActionKind.TYPE, "windows_1"), + (ActionKind.KEY, "windows_1"), + ] + switch = workflow.steps[1] + assert switch.surface_origin is None # not a browser target + # Its identity check is the new surface's own screen text. + assert [pc.kind for pc in switch.expect] == [PostconditionKind.TEXT_PRESENT] + assert switch.identity_armed is True + + +def test_replays_web_then_windows_with_zero_model_calls( + bundle, app_url: str, browser: object, waa: MockWaa, tmp_path: Path +) -> None: + bundle_dir, _ = bundle + waa.reset() + page = _page(browser, app_url) + try: + primary = PlaywrightBackend(page) + replayer = Replayer( + primary, + poll_interval_s=0.02, + surface_backends={"windows_1": _windows(waa)}, + ) + report = replayer.run( + Workflow.load(bundle_dir), + params={"note": NOTE_VALUE}, + bundle_dir=bundle_dir, + run_dir=tmp_path / "run", + ) + assert report.success, [(r.step_id, r.error) for r in report.results] + assert report.model_calls == 0 + assert page.locator("#s").inner_text() == "Item Claimed For Desktop" + assert waa.state == 3 + # The primary backend is restored when the run ends. + assert replayer.backend is primary + finally: + page.context.close() + + +def _replay_with(bundle_dir: Path, browser, url, tmp_path, surface_backends): + page = _page(browser, url) + try: + return Replayer( + PlaywrightBackend(page), + poll_interval_s=0.02, + surface_backends=surface_backends, + ).run( + Workflow.load(bundle_dir), + params={"note": NOTE_VALUE}, + bundle_dir=bundle_dir, + run_dir=tmp_path / "run", + ) + finally: + page.context.close() + + +def test_switch_without_a_supplied_backend_halts( + bundle, app_url, browser, waa: MockWaa, tmp_path +) -> None: + bundle_dir, _ = bundle + waa.reset() + report = _replay_with(bundle_dir, browser, app_url, tmp_path, None) + assert not report.success + last = report.results[-1] + assert last.step_id == "step_001" and last.safety_halt + assert "no backend was supplied for surface 'windows_1'" in (last.error or "") + assert waa.state == 0 and waa.commands == [] + + +def test_switch_to_a_backend_of_the_wrong_kind_halts( + bundle, app_url, browser, waa: MockWaa, tmp_path +) -> None: + bundle_dir, _ = bundle + waa.reset() + other = _page(browser, app_url) + try: + report = _replay_with( + bundle_dir, + browser, + app_url, + tmp_path, + {"windows_1": PlaywrightBackend(other)}, + ) + finally: + other.context.close() + assert not report.success + last = report.results[-1] + assert last.step_id == "step_001" + assert "declared windows, but the supplied backend serves web" in (last.error or "") + assert waa.commands == [] + + +def test_switch_to_a_windows_app_that_is_not_the_recorded_one_halts( + bundle, app_url, browser, tmp_path +) -> None: + bundle_dir, _ = bundle + other_app = blank() + draw_text(other_app, 420, 84, "Unrelated Billing Console") + cv2.rectangle(other_app, (560, 400), (720, 448), (205, 205, 205), -1) + wrong = MockWaa([png_bytes(other_app)] * 4) + try: + report = _replay_with( + bundle_dir, browser, app_url, tmp_path, {"windows_1": _windows(wrong)} + ) + assert not report.success + last = report.results[-1] + assert last.step_id == "step_001" and last.safety_halt + assert "Surface identity check failed" in (last.error or "") + # No input reached the wrong application. + assert wrong.commands == [] + finally: + wrong.close() diff --git a/packages/flow/tests/test_multi_surface_contract.py b/packages/flow/tests/test_multi_surface_contract.py new file mode 100644 index 000000000..1f6ae6ffb --- /dev/null +++ b/packages/flow/tests/test_multi_surface_contract.py @@ -0,0 +1,225 @@ +"""Unit tests for the multi-surface IR contract and the replayer's refusals.""" + +from __future__ import annotations + +import json + +import pytest +from pydantic import ValidationError + +from openadapt_flow.ir import ( + ActionKind, + Postcondition, + PostconditionKind, + Step, + SurfaceSpec, + Workflow, + surface_contract_errors, +) +from openadapt_flow.runtime.replayer import Replayer +from tests.test_replayer import FakeBackend + + +def _wait(step_id: str, **kw) -> Step: + return Step(id=step_id, intent="wait", action=ActionKind.WAIT, **kw) + + +def _switch(step_id: str, surface: str, **kw) -> Step: + kw.setdefault( + "expect", + [Postcondition(kind=PostconditionKind.TEXT_PRESENT, text="Desktop Home")], + ) + return Step( + id=step_id, + intent=f"switch to surface '{surface}'", + action=ActionKind.SWITCH_SURFACE, + surface=surface, + **{"risk": "reversible", **kw}, + ) + + +_MAIN = SurfaceSpec(id="main", kind="web") +_WIN = SurfaceSpec(id="windows_1", kind="windows") +_POPUP = SurfaceSpec( + id="popup_1", + kind="web", + opened_by_step="step_000", + opener="main", + open_order=0, + opener_link="window_opener", + origin="http://127.0.0.1:8000", + url_pattern=r"http://127\.0\.0\.1:8000/popup\.html(?:[?#].*)?", +) + + +def _multi() -> Workflow: + return Workflow( + name="wf", + surfaces=[_MAIN, _WIN], + steps=[ + _wait("step_000"), + _switch("step_001", "windows_1"), + _wait("step_002", surface="windows_1"), + ], + ) + + +def test_single_surface_bundle_serializes_without_surface_fields() -> None: + workflow = Workflow(name="wf", steps=[_wait("step_000")]) + dumped = json.loads(workflow.model_dump_json()) + assert "surfaces" not in dumped + assert surface_contract_errors(workflow) == [] + assert workflow.is_multi_surface is False + + +def test_multi_surface_round_trips() -> None: + workflow = _multi() + again = Workflow.model_validate_json(workflow.model_dump_json()) + assert [s.id for s in again.surfaces] == ["main", "windows_1"] + assert again.steps[1].action is ActionKind.SWITCH_SURFACE + assert again.is_multi_surface + + +def test_surface_fields_without_declared_surfaces_are_refused() -> None: + with pytest.raises(ValidationError, match="declares no surfaces"): + Workflow(name="wf", steps=[_wait("step_000", surface="windows_1")]) + + +def test_step_on_a_surface_never_switched_to_is_refused() -> None: + with pytest.raises(ValidationError): + Workflow( + name="wf", + surfaces=[_MAIN, _WIN], + steps=[_wait("step_000", surface="windows_1")], + ) + + +def test_switch_to_a_page_before_it_opens_is_refused() -> None: + with pytest.raises(ValidationError): + Workflow( + name="wf", + surfaces=[_MAIN, _POPUP], + steps=[_switch("step_000", "popup_1"), _wait("step_001")], + ) + + +def test_declared_page_must_match_the_step_opens_list() -> None: + with pytest.raises(ValidationError, match="opens"): + Workflow( + name="wf", + surfaces=[_MAIN, _POPUP], + steps=[_wait("step_000"), _switch("step_001", "popup_1")], + ) + ok = Workflow( + name="wf", + surfaces=[_MAIN, _POPUP], + steps=[ + _wait("step_000", opens=["popup_1"]), + _switch("step_001", "popup_1"), + ], + ) + assert ok.steps[0].opens == ["popup_1"] + + +def test_primary_surface_cannot_be_a_page() -> None: + with pytest.raises(ValidationError, match="primary"): + Workflow(name="wf", surfaces=[_POPUP], steps=[_wait("step_000")]) + + +@pytest.mark.parametrize( + "extra", + [{"text": "x"}, {"param": "p"}, {"risk": "irreversible"}, {"opens": ["a"]}], +) +def test_switch_step_carries_no_input(extra) -> None: + with pytest.raises(ValidationError): + _switch("step_001", "windows_1", **extra) + + +def test_switch_step_requires_a_surface() -> None: + with pytest.raises(ValidationError): + Step(id="s", intent="switch", action=ActionKind.SWITCH_SURFACE) + + +def test_surface_origin_only_on_switch() -> None: + with pytest.raises(ValidationError): + _wait("s", surface_origin="http://127.0.0.1:8000") + + +def test_page_surface_requires_its_identity_fields() -> None: + with pytest.raises(ValidationError): + SurfaceSpec(id="popup_1", kind="web", opened_by_step="step_000", opener="main") + with pytest.raises(ValidationError): + SurfaceSpec(id="Bad-Id", kind="web") + + +class _WindowsFake(FakeBackend): + surface_kind = "windows" + + +def _run(replayer: Replayer, workflow: Workflow, tmp_path, **kw): + bundle_dir = tmp_path / "bundle" + workflow.save(bundle_dir) + workflow = Workflow.load(bundle_dir) + return replayer.run(workflow, bundle_dir=bundle_dir, run_dir=tmp_path / "run", **kw) + + +def test_resume_without_a_surface_observation_is_refused(tmp_path) -> None: + # A resume re-binds surfaces by identity from the last checkpoint. A + # checkpoint that carries no surface observation (written by an older + # runtime) cannot be re-bound, so the resume is refused before any input. + windows = _WindowsFake() + replayer = Replayer(FakeBackend(), surface_backends={"windows_1": windows}) + refusal = replayer._begin_surfaces(_multi(), resume=True, resume_from=1) + assert "cannot be re-bound" in (refusal or "") + assert windows.actions == [] + + +def test_resume_of_a_multi_surface_program_is_refused() -> None: + replayer = Replayer(FakeBackend(), surface_backends={"windows_1": _WindowsFake()}) + refusal = replayer._begin_surfaces(_multi(), resume=True, resume_program=True) + assert "program" in (refusal or "") + + +def test_resume_from_the_start_binds_the_primary() -> None: + replayer = Replayer(FakeBackend(), surface_backends={"windows_1": _WindowsFake()}) + assert replayer._begin_surfaces(_multi(), resume=True, resume_from=0) is None + assert replayer._surface_router is not None + assert replayer._surface_router.active_id == "main" + + +def test_managed_dispatch_needs_every_backend_surface_bound() -> None: + # Cloud dispatch needs only the deployment profile's surfaces: map. The + # one refusal kept guards something real: an unattended run must not + # start when a surface it will switch to has no target. + unbound = Replayer(FakeBackend()) + unbound._managed_dispatch_snapshot = ("run", "sha", "{}") + refusal = unbound._begin_surfaces(_multi(), resume=False) + assert "surfaces: map" in (refusal or "") and "windows_1" in (refusal or "") + bound = Replayer(FakeBackend(), surface_backends={"windows_1": _WindowsFake()}) + bound._managed_dispatch_snapshot = ("run", "sha", "{}") + assert bound._begin_surfaces(_multi(), resume=False) is None + + +def test_surface_backends_for_a_single_surface_bundle_is_refused(tmp_path) -> None: + replayer = Replayer(FakeBackend(), surface_backends={"windows_1": _WindowsFake()}) + report = _run(replayer, Workflow(name="wf", steps=[_wait("step_000")]), tmp_path) + assert not report.success + assert report.results[-1].step_id == "" + + +def test_switch_without_supplied_backend_halts_before_later_steps(tmp_path) -> None: + primary = FakeBackend() + replayer = Replayer(primary) + report = _run(replayer, _multi(), tmp_path) + assert not report.success + assert report.results[-1].step_id == "step_001" + assert report.results[-1].safety_halt + assert "no backend was supplied" in (report.results[-1].error or "") + assert replayer.backend is primary + + +def test_switch_to_wrong_kind_backend_halts(tmp_path) -> None: + replayer = Replayer(FakeBackend(), surface_backends={"windows_1": FakeBackend()}) + report = _run(replayer, _multi(), tmp_path) + assert not report.success + assert report.results[-1].step_id == "step_001" diff --git a/packages/flow/tests/test_multi_surface_dialog.py b/packages/flow/tests/test_multi_surface_dialog.py new file mode 100644 index 000000000..c5f546ef5 --- /dev/null +++ b/packages/flow/tests/test_multi_surface_dialog.py @@ -0,0 +1,130 @@ +"""A JavaScript dialog is a screen state answered by a backend action. + +A ``confirm()`` dialog is not a surface. The step whose click raises it +carries the answer (``Step.dialog``) and the dialog's message as its identity. +The browser backend answers it only when the message matches; a different +message is dismissed, never accepted, and the run halts +(docs/MULTI_SURFACE.md, "Dialogs"). +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +pytest.importorskip("playwright.sync_api") + +import tests.test_multi_surface_e2e as _e2e # noqa: E402 +from openadapt_flow.backends.playwright_backend import PlaywrightBackend # noqa: E402 +from openadapt_flow.compiler import compile_recording # noqa: E402 +from openadapt_flow.ir import ActionKind, DialogResponse, Step, Workflow # noqa: E402 +from openadapt_flow.recorder import Recorder # noqa: E402 + +pytestmark = pytest.mark.timeout(600) + +app_url = _e2e.app_url +browser = _e2e.browser + + +@pytest.fixture(scope="module") +def dialog_bundle(app_url, browser, tmp_path_factory) -> tuple[Path, Workflow]: + root = tmp_path_factory.mktemp("dialog") + url = app_url + "/confirm.html" + page = _e2e._fresh_page(browser, url) + try: + recorder = Recorder( + PlaywrightBackend(page), + root / "recording", + app_url=url, + settle_interval_s=0.05, + settle_timeout_s=2.0, + ) + recorder.expect_dialog("accept") + recorder.click(*_e2e._center(page, "#delete")) + assert page.locator("#record-status").inner_text() == "Record Deleted" + recording = recorder.finish() + finally: + page.close() + workflow = compile_recording(recording, root / "bundle", name="dialog") + return root / "bundle", workflow + + +def test_the_dialog_answer_is_compiled_onto_the_step_that_raises_it( + dialog_bundle, +) -> None: + _bundle_dir, workflow = dialog_bundle + assert workflow.surfaces == [] # a dialog is not a surface + [step] = workflow.steps + assert step.action is ActionKind.CLICK + # The record number is generalized, never sealed into the bundle. + assert step.dialog == DialogResponse( + response="accept", message_pattern=r"Delete\ record\ \d+\?" + ) + assert "record 42" not in (_bundle_dir / "workflow.json").read_text() + + +def test_a_matching_dialog_is_answered(dialog_bundle, app_url, browser, tmp_path): + bundle_dir, _workflow = dialog_bundle + report, page = _e2e._replay( + browser, app_url + "/confirm.html", bundle_dir, tmp_path / "run" + ) + try: + assert report.success, _e2e._summary(report) + assert report.model_calls == 0 + assert page.locator("#record-status").inner_text() == "Record Deleted" + finally: + page.close() + + +def test_a_dialog_identity_mismatch_halts_and_is_dismissed( + dialog_bundle, app_url, browser, tmp_path +): + bundle_dir, _workflow = dialog_bundle + report, page = _e2e._replay( + browser, + app_url + "/confirm.html?msg=Delete%20ALL%20records%3F", + bundle_dir, + tmp_path / "run", + ) + try: + assert not report.success + [result] = report.results + assert not result.ok + # A code, never the dialog's text: the error lands in the reports. + assert "dialog_mismatch" in str(result.error) + assert "ALL records" not in str(result.error) + assert "record 42" not in str(result.error) + # Dismissed, never accepted: nothing was deleted. + assert page.locator("#record-status").inner_text() == "Record Kept" + finally: + page.close() + + +def test_a_screen_driven_backend_refuses_a_dialog_step() -> None: + class _Pixels: + pass + + from openadapt_flow.runtime.replayer import Replayer + + replayer = Replayer.__new__(Replayer) + replayer.backend = _Pixels() # type: ignore[assignment] + step = Step( + id="s", + intent="delete", + action=ActionKind.KEY, + key="Enter", + dialog=DialogResponse(response="accept", message="Sure?"), + ) + error = Replayer._act(replayer, step, None, {}) + assert error is not None and "cannot see browser dialogs" in error + + +def test_a_wait_step_cannot_carry_a_dialog() -> None: + with pytest.raises(ValueError, match="raises no dialog"): + Step( + id="s", + intent="wait", + action=ActionKind.WAIT, + dialog=DialogResponse(response="accept", message="Sure?"), + ) diff --git a/packages/flow/tests/test_multi_surface_e2e.py b/packages/flow/tests/test_multi_surface_e2e.py new file mode 100644 index 000000000..f56d7457a --- /dev/null +++ b/packages/flow/tests/test_multi_surface_e2e.py @@ -0,0 +1,369 @@ +"""Multi-surface workflows, end to end (docs/MULTI_SURFACE.md). + +A local fixture app opens a popup (``window.open``) and a second tab (a +``target=_blank`` link). The UNMODIFIED Recorder -> compiler -> Replayer stack +records the demonstration across all three pages, compiles one bundle, and +replays it on a fresh browser with zero model calls. Refusal cases: an +unexpected popup, a declared popup that never opens, and a popup on a +different URL all halt with a reason and drive nothing else. +""" + +from __future__ import annotations + +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Iterator +from urllib.parse import parse_qs, urlsplit + +import pytest + +pytest.importorskip("playwright.sync_api") + +from openadapt_flow.backends.playwright_backend import PlaywrightBackend # noqa: E402 +from openadapt_flow.compiler import compile_recording # noqa: E402 +from openadapt_flow.ir import ActionKind, Workflow # noqa: E402 +from openadapt_flow.recorder import Recorder # noqa: E402 +from openadapt_flow.runtime.replayer import Replayer # noqa: E402 + +pytestmark = pytest.mark.timeout(600) + +VIEWPORT = {"width": 1280, "height": 800} + +_STYLE = ( + "" +) + + +def _index(query: dict[str, list[str]]) -> str: + popup_path = query.get("popup", ["/popup.html"])[0] + no_popup = "nopopup" in query + surprise = "surprise" in query + open_popup = ( + "document.getElementById('popup-status').textContent='Nothing opened';" + if no_popup + else f"window.open('{popup_path}','approval');" + ) + surprise_js = "window.open('/promo.html','promo');" if surprise else "" + return f"""Referral Portal +{_STYLE} +

Referral Portal Home

+ +Open Report + + +

+""" + + +_POPUP = f"""Approval Window +{_STYLE} +

Approval Window Ready

+ +

+""" + +_OTHER = f"""Approval Window +{_STYLE}

Approval Window Ready

+""" + +_REPORT = f"""Quarterly Report +{_STYLE} +

Quarterly Report Viewer

+ +

+""" + +_SELFCLOSE = f"""Sign In Window +{_STYLE} +

Sign In Window Ready

+

Terms Pending

+ + +""" + +_CONFIRM = f"""Record Desk +{_STYLE} +

Record Desk Ready

+

Record Present

+ +""" + +_PROMO = f"""Special Offer +{_STYLE}

Unexpected Offer Window

""" + + +class _Handler(BaseHTTPRequestHandler): + def log_message(self, *args: object) -> None: + pass + + def do_GET(self) -> None: # noqa: N802 - http.server API + parts = urlsplit(self.path) + pages = { + "/popup.html": _POPUP, + "/other.html": _OTHER, + "/report.html": _REPORT, + "/promo.html": _PROMO, + "/selfclose.html": _SELFCLOSE, + "/confirm.html": _CONFIRM, + } + if parts.path in ("/", "/index.html"): + body = _index(parse_qs(parts.query)) + elif parts.path in pages: + body = pages[parts.path] + else: + self.send_response(404) + self.end_headers() + return + data = body.encode() + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + +@pytest.fixture(scope="module") +def app_url() -> Iterator[str]: + server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + yield f"http://127.0.0.1:{server.server_address[1]}" + server.shutdown() + server.server_close() + + +@pytest.fixture(scope="module") +def browser() -> Iterator[object]: + from openadapt_flow._browser_setup import ensure_chromium_installed + + ensure_chromium_installed() + from playwright.sync_api import sync_playwright + + with sync_playwright() as pw: + chromium = pw.chromium.launch(headless=True) + yield chromium + chromium.close() + + +def _fresh_page(browser: object, url: str): + page = browser.new_page(viewport=VIEWPORT, device_scale_factor=1) # type: ignore[attr-defined] + page.goto(url) + page.wait_for_load_state("load") + return page + + +def _center(page, selector: str) -> tuple[int, int]: + box = page.locator(selector).bounding_box() + assert box is not None, selector + return int(box["x"] + box["width"] / 2), int(box["y"] + box["height"] / 2) + + +def _record(recorder: Recorder, primary_page) -> None: + recorder.click(*_center(primary_page, "#open-popup")) + assert recorder.surface_ids == ("main", "popup_1") + recorder.switch_surface("popup_1") + popup_page = recorder.surface_backend("popup_1").page # type: ignore[attr-defined] + recorder.click(*_center(popup_page, "#approve")) + recorder.switch_surface("main") + recorder.click(*_center(primary_page, "#open-report")) + assert recorder.surface_ids == ("main", "popup_1", "tab_1") + recorder.switch_surface("tab_1") + report_page = recorder.surface_backend("tab_1").page # type: ignore[attr-defined] + recorder.click(*_center(report_page, "#reviewed")) + recorder.switch_surface("main") + recorder.click(*_center(primary_page, "#finish")) + + +@pytest.fixture(scope="module") +def bundle(app_url: str, browser: object, tmp_path_factory) -> tuple[Path, Workflow]: + root = tmp_path_factory.mktemp("multi_surface") + page = _fresh_page(browser, app_url) + try: + recorder = Recorder( + PlaywrightBackend(page), + root / "recording", + app_url=app_url, + settle_interval_s=0.05, + settle_timeout_s=2.0, + ) + _record(recorder, page) + recording = recorder.finish() + finally: + page.context.close() + workflow = compile_recording(recording, root / "bundle", name="multi-surface") + return root / "bundle", workflow + + +def _replay( + browser: object, url: str, bundle_dir: Path, run_dir: Path +) -> tuple[object, object]: + page = _fresh_page(browser, url) + report = Replayer(PlaywrightBackend(page), poll_interval_s=0.05).run( + Workflow.load(bundle_dir), bundle_dir=bundle_dir, run_dir=run_dir + ) + return report, page + + +def _summary(report) -> str: + return "\n".join(f"{r.step_id} ok={r.ok} err={r.error}" for r in report.results) + + +def test_compiles_one_bundle_across_popup_and_tab(bundle) -> None: + _bundle_dir, workflow = bundle + assert [s.id for s in workflow.surfaces] == ["main", "popup_1", "tab_1"] + popup = workflow.surfaces[1] + tab = workflow.surfaces[2] + assert popup.opened_by_step == "step_000" and popup.opener == "main" + assert popup.opener_link == "window_opener" + assert tab.opener_link == "none" # target=_blank implies noopener + assert popup.url_pattern is not None and popup.url_pattern.endswith( + r"popup\.html(?:[?#].*)?" + ) + # Titles are not recorded (they carry record names), so not asserted. + assert popup.title_pattern is None + actions = [(s.action, s.surface) for s in workflow.steps] + assert actions == [ + (ActionKind.CLICK, "main"), + (ActionKind.SWITCH_SURFACE, "popup_1"), + (ActionKind.CLICK, "popup_1"), + (ActionKind.SWITCH_SURFACE, "main"), + (ActionKind.CLICK, "main"), + (ActionKind.SWITCH_SURFACE, "tab_1"), + (ActionKind.CLICK, "tab_1"), + (ActionKind.SWITCH_SURFACE, "main"), + (ActionKind.CLICK, "main"), + ] + assert workflow.steps[0].opens == ["popup_1"] + assert workflow.steps[4].opens == ["tab_1"] + for step in workflow.steps: + if step.action is ActionKind.SWITCH_SURFACE: + assert step.risk == "reversible" + assert step.surface_origin is not None + assert step.identity_armed is True + # The bundle round-trips through its sealed manifest. + loaded = Workflow.load(_bundle_dir) + assert loaded.surfaces == workflow.surfaces + + +def test_replays_across_popup_and_tab_with_zero_model_calls( + bundle, app_url: str, browser: object, tmp_path +) -> None: + bundle_dir, _ = bundle + report, page = _replay(browser, app_url, bundle_dir, tmp_path / "run") + try: + assert report.success, _summary(report) + assert report.model_calls == 0 + # The demo profile never claims VERIFIED (that needs a governed run + # with independently confirmed effects; see + # test_multi_surface_governed.py); every step's own contract held. + assert report.execution_outcome == "COMPLETED_UNVERIFIED" + assert all(r.postconditions_ok is not False for r in report.results) + # The popup's click wrote back into the opener, the report tab got its + # click, and the final click ran on the main page again. + assert page.locator("#popup-status").inner_text() == "Approval Granted" + assert page.locator("#done").inner_text() == "Referral Finished" + pages = page.context.pages + assert len(pages) == 3 + report_page = pages[2] + assert report_page.locator("#ack").inner_text() == "Review Recorded Today" + switch_results = [ + r for r in report.results if r.step_id in ("step_001", "step_003") + ] + assert all(r.ok and r.postconditions_ok for r in switch_results) + # Every result records the surface it ran on. + assert [r.surface for r in report.results][:2] == ["main", "popup_1"] + finally: + page.context.close() + + +def test_unexpected_popup_halts_without_guessing( + bundle, app_url: str, browser: object, tmp_path +) -> None: + bundle_dir, _ = bundle + report, page = _replay( + browser, f"{app_url}/?surprise=1", bundle_dir, tmp_path / "run" + ) + try: + assert not report.success + last = report.results[-1] + assert last.step_id == "step_008" + assert last.safety_halt and last.failure_category == "safety_halt" + assert "no step of this workflow declares" in (last.error or "") + # The message never quotes the page's URL or title. + assert "promo" not in (last.error or "").lower() + assert "Offer" not in (last.error or "") + finally: + page.context.close() + + +def test_declared_popup_that_never_opens_halts( + bundle, app_url: str, browser: object, tmp_path +) -> None: + bundle_dir, _ = bundle + workflow = Workflow.load(bundle_dir) + workflow.steps[0].timeout_s = 1.0 + # The compiled new_tab_opened postcondition already catches this; drop it + # so the test exercises the surface check that backs it up. + workflow.steps[0].expect = [] + page = _fresh_page(browser, f"{app_url}/?nopopup=1") + try: + report = Replayer(PlaywrightBackend(page), poll_interval_s=0.05).run( + workflow, bundle_dir=bundle_dir, run_dir=tmp_path / "run" + ) + assert not report.success + assert len(report.results) == 1 + assert "expected to open 1 browser page(s)" in (report.results[0].error or "") + # Nothing ran on the opener after the refusal. + assert page.locator("#done").inner_text() == "" + finally: + page.context.close() + + +def test_popup_on_a_different_url_halts( + bundle, app_url: str, browser: object, tmp_path +) -> None: + bundle_dir, _ = bundle + report, page = _replay( + browser, f"{app_url}/?popup=/other.html", bundle_dir, tmp_path / "run" + ) + try: + assert not report.success + assert report.results[-1].step_id == "step_000" + assert "does not match the recorded pattern" in (report.results[-1].error or "") + assert "other" not in (report.results[-1].error or "") + finally: + page.context.close() + + +def test_recorded_surfaces_keep_no_query_string_or_title( + app_url: str, browser: object, tmp_path +) -> None: + import json + + # An OAuth-style popup: the code and state ride in the query string. + popup = "/popup.html?code=oauth-secret-code&state=state-token" + page = _fresh_page(browser, f"{app_url}/?popup={popup.replace('&', '%26')}") + try: + recorder = Recorder( + PlaywrightBackend(page), + tmp_path / "recording", + app_url=app_url, + settle_interval_s=0.05, + settle_timeout_s=2.0, + ) + recorder.click(*_center(page, "#open-popup")) + assert recorder.surface_ids == ("main", "popup_1") + recording = recorder.finish() + finally: + page.context.close() + meta_text = (recording / "meta.json").read_text() + [entry] = [s for s in json.loads(meta_text)["surfaces"] if s["id"] == "popup_1"] + assert entry["url"] == f"{app_url}/popup.html" + assert "title" not in entry + assert "oauth-secret-code" not in meta_text and "state-token" not in meta_text + assert "Approval Window" not in meta_text diff --git a/packages/flow/tests/test_multi_surface_fail_closed.py b/packages/flow/tests/test_multi_surface_fail_closed.py new file mode 100644 index 000000000..a32f74b4d --- /dev/null +++ b/packages/flow/tests/test_multi_surface_fail_closed.py @@ -0,0 +1,377 @@ +"""Fail-closed surface routing and PHI-free surface evidence. + +Each test pins one rule (docs/MULTI_SURFACE.md): + +- a step never runs on a page that closed under it, whatever the surface kind; +- a page surface rebinds only to a page claimed on this run (a resume, before + the surface is first bound, is the one exception), and only on visual + evidence; a held page that fails its check halts instead of being used; +- a switch to a page surface needs identity evidence, as primary and backend + switches do; +- dialog errors carry a code, not the dialog's text; the bundle seals a + pattern, not a record number or a parameter value; URL patterns generalize + identifier path segments; +- the layout-drift check follows the active surface. + +The router tests drive :class:`SurfaceRouter` with in-memory page doubles, so +an attach-mode browser (the person's own tabs already open) is cheap to set up. +""" + +from __future__ import annotations + +import re +from typing import Any, Optional + +import pytest + +from openadapt_flow.compiler.compile import _dialog_identity, _surface_url_pattern +from openadapt_flow.ir import ( + ActionKind, + DialogResponse, + Postcondition, + PostconditionKind, + Step, + SurfaceSpec, + Workflow, +) +from openadapt_flow.runtime.replayer import Replayer +from openadapt_flow.runtime.surfaces import SurfaceRouter + +ORIGIN = "http://127.0.0.1:8000" + + +class _Context: + def __init__(self) -> None: + self.pages: list[_Page] = [] + + +class _Page: + def __init__( + self, context: _Context, url: str, *, opener: Optional["_Page"] = None + ) -> None: + self.context = context + self.url = url + self._opener = opener + self.closed = False + context.pages.append(self) + + def is_closed(self) -> bool: + return self.closed + + def opener(self) -> Optional["_Page"]: + return self._opener + + def title(self) -> str: + return "" + + def evaluate(self, _script: str) -> bool: + return self._opener is not None # window.opener is set + + def bring_to_front(self) -> None: + pass + + def wait_for_timeout(self, _ms: float) -> None: + pass + + def close(self) -> None: + self.closed = True + self.context.pages.remove(self) + + +class _PageBackend: + def __init__(self, page: _Page) -> None: + self.page = page + + def screenshot(self) -> bytes: + return b"" + + +_MAIN = SurfaceSpec(id="main", kind="web") +_POPUP = SurfaceSpec( + id="popup_1", + kind="web", + opened_by_step="step_000", + opener="main", + open_order=0, + opener_link="window_opener", + origin=ORIGIN, + url_pattern=re.escape(f"{ORIGIN}/popup.html") + r"(?:[?#].*)?", +) +_SHOWS_POPUP = [Postcondition(kind=PostconditionKind.TEXT_PRESENT, text="Approval")] + + +def _workflow() -> Workflow: + return Workflow( + name="wf", + surfaces=[_MAIN, _POPUP], + steps=[ + Step( + id="step_000", + intent="open the popup", + action=ActionKind.WAIT, + opens=["popup_1"], + ), + _switch("step_001", "popup_1"), + _act("step_002"), + ], + ) + + +def _switch(step_id: str, surface: str, **kw: Any) -> Step: + kw.setdefault("expect", list(_SHOWS_POPUP)) + kw.setdefault("surface_origin", ORIGIN) + return Step( + id=step_id, + intent=f"switch to surface '{surface}'", + action=ActionKind.SWITCH_SURFACE, + surface=surface, + risk="reversible", + **kw, + ) + + +def _act(step_id: str, surface: str = "popup_1") -> Step: + return Step(id=step_id, intent="act", action=ActionKind.WAIT, surface=surface) + + +def _router( + *, visual: Optional[bool] = True, resume_active: Optional[str] = None +) -> tuple[SurfaceRouter, _Context, _Page]: + context = _Context() + main = _Page(context, f"{ORIGIN}/") + router = SurfaceRouter( + _workflow(), + _PageBackend(main), + page_backend_factory=_PageBackend, + poll_interval_s=0.01, + identity_check=lambda _backend, _step: visual, + resume_active=resume_active, + ) + return router, context, main + + +def _open_and_switch(router: SurfaceRouter, context: _Context, main: _Page) -> _Page: + workflow = _workflow() + popup = _Page(context, f"{ORIGIN}/popup.html", opener=main) + assert router.after_action_refusal(workflow.steps[0], timeout_s=0.0) is None + backend, refusal = router.switch(workflow.steps[1]) + assert refusal is None and backend.page is popup + return popup + + +# -- closed pages --------------------------------------------------------------- + + +def test_a_step_on_a_closed_page_surface_is_refused() -> None: + router, context, main = _router() + popup = _open_and_switch(router, context, main) + popup.close() + refusal = router.entry_refusal(_act("step_002")) + assert refusal is not None and "was closed" in refusal + + +# -- rebinding -------------------------------------------------------------------- + + +def test_a_switch_never_binds_a_page_this_run_did_not_open() -> None: + context = _Context() + main = _Page(context, f"{ORIGIN}/") + # Attach mode: the person already has a tab that looks like the popup. + theirs = _Page(context, f"{ORIGIN}/popup.html", opener=main) + router = SurfaceRouter( + _workflow(), + _PageBackend(main), + page_backend_factory=_PageBackend, + identity_check=lambda _backend, _step: True, + ) + backend, refusal = router.switch(_switch("step_001", "popup_1")) + assert backend is None + assert refusal is not None and "not opened on this run" in refusal + assert "popup_1" not in router.backends + assert not theirs.closed + + +def test_a_closed_popup_does_not_rebind_to_the_persons_tab() -> None: + router, context, main = _router() + popup = _open_and_switch(router, context, main) + popup.close() + theirs = _Page(context, f"{ORIGIN}/popup.html", opener=main) + router._remember(theirs) # in the inventory, as an attached tab would be + error = router.route(_act("step_002")) + assert error is not None and "no open browser page passes" in error + assert router.backends["popup_1"].page is not theirs + + +def test_a_resume_rebinds_only_on_visual_evidence() -> None: + # A resumed run may bind a page the earlier run opened, but only when the + # step's recorded screen evidence shows on it. + for visual, binds in ((None, False), (False, False), (True, True)): + context = _Context() + main = _Page(context, f"{ORIGIN}/") + earlier = _Page(context, f"{ORIGIN}/popup.html", opener=main) + router = SurfaceRouter( + _workflow(), + _PageBackend(main), + page_backend_factory=_PageBackend, + identity_check=lambda _backend, _step, v=visual: v, + resume_active="main", + ) + backend, refusal = router.switch(_switch("step_001", "popup_1")) + if binds: + assert refusal is None and backend.page is earlier + else: + assert backend is None and refusal is not None, visual + + +def test_a_held_page_that_fails_identity_halts() -> None: + router, context, main = _router(visual=True) + popup = _open_and_switch(router, context, main) + # A page claimed earlier on this run that no surface holds now. + spare = _Page(context, f"{ORIGIN}/popup.html", opener=main) + router._remember(spare) + router._claimed_on_run.append(spare) + router._identity_check = lambda _backend, _step: False + error = router.route(_act("step_002")) + assert error is not None and "no longer passes its identity check" in error + assert router.backends["popup_1"].page is popup + + +# -- switching --------------------------------------------------------------------- + + +def test_a_page_switch_without_identity_evidence_halts() -> None: + router, context, main = _router() + _Page(context, f"{ORIGIN}/popup.html", opener=main) + assert router.after_action_refusal(_workflow().steps[0], timeout_s=0.0) is None + backend, refusal = router.switch( + _switch("step_001", "popup_1", expect=[], surface_origin=None) + ) + assert backend is None + assert refusal is not None and "no identity evidence" in refusal + assert router.active_id == "main" + + +# -- dialogs ------------------------------------------------------------------------- + + +class _DialogBackend: + def __init__(self, seen: list[dict[str, Any]]) -> None: + self.seen = seen + self.armed: list[tuple[Any, ...]] = [] + + def arm_dialog(self, *args: Any, **kwargs: Any) -> None: + self.armed.append((args, kwargs)) + + def take_dialogs(self, _timeout_s: float = 0.0) -> list[dict[str, Any]]: + seen, self.seen = self.seen, [] + return seen + + +def _dialog_error(seen: list[dict[str, Any]], dialog: DialogResponse) -> str: + replayer = Replayer.__new__(Replayer) + replayer.backend = _DialogBackend(seen) # type: ignore[assignment] + replayer._act_without_dialog = lambda *_a, **_k: None # type: ignore[method-assign] + step = Step( + id="s", intent="delete", action=ActionKind.KEY, key="Enter", dialog=dialog + ) + error = Replayer._act(replayer, step, None, {}) + assert error is not None + return error + + +def test_a_dialog_mismatch_error_quotes_no_dialog_text(monkeypatch) -> None: + import openadapt_flow.runtime.replayer as replayer_mod + + monkeypatch.setattr(replayer_mod, "_DIALOG_WAIT_S", 0.0) + live = "Discharge patient Jane Roe (MRN 991)?" + dialog = DialogResponse(response="accept", message="Discharge this patient?") + error = _dialog_error([{"message": live, "matched": False}], dialog) + assert "dialog_mismatch" in error + assert "Jane" not in error and "991" not in error + assert "Discharge" not in error + missing = _dialog_error([], dialog) + assert "dialog_missing" in missing and "Discharge" not in missing + + +def test_a_dialog_pattern_is_armed_as_a_pattern(monkeypatch) -> None: + import openadapt_flow.runtime.replayer as replayer_mod + + monkeypatch.setattr(replayer_mod, "_DIALOG_WAIT_S", 0.0) + dialog = DialogResponse(response="accept", message_pattern=r"Delete\ record\ \d+\?") + replayer = Replayer.__new__(Replayer) + backend = _DialogBackend([{"message": "Delete record 7?", "matched": True}]) + replayer.backend = backend # type: ignore[assignment] + replayer._act_without_dialog = lambda *_a, **_k: None # type: ignore[method-assign] + step = Step( + id="s", intent="delete", action=ActionKind.KEY, key="Enter", dialog=dialog + ) + assert Replayer._act(replayer, step, None, {}) is None + assert backend.armed[0] == ( + ("accept", None), + {"message_pattern": r"Delete\ record\ \d+\?"}, + ) + + +def test_a_dialog_message_is_sealed_without_its_values() -> None: + sealed = _dialog_identity( + {"response": "accept", "message": "Remove Jane Roe from list 2024-01-03?"}, + ("Jane Roe",), + ) + assert sealed.message is None and sealed.message_pattern is not None + dumped = sealed.model_dump_json() + assert "Jane" not in dumped and "2024" not in dumped + pattern = re.compile(sealed.message_pattern, re.DOTALL) + assert pattern.fullmatch("Remove John Poe from list 2025-11-30?") + assert not pattern.fullmatch("Remove everyone from every list?") + # A message with no value and no digit is kept exactly. + kept = _dialog_identity({"response": "dismiss", "message": "Sure?"}, ()) + assert kept == DialogResponse(response="dismiss", message="Sure?") + + +def test_a_dialog_response_needs_exactly_one_identity() -> None: + with pytest.raises(ValueError, match="exactly one"): + DialogResponse(response="accept") + with pytest.raises(ValueError, match="exactly one"): + DialogResponse(response="accept", message="a", message_pattern="a") + + +# -- URL patterns ---------------------------------------------------------------------- + + +def test_url_patterns_generalize_identifier_segments() -> None: + pattern = _surface_url_pattern( + "https://ehr.example/patient/12345/chart?state=abc", () + ) + assert "12345" not in pattern and "state" not in pattern + assert re.fullmatch(pattern, "https://ehr.example/patient/777/chart") + assert not re.fullmatch(pattern, "https://ehr.example/billing/777/chart") + token = "AbCdEfGhIjKlMnOpQrStUvWxYz" + pattern = _surface_url_pattern( + f"https://a.example/api/v2/s/{token}/order-history", () + ) + assert token not in pattern + assert r"/api/v2/" in pattern and r"order\-history" in pattern + + +# -- layout drift ----------------------------------------------------------------------- + + +def test_layout_drift_follows_the_active_surface() -> None: + class _Router: + def active_kind(self) -> str: + return "rdp" + + replayer = Replayer.__new__(Replayer) + replayer.layout_drift = None + replayer._layout_target_kind = None + replayer.backend = object() # type: ignore[assignment] + replayer._surface_router = _Router() # type: ignore[assignment] + # A web-first bundle on its RDP segment keeps the check. + workflow = Workflow( + name="wf", + surface="web", + steps=[Step(id="s", intent="wait", action=ActionKind.WAIT)], + ) + assert Replayer._layout_drift_on(replayer, workflow) is True + replayer._surface_router = None + assert Replayer._layout_drift_on(replayer, workflow) is False diff --git a/packages/flow/tests/test_multi_surface_governed.py b/packages/flow/tests/test_multi_surface_governed.py new file mode 100644 index 000000000..382ea1650 --- /dev/null +++ b/packages/flow/tests/test_multi_surface_governed.py @@ -0,0 +1,324 @@ +"""Governed, durable, and qualified runs of multi-surface bundles. + +docs/MULTI_SURFACE.md, "Governance and verification". A multi-surface run +reaches VERIFIED under exactly the single-surface rules (governed +authorization, durable checkpoints, identity on writes, an effect confirmed by +an independent read of the system of record), plus one rule of its own: every +step result records the surface it ran on, and that record must match the +sealed contract. Screen-only evidence never reaches VERIFIED on any surface. +""" + +from __future__ import annotations + +import hashlib +import json +from types import SimpleNamespace + +from openadapt_flow.execution_profiles import ( + ExecutionOutcome, + ExecutionProfile, + classify_execution_outcome, +) +from openadapt_flow.ir import ( + ActionKind, + ObservedSurfaceEnvironment, + Postcondition, + PostconditionKind, + RunReport, + Step, + StepResult, + SurfaceSpec, + Workflow, +) +from openadapt_flow.qualification import ( + EnvironmentBoundary, + QualificationProject, + _surface_environment_evidence_refusal, +) +from openadapt_flow.qualification_environment import ( + QualificationEnvironmentObservation, +) +from openadapt_flow.run_gate import build_runtime_authorization +from openadapt_flow.runtime.durable import CheckpointStore +from openadapt_flow.runtime.replayer import Replayer +from openadapt_flow.verification import VerificationTier +from tests.test_execution_profiles import ( + _gate, + _key_workflow, + _ReadyVision, + _sealed, + _TieredVerifier, +) +from tests.test_replayer import FakeBackend + + +class _WindowsFake(FakeBackend): + surface_kind = "windows" + + +class _PersistedVerifier(_TieredVerifier): + verification_tier = VerificationTier.PERSISTED_STATE_REACQUISITION + + +def _multi_surface_write_workflow() -> Workflow: + """Primary web surface, then a switch to Windows and a governed write.""" + + base = _key_workflow("multi-surface-governed", with_effect=True) + write = base.steps[0].model_copy(update={"surface": "windows_1"}) + switch = Step( + id="to_windows", + intent="switch to surface 'windows_1'", + action=ActionKind.SWITCH_SURFACE, + surface="windows_1", + expect=[ + Postcondition(kind=PostconditionKind.TEXT_PRESENT, text="Desktop Home") + ], + ) + return Workflow( + name=base.name, + surfaces=[ + SurfaceSpec(id="main", kind="web"), + SurfaceSpec(id="windows_1", kind="windows"), + ], + steps=[switch, write], + ) + + +def _governed_run(tmp_path, *, verifier=None, admit_with=None): + workflow, bundle = _sealed( + tmp_path, _multi_surface_write_workflow(), encrypted=False + ) + verifier = verifier or _TieredVerifier() + gate = _gate( + workflow, + bundle, + ExecutionProfile.STANDARD, + verifier=admit_with or verifier, + durable=True, + ) + assert gate.passed, gate.render() + primary, windows = FakeBackend(), _WindowsFake() + vision = _ReadyVision() + vision.text_results["Desktop Home"] = object() + replayer = Replayer( + primary, + vision=vision, + effect_verifier=verifier, + governed_authorization=build_runtime_authorization(workflow, gate), + durable=True, + require_settled=True, + surface_backends={"windows_1": windows}, + ) + report = replayer.run(workflow, bundle_dir=bundle, run_dir=tmp_path / "run") + return report, workflow, primary, windows, replayer + + +def test_governed_durable_multi_surface_run_reaches_verified(tmp_path) -> None: + report, workflow, primary, windows, replayer = _governed_run(tmp_path) + + assert report.execution_outcome == ExecutionOutcome.VERIFIED.value, [ + (r.step_id, r.ok, r.error) for r in report.results + ] + assert report.execution_profile == "standard" + # Each result records the surface it ran on; the write drove only the + # Windows backend, and the primary is restored after the run. + assert [(r.step_id, r.surface) for r in report.results] == [ + ("to_windows", "windows_1"), + ("submit", "windows_1"), + ] + assert windows.actions and not primary.actions + assert replayer.backend is primary + write = report.results[1] + assert write.effect_verified is True + assert write.identity is not None and write.identity.status == "verified" + # The durable checkpoint of the confirmed write exists. + store = CheckpointStore(tmp_path / "run") + assert store.last_checkpoint() is not None + # report.json carries the per-step surface record. + saved = json.loads((tmp_path / "run" / "report.json").read_text()) + assert [r.get("surface") for r in saved["results"]] == ["windows_1"] * 2 + + +def test_surface_record_must_match_the_contract(tmp_path) -> None: + report, workflow, *_ = _governed_run(tmp_path) + assert report.execution_outcome == ExecutionOutcome.VERIFIED.value + + for tampered_surface in (None, "main"): + tampered = report.model_copy(deep=True) + tampered.results[1].surface = tampered_surface + assert ( + classify_execution_outcome(tampered, workflow, ExecutionProfile.STANDARD) + is ExecutionOutcome.COMPLETED_UNVERIFIED + ) + + +def test_screen_reacquisition_never_verifies_a_multi_surface_run(tmp_path) -> None: + # Admitted with a system-of-record verifier, but the run's verifier only + # re-reads the screen: VERIFIED is out of reach, on any surface. + report, *_ = _governed_run( + tmp_path, verifier=_PersistedVerifier(), admit_with=_TieredVerifier() + ) + assert report.execution_outcome != ExecutionOutcome.VERIFIED.value + + +def test_single_surface_results_carry_no_surface_record(tmp_path) -> None: + workflow, bundle = _sealed( + tmp_path, _key_workflow("single", with_effect=True), encrypted=False + ) + verifier = _TieredVerifier() + gate = _gate( + workflow, bundle, ExecutionProfile.STANDARD, verifier=verifier, durable=True + ) + report = Replayer( + FakeBackend(), + vision=_ReadyVision(), + effect_verifier=verifier, + governed_authorization=build_runtime_authorization(workflow, gate), + durable=True, + require_settled=True, + ).run(workflow, bundle_dir=bundle, run_dir=tmp_path / "run") + assert report.execution_outcome == ExecutionOutcome.VERIFIED.value + assert all(r.surface is None for r in report.results) + saved = json.loads((tmp_path / "run" / "report.json").read_text()) + assert all("surface" not in r for r in saved["results"]) + assert "observed_surface_environments" not in saved + # A surface record on a single-surface report is not evidence for it. + tampered = report.model_copy(deep=True) + tampered.results[0].surface = "main" + assert ( + classify_execution_outcome(tampered, workflow, ExecutionProfile.STANDARD) + is ExecutionOutcome.COMPLETED_UNVERIFIED + ) + + +# -- qualification: one environment boundary per surface --------------------- + +_DIGEST = "a" * 64 + + +def _boundary(kind: str, identity: str) -> EnvironmentBoundary: + return EnvironmentBoundary( + target_kind=kind, + application=identity, + application_identity=identity, + application_version="1.0", + environment_observer_id="test-observer", + environment_observer_contract_sha256="b" * 64, + environment_digest=_DIGEST, + runtime_version="test", + ) + + +def _project(**surface_environments) -> QualificationProject: + return QualificationProject( + environment=_boundary("web", "https://app.example.test"), + surface_environments=surface_environments, + ) + + +def test_single_surface_project_contract_bytes_are_unchanged() -> None: + project = _project() + assert "surface_environments" not in project.model_dump(mode="json") + with_surface = _project(windows_1=_boundary("windows", "Desk.exe")) + assert with_surface.contract_sha256() != project.contract_sha256() + + +def _case_replayer(project: QualificationProject) -> tuple[Replayer, Workflow]: + workflow = _multi_surface_write_workflow().model_copy( + update={"qualification": project} + ) + replayer = Replayer(FakeBackend(), surface_backends={"windows_1": _WindowsFake()}) + replayer.governed_authorization = SimpleNamespace(qualification_case_id="case-1") + return replayer, workflow + + +def test_qualification_case_refuses_an_unqualified_surface() -> None: + replayer, workflow = _case_replayer(_project()) + refusal = replayer._begin_surfaces(workflow, resume=False) + assert "no environment boundary for surface 'windows_1'" in (refusal or "") + + replayer, workflow = _case_replayer( + _project(windows_1=_boundary("rdp", "Desk.exe")) + ) + refusal = replayer._begin_surfaces(workflow, resume=False) + assert "declared windows" in (refusal or "") + + +class _ObservedWindows(_WindowsFake): + def __init__(self, identity: str) -> None: + super().__init__() + self.identity = identity + self.guard = None + + def set_qualification_input_guard(self, guard) -> None: + self.guard = guard + + +class _Observer: + observer_id = "test-observer" + contract_sha256 = "b" * 64 + + def observe(self, backend, target_kind): + return QualificationEnvironmentObservation( + target_kind=target_kind, + application_identity=backend.identity, + application_version="1.0", + session_identity_sha256="c" * 64, + environment_digest=_DIGEST, + ) + + +def _switch_into(identity: str): + project = _project(windows_1=_boundary("windows", "Desk.exe")) + replayer, workflow = _case_replayer(project) + windows = _ObservedWindows(identity) + replayer.surface_backends = {"windows_1": windows} + replayer.qualification_environment_observer = _Observer() + assert replayer._begin_surfaces(workflow, resume=False) is None + replayer.backend = windows + refusal = replayer._bind_surface_qualification_environment(workflow, "windows_1") + return refusal, replayer, windows + + +def test_switch_observes_the_surface_against_its_boundary() -> None: + refusal, replayer, windows = _switch_into("Desk.exe") + assert refusal is None + observed = replayer._surface_environment_observations["windows_1"] + assert observed.application_sha256 == hashlib.sha256(b"Desk.exe").hexdigest() + # The input-edge guard now rechecks THIS surface before each input. + assert windows.guard is not None + windows.guard() # unchanged environment: passes + + refusal, _replayer, windows = _switch_into("Other.exe") + assert "does not match the project" in (refusal or "") + assert windows.guard is None + + +def test_case_evidence_needs_a_matching_observation_per_used_surface() -> None: + project = _project(windows_1=_boundary("windows", "Desk.exe")) + workflow = _multi_surface_write_workflow() + refusal, replayer, _ = _switch_into("Desk.exe") + observed = replayer._surface_environment_observations["windows_1"] + report = RunReport( + workflow_name=workflow.name, + started_at="2026-10-17T00:00:00Z", + success=True, + results=[ + StepResult(step_id="to_windows", intent="s", ok=True, surface="windows_1"), + StepResult(step_id="submit", intent="w", ok=True, surface="windows_1"), + ], + observed_surface_environments={"windows_1": observed}, + ) + assert _surface_environment_evidence_refusal(report, workflow, project) is None + + missing = report.model_copy(update={"observed_surface_environments": {}}) + assert _surface_environment_evidence_refusal(missing, workflow, project) + + forged = report.model_copy(deep=True) + forged.observed_surface_environments["windows_1"] = ObservedSurfaceEnvironment( + **{ + **observed.model_dump(), + "application_sha256": hashlib.sha256(b"Other.exe").hexdigest(), + } + ) + assert _surface_environment_evidence_refusal(forged, workflow, project) diff --git a/packages/flow/tests/test_multi_surface_identity.py b/packages/flow/tests/test_multi_surface_identity.py new file mode 100644 index 000000000..30dbe2b04 --- /dev/null +++ b/packages/flow/tests/test_multi_surface_identity.py @@ -0,0 +1,223 @@ +"""Surfaces bound by identity, not handles (docs/MULTI_SURFACE.md). + +Uses the popup + tab fixture app and bundle from test_multi_surface_e2e.py. + +- A durable run that pauses on the report tab resumes in a NEW Replayer: the + checkpoint carries surface names and last-seen observations, never page + handles, and the resume re-binds the tab by the step's identity check. The + pause after a consequential click is a "Check the record" stop, and a resume + still needs the operator's approval. +- A second page that passes the same identity check makes the re-bind + ambiguous: the run halts and drives neither page. +- A page that no longer passes the identity check is a routing mismatch: the + run halts instead of acting on it. +""" + +from __future__ import annotations + +import shutil +from pathlib import Path + +import pytest + +pytest.importorskip("playwright.sync_api") + +import tests.test_multi_surface_e2e as _e2e # noqa: E402 +from openadapt_flow.backends.playwright_backend import PlaywrightBackend # noqa: E402 +from openadapt_flow.ir import Workflow # noqa: E402 +from openadapt_flow.runtime.durable import ( # noqa: E402 + ApprovalRequired, + CheckpointStore, + StateDiverged, + resume, +) +from openadapt_flow.runtime.effects import Effect, EffectKind # noqa: E402 +from openadapt_flow.runtime.replayer import Replayer # noqa: E402 +from tests.test_durable_runtime import FakeSoRVerifier, _approval # noqa: E402 + +pytestmark = pytest.mark.timeout(600) + +app_url = _e2e.app_url +browser = _e2e.browser +bundle = _e2e.bundle + +#: step_006 clicks "Mark Reviewed" on the report tab (surface tab_1). +PAUSE_STEP = "step_006" + + +def _durable_bundle(bundle, tmp_path: Path) -> Path: + """The e2e bundle with a system-of-record effect on the report-tab step.""" + + source, _ = bundle + target = tmp_path / "bundle" + shutil.copytree(source, target) + workflow = Workflow.load(target) + steps = [ + step.model_copy( + update={ + "effects": [ + Effect( + kind=EffectKind.RECORD_WRITTEN, + match={"step": step.id}, + expected_count=1, + timeout_s=0.5, + ) + ], + "risk": "irreversible", + } + ) + if step.id == PAUSE_STEP + else step + for step in workflow.steps + ] + workflow.model_copy(update={"steps": steps}).save(target) + return target + + +def _pause_on_the_tab(browser, app_url, bundle, tmp_path): + bundle_dir = _durable_bundle(bundle, tmp_path) + run_dir = tmp_path / "run" + verifier = FakeSoRVerifier() + verifier.refute.add((("step", PAUSE_STEP),)) + page = _e2e._fresh_page(browser, app_url) + report = Replayer( + PlaywrightBackend(page), + durable=True, + effect_verifier=verifier, + poll_interval_s=0.05, + ).run(Workflow.load(bundle_dir), bundle_dir=bundle_dir, run_dir=run_dir) + return report, page, bundle_dir, run_dir, verifier + + +def test_resume_rebinds_surfaces_by_identity(bundle, app_url, browser, tmp_path): + report, page, bundle_dir, run_dir, verifier = _pause_on_the_tab( + browser, app_url, bundle, tmp_path + ) + try: + assert not report.success, _e2e._summary(report) + assert report.results[-1].step_id == PAUSE_STEP + # A consequential click whose save was not confirmed: a person checks + # the record before anything is retried. + assert report.transaction_outcome == "RECONCILIATION_REQUIRED" + + store = CheckpointStore(run_dir) + last = store.last_checkpoint() + assert last is not None and last.step_id == "step_005" + # Names and observations, never handles. + assert last.active_surface == "tab_1" + assert last.surface == "tab_1" + assert set(last.surface_observations) == {"main", "popup_1", "tab_1"} + tab_seen = last.surface_observations["tab_1"] + assert tab_seen.origin == app_url.lower() and tab_seen.screen_phash + # An observation holds no URL path, title, screen text, or handle. + assert set(tab_seen.model_dump()) == { + "surface", + "kind", + "observed_after_step", + "origin", + "screen_phash", + } + + verifier.refute.clear() + fresh = Replayer( + PlaywrightBackend(page), effect_verifier=verifier, poll_interval_s=0.05 + ) + # The "Check the record" stop: no resume without the operator. + with pytest.raises(ApprovalRequired): + resume(run_dir, fresh, approval=None) + + resumed = resume(run_dir, fresh, approval=_approval(bundle_dir, run_dir)) + assert resumed.success, _e2e._summary(resumed) + assert resumed.model_calls == 0 + by_id = {r.step_id: r for r in resumed.results} + assert by_id[PAUSE_STEP].surface == "tab_1" + assert by_id["step_008"].surface == "main" + assert page.locator("#done").inner_text() == "Referral Finished" + finally: + page.context.close() + + +def test_ambiguous_rebind_on_resume_halts(bundle, app_url, browser, tmp_path): + report, page, bundle_dir, run_dir, verifier = _pause_on_the_tab( + browser, app_url, bundle, tmp_path + ) + try: + assert not report.success + # A second report tab that looks exactly like the first. + with page.expect_popup() as extra_info: + page.click("#open-report") + extra = extra_info.value + extra.wait_for_load_state("load") + verifier.refute.clear() + fresh = Replayer( + PlaywrightBackend(page), effect_verifier=verifier, poll_interval_s=0.05 + ) + with pytest.raises(StateDiverged, match="2 open browser pages pass"): + resume(run_dir, fresh, approval=_approval(bundle_dir, run_dir)) + # Neither report tab, nor the main page, was driven. + assert extra.locator("#ack").inner_text() == "" + assert page.locator("#done").inner_text() == "" + assert CheckpointStore(run_dir).read_pending() is not None + finally: + page.context.close() + + +def test_page_routing_mismatch_halts(bundle, app_url, browser, tmp_path): + report, page, bundle_dir, run_dir, verifier = _pause_on_the_tab( + browser, app_url, bundle, tmp_path + ) + try: + assert not report.success + # The only candidate for the report tab now shows another document. + report_tab = page.context.pages[2] + report_tab.goto(f"{app_url}/promo.html") + verifier.refute.clear() + fresh = Replayer( + PlaywrightBackend(page), effect_verifier=verifier, poll_interval_s=0.05 + ) + with pytest.raises(StateDiverged, match="no open browser page passes"): + resume(run_dir, fresh, approval=_approval(bundle_dir, run_dir)) + assert page.locator("#done").inner_text() == "" + finally: + page.context.close() + + +def test_step_routing_binds_only_the_page_that_passes_its_identity( + bundle, app_url, browser +): + """Per-step routing inside the browser, without a checkpoint.""" + + from openadapt_flow.runtime.surfaces import SurfaceRouter + + bundle_dir, workflow = bundle + page = _e2e._fresh_page(browser, app_url) + try: + with page.expect_popup() as info: + page.click("#open-report") + report_tab = info.value + report_tab.wait_for_load_state("load") + replayer = Replayer(PlaywrightBackend(page)) + replayer._surface_identity_context = (workflow, bundle_dir) + step = next(s for s in workflow.steps if s.id == PAUSE_STEP) + + router = SurfaceRouter( + workflow, + PlaywrightBackend(page), + identity_check=replayer._surface_identity_check, + resume_active="tab_1", + ) + assert router.route(step) is None + assert router.active_backend.page is report_tab + + report_tab.goto(f"{app_url}/promo.html") + mismatch = SurfaceRouter( + workflow, + PlaywrightBackend(page), + identity_check=replayer._surface_identity_check, + resume_active="tab_1", + ) + error = mismatch.route(step) + assert error is not None and "no open browser page passes" in error + assert mismatch.active_backend is None + finally: + page.context.close() diff --git a/packages/flow/tests/test_multi_surface_interactive.py b/packages/flow/tests/test_multi_surface_interactive.py new file mode 100644 index 000000000..d02131647 --- /dev/null +++ b/packages/flow/tests/test_multi_surface_interactive.py @@ -0,0 +1,99 @@ +"""Interactive ``record --url`` follows popups and tabs as surfaces. + +The same fixture app as test_multi_surface_e2e.py: a ``window.open`` popup and +a ``target=_blank`` tab. A scripted user drives the OWNED (headless) browser +the InteractiveRecorder launched; nothing tells the recorder about the extra +pages. It must claim each page as a surface when it opens and record a +surface switch when the user acts in another page, producing the same bundle +shape the scripted Recorder produces, which then replays with zero model +calls. See docs/MULTI_SURFACE.md ("Interactive recording"). +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +pytest.importorskip("playwright.sync_api") + +import tests.test_multi_surface_e2e as _e2e # noqa: E402 +from openadapt_flow.compiler import compile_recording # noqa: E402 +from openadapt_flow.interactive_recorder import record_interactive # noqa: E402 +from openadapt_flow.ir import ActionKind, Workflow # noqa: E402 + +pytestmark = pytest.mark.timeout(600) + +app_url = _e2e.app_url +browser = _e2e.browser + + +def _drive(page, pump) -> None: + def settle(n: int = 3) -> None: + for _ in range(n): + pump() + + page.wait_for_selector("#open-popup", state="visible", timeout=20000) + with page.expect_popup() as popup_info: + page.click("#open-popup") + popup = popup_info.value + popup.wait_for_load_state("load") + settle() + popup.click("#approve") + settle() + with page.expect_popup() as tab_info: + page.click("#open-report") + tab = tab_info.value + tab.wait_for_load_state("load") + settle() + tab.click("#reviewed") + settle() + page.click("#finish") + settle(5) + + +@pytest.fixture(scope="module") +def bundle(app_url: str, tmp_path_factory) -> tuple[Path, Workflow]: + root = tmp_path_factory.mktemp("interactive_multi_surface") + recording = record_interactive( + app_url, root / "recording", headless=True, script=_drive + ) + workflow = compile_recording(recording, root / "bundle", name="interactive-ms") + return root / "bundle", workflow + + +def test_interactive_recording_has_the_scripted_bundle_shape(bundle) -> None: + _bundle_dir, workflow = bundle + assert [s.id for s in workflow.surfaces] == ["main", "popup_1", "tab_1"] + popup, tab = workflow.surfaces[1], workflow.surfaces[2] + assert popup.opened_by_step == "step_000" and popup.opener_link == "window_opener" + assert tab.opener_link == "none" + assert [(s.action, s.surface) for s in workflow.steps] == [ + (ActionKind.CLICK, "main"), + (ActionKind.SWITCH_SURFACE, "popup_1"), + (ActionKind.CLICK, "popup_1"), + (ActionKind.SWITCH_SURFACE, "main"), + (ActionKind.CLICK, "main"), + (ActionKind.SWITCH_SURFACE, "tab_1"), + (ActionKind.CLICK, "tab_1"), + (ActionKind.SWITCH_SURFACE, "main"), + (ActionKind.CLICK, "main"), + ] + assert workflow.steps[0].opens == ["popup_1"] + assert workflow.steps[4].opens == ["tab_1"] + + +def test_interactive_multi_surface_bundle_replays_with_zero_model_calls( + bundle, app_url: str, browser: object, tmp_path +) -> None: + bundle_dir, _ = bundle + report, page = _e2e._replay(browser, app_url, bundle_dir, tmp_path / "run") + try: + assert report.success, _e2e._summary(report) + assert report.model_calls == 0 + assert page.locator("#popup-status").inner_text() == "Approval Granted" + assert page.locator("#done").inner_text() == "Referral Finished" + report_page = page.context.pages[2] + assert report_page.locator("#ack").inner_text() == "Review Recorded Today" + finally: + page.context.close() diff --git a/packages/flow/tests/test_multi_surface_selfclose.py b/packages/flow/tests/test_multi_surface_selfclose.py new file mode 100644 index 000000000..0b9655cea --- /dev/null +++ b/packages/flow/tests/test_multi_surface_selfclose.py @@ -0,0 +1,166 @@ +"""A popup that closes itself, then an explicit switch back (docs/MULTI_SURFACE.md). + +The fixture's sign-in popup writes back to its opener and calls +``window.close()``. Flow never returns to the opener on its own: both the +scripted and the interactive recorder write an explicit SWITCH_SURFACE back to +the main page, with the identity evidence of that page, and replay crosses +the closed popup only through that switch. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +pytest.importorskip("playwright.sync_api") + +import tests.test_multi_surface_e2e as _e2e # noqa: E402 +from openadapt_flow.backends.playwright_backend import PlaywrightBackend # noqa: E402 +from openadapt_flow.compiler import compile_recording # noqa: E402 +from openadapt_flow.interactive_recorder import record_interactive # noqa: E402 +from openadapt_flow.ir import ActionKind, Workflow # noqa: E402 +from openadapt_flow.recorder import Recorder # noqa: E402 + +pytestmark = pytest.mark.timeout(600) + +app_url = _e2e.app_url +browser = _e2e.browser + +QUERY = "/?popup=/selfclose.html" +SHAPE = [ + (ActionKind.CLICK, "main"), + (ActionKind.SWITCH_SURFACE, "popup_1"), + (ActionKind.CLICK, "popup_1"), + (ActionKind.CLICK, "popup_1"), + (ActionKind.SWITCH_SURFACE, "main"), + (ActionKind.CLICK, "main"), +] + + +def _assert_shape(workflow: Workflow) -> None: + assert [s.id for s in workflow.surfaces] == ["main", "popup_1"] + assert [(s.action, s.surface) for s in workflow.steps] == SHAPE + back = workflow.steps[4] + # The explicit return carries the main page's own identity evidence. + assert back.surface_origin is not None or back.expect + assert back.identity_armed is True + + +def _replay_ok(browser, url: str, bundle_dir: Path, run_dir: Path) -> None: + report, page = _e2e._replay(browser, url, bundle_dir, run_dir) + try: + assert report.success, _e2e._summary(report) + assert report.model_calls == 0 + assert page.locator("#popup-status").inner_text() == "Signed In Elsewhere" + assert page.locator("#done").inner_text() == "Referral Finished" + # The popup really closed; only the main page is left. + assert [p for p in page.context.pages if not p.is_closed()] == [page] + assert [r.surface for r in report.results] == [s for _, s in SHAPE] + finally: + page.context.close() + + +def _drive(page, pump) -> None: + def settle(n: int = 3) -> None: + for _ in range(n): + pump() + + page.wait_for_selector("#open-popup", state="visible", timeout=20000) + with page.expect_popup() as info: + page.click("#open-popup") + popup = info.value + popup.wait_for_load_state("load") + settle() + popup.click("#agree") + settle() + popup.click("#approve-close") + for _ in range(100): + if popup.is_closed(): + break + page.wait_for_timeout(50) + assert popup.is_closed() + settle() + page.click("#finish") + settle(5) + + +@pytest.fixture(scope="module") +def interactive_bundle(app_url, tmp_path_factory) -> tuple[Path, Workflow]: + # Recorded before the module's shared browser starts (one sync Playwright + # session at a time). + root = tmp_path_factory.mktemp("selfclose_interactive") + recording = record_interactive( + app_url + QUERY, root / "recording", headless=True, script=_drive + ) + workflow = compile_recording(recording, root / "bundle", name="selfclose-i") + return root / "bundle", workflow + + +def test_interactive_recorder_switches_back_from_a_self_closed_popup( + interactive_bundle, app_url, browser, tmp_path +): + bundle_dir, workflow = interactive_bundle + _assert_shape(workflow) + _replay_ok(browser, app_url + QUERY, bundle_dir, tmp_path / "run") + + +def test_scripted_recorder_switches_back_from_a_self_closed_popup( + app_url, browser, tmp_path +): + url = app_url + QUERY + page = _e2e._fresh_page(browser, url) + try: + recorder = Recorder( + PlaywrightBackend(page), + tmp_path / "recording", + app_url=url, + settle_interval_s=0.05, + settle_timeout_s=2.0, + ) + recorder.click(*_e2e._center(page, "#open-popup")) + recorder.switch_surface("popup_1") + popup = recorder.surface_backend("popup_1").page # type: ignore[attr-defined] + recorder.click(*_e2e._center(popup, "#agree")) + recorder.click(*_e2e._center(popup, "#approve-close")) + assert popup.is_closed() + recorder.switch_surface("main") + recorder.click(*_e2e._center(page, "#finish")) + recording = recorder.finish() + finally: + page.context.close() + workflow = compile_recording(recording, tmp_path / "bundle", name="selfclose") + _assert_shape(workflow) + _replay_ok(browser, url, tmp_path / "bundle", tmp_path / "run") + + +def test_nothing_returns_to_the_opener_on_its_own(app_url, browser, tmp_path): + """Without the recorded switch back, the bundle is not a valid workflow.""" + + url = app_url + QUERY + page = _e2e._fresh_page(browser, url) + try: + recorder = Recorder( + PlaywrightBackend(page), + tmp_path / "recording", + app_url=url, + settle_interval_s=0.05, + settle_timeout_s=2.0, + ) + recorder.click(*_e2e._center(page, "#open-popup")) + recorder.switch_surface("popup_1") + popup = recorder.surface_backend("popup_1").page # type: ignore[attr-defined] + recorder.click(*_e2e._center(popup, "#agree")) + recorder.click(*_e2e._center(popup, "#approve-close")) + recorder.switch_surface("main") + recorder.click(*_e2e._center(page, "#finish")) + recording = recorder.finish() + finally: + page.context.close() + workflow = compile_recording(recording, tmp_path / "bundle", name="selfclose") + data = workflow.model_dump(mode="json") + data["steps"] = [s for s in data["steps"] if s["id"] != workflow.steps[4].id] + with pytest.raises( + ValueError, match="a surface change must be an explicit switch_surface step" + ): + Workflow.model_validate(data) diff --git a/packages/flow/tests/test_risk.py b/packages/flow/tests/test_risk.py index c0d7e20ad..2cf159676 100644 --- a/packages/flow/tests/test_risk.py +++ b/packages/flow/tests/test_risk.py @@ -700,7 +700,11 @@ def _drafted(step: Step): @pytest.mark.parametrize( "kind", - [kind for kind in ActionKind if kind not in {ActionKind.WAIT, ActionKind.SCROLL}], + [ + kind + for kind in ActionKind + if kind not in {ActionKind.WAIT, ActionKind.SCROLL, ActionKind.SWITCH_SURFACE} + ], ) @pytest.mark.parametrize("label", [None, "日", "日日", "OK", "Close", "Save"]) @pytest.mark.parametrize("operator_override", [False, True]) diff --git a/packages/flow/tests/test_surface_bindings.py b/packages/flow/tests/test_surface_bindings.py new file mode 100644 index 000000000..8d76e0f34 --- /dev/null +++ b/packages/flow/tests/test_surface_bindings.py @@ -0,0 +1,266 @@ +"""Binding backends to backend surfaces at replay time (docs/MULTI_SURFACE.md). + +Unit tests for ``--surface NAME=KIND[:TARGET]`` and the config ``surfaces:`` +map, plus one CLI end to end: a bundle that switches from the app's tab to a +second web app replays through ``openadapt-flow replay --surface``. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from openadapt_flow.deployment import BackendConfig, DeploymentConfig +from openadapt_flow.ir import ActionKind, Step, SurfaceSpec, Workflow +from openadapt_flow.surface_bindings import ( + SurfaceBindingError, + build_surface_backends, + parse_surface_flag, + resolve_surface_configs, + unbound_backend_surfaces, +) + +_POPUP = SurfaceSpec( + id="popup_1", + kind="web", + opened_by_step="step_000", + opener="main", + open_order=0, + opener_link="window_opener", + origin="http://127.0.0.1:8000", + url_pattern=r"http://127\.0\.0\.1:8000/popup\.html(?:[?#].*)?", +) + + +def _workflow() -> Workflow: + return Workflow( + name="wf", + surfaces=[ + SurfaceSpec(id="main", kind="web"), + _POPUP, + SurfaceSpec(id="windows_1", kind="windows"), + SurfaceSpec(id="citrix_1", kind="citrix"), + ], + steps=[ + Step( + id="step_000", + intent="open", + action=ActionKind.WAIT, + opens=["popup_1"], + ) + ], + ) + + +@pytest.mark.parametrize( + ("text", "expected"), + [ + ("windows_1=windows", ("windows_1", "windows", None)), + ( + "windows_1=windows:http://10.0.0.5:5001", + ("windows_1", "windows", "http://10.0.0.5:5001"), + ), + ("windows_1=http://10.0.0.5:5001", ("windows_1", None, "http://10.0.0.5:5001")), + ("citrix_1=citrix", ("citrix_1", "citrix", None)), + ( + "app_2=web:https://portal.example.test/", + ("app_2", "web", "https://portal.example.test/"), + ), + ], +) +def test_parse_surface_flag(text, expected) -> None: + assert parse_surface_flag(text) == expected + + +@pytest.mark.parametrize("text", ["windows_1", "=windows", "windows_1="]) +def test_parse_surface_flag_refuses_malformed(text) -> None: + with pytest.raises(SurfaceBindingError): + parse_surface_flag(text) + + +def test_kind_defaults_from_the_bundle_and_flag_overrides_config() -> None: + configured = { + "windows_1": BackendConfig.model_validate( + {"agent_url": "http://config:5001", "agent_token": "t"} + ) + } + resolved = resolve_surface_configs( + _workflow(), ["windows_1=http://flag:5001", "citrix_1=citrix"], configured + ) + assert resolved["windows_1"].kind == "windows" + assert resolved["windows_1"].agent_url == "http://flag:5001" + assert resolved["windows_1"].agent_token == "t" # kept from the config + assert resolved["citrix_1"].kind == "citrix" + assert unbound_backend_surfaces(_workflow(), resolved) == [] + assert unbound_backend_surfaces(_workflow(), {}) == ["windows_1", "citrix_1"] + + +@pytest.mark.parametrize( + ("flag", "message"), + [ + ("nope=windows", "not declared"), + ("main=web", "primary surface"), + ("popup_1=web", "needs no binding"), + ("windows_1=rdp:10.0.0.5", "declared windows"), + ], +) +def test_resolve_refuses_a_binding_the_bundle_cannot_use(flag, message) -> None: + with pytest.raises(SurfaceBindingError, match=message): + resolve_surface_configs(_workflow(), [flag]) + + +def test_config_surfaces_map_round_trips_and_is_omitted_when_empty() -> None: + assert "surfaces" not in DeploymentConfig().model_dump() + cfg = DeploymentConfig.model_validate( + {"surfaces": {"windows_1": {"agent_url": "http://x:5001"}}} + ) + resolved = resolve_surface_configs(_workflow(), [], cfg.surfaces) + assert resolved["windows_1"].kind == "windows" + + +def test_build_uses_the_factory_and_opens_web_surfaces_in_the_browser() -> None: + calls: list = [] + + def fake_build(cfg, page=None): + calls.append((cfg.kind, page)) + return object() + + pages: list[str] = [] + backends = build_surface_backends( + { + "windows_1": BackendConfig(kind="windows", agent_url="http://x:1"), + "app_2": BackendConfig(kind="web", url="http://y/"), + }, + new_page=lambda url: pages.append(url) or f"page:{url}", + build=fake_build, + ) + assert set(backends) == {"windows_1", "app_2"} + assert calls == [("windows", None), ("web", "page:http://y/")] + with pytest.raises(SurfaceBindingError, match="needs a browser"): + build_surface_backends( + {"app_2": BackendConfig(kind="web", url="http://y/")}, build=fake_build + ) + + +# -- CLI end to end ------------------------------------------------------------ + +pw = pytest.importorskip("playwright.sync_api") + +import tests.test_multi_surface_e2e as _e2e # noqa: E402 +from openadapt_flow.__main__ import main as cli_main # noqa: E402 +from openadapt_flow.backends.playwright_backend import PlaywrightBackend # noqa: E402 +from openadapt_flow.compiler import compile_recording # noqa: E402 +from openadapt_flow.recorder import Recorder # noqa: E402 + +_center = _e2e._center +_fresh_page = _e2e._fresh_page +#: The e2e module's local fixture app (served on a free loopback port). +app_url = _e2e.app_url + + +@pytest.fixture(scope="module") +def two_app_bundle(app_url: str, tmp_path_factory) -> Path: + """Main app tab, then a separately opened second app (a backend surface). + + Uses its own Playwright session, closed before the CLI tests start theirs + (the sync API cannot nest). + """ + + from openadapt_flow._browser_setup import ensure_chromium_installed + + ensure_chromium_installed() + root = tmp_path_factory.mktemp("surface_cli") + with pw.sync_playwright() as playwright: + browser = playwright.chromium.launch(headless=True) + try: + _record_two_apps(browser, app_url, root) + finally: + browser.close() + compile_recording(root / "recording", root / "bundle", name="two-apps") + return root / "bundle" + + +def _record_two_apps(browser, app_url: str, root: Path) -> None: + main_page = _fresh_page(browser, app_url) + other_page = _fresh_page(browser, f"{app_url}/report.html") + try: + recorder = Recorder( + PlaywrightBackend(main_page), + root / "recording", + app_url=app_url, + settle_interval_s=0.05, + settle_timeout_s=2.0, + ) + recorder.switch_surface("app_2", backend=PlaywrightBackend(other_page)) + recorder.click(*_center(other_page, "#reviewed")) + recorder.switch_surface("main") + recorder.click(*_center(main_page, "#finish")) + recorder.finish() + finally: + main_page.context.close() + other_page.context.close() + + +def _report(run_dir: Path) -> dict: + return json.loads((run_dir / "report.json").read_text()) + + +def test_cli_replay_binds_a_web_backend_surface( + two_app_bundle: Path, app_url: str, tmp_path +) -> None: + workflow = Workflow.load(two_app_bundle) + assert [(s.id, s.kind, s.is_page) for s in workflow.surfaces] == [ + ("main", "web", False), + ("app_2", "web", False), + ] + run_dir = tmp_path / "bound" + code = cli_main( + [ + "replay", + str(two_app_bundle), + "--url", + app_url, + "--surface", + f"app_2={app_url}/report.html", + "--run-dir", + str(run_dir), + ] + ) + report = _report(run_dir) + assert code == 0, [(r["step_id"], r.get("error")) for r in report["results"]] + assert report["model_calls"] == 0 + assert [r.get("surface") for r in report["results"]] == [ + "app_2", + "app_2", + "main", + "main", + ] + + +def test_cli_replay_without_a_binding_halts_at_the_switch( + two_app_bundle: Path, app_url: str, tmp_path +) -> None: + run_dir = tmp_path / "unbound" + code = cli_main( + ["replay", str(two_app_bundle), "--url", app_url, "--run-dir", str(run_dir)] + ) + assert code == 1 + last = _report(run_dir)["results"][-1] + assert last["step_id"] == "step_000" + assert "--surface app_2=" in last["error"] + + +def test_cli_refuses_an_undeclared_surface_name(two_app_bundle: Path, app_url) -> None: + with pytest.raises(SystemExit, match="not declared"): + cli_main( + [ + "replay", + str(two_app_bundle), + "--url", + app_url, + "--surface", + "nope=web:http://127.0.0.1:1/", + ] + ) diff --git a/tests/test_cli_seal.py b/tests/test_cli_seal.py index 257bba257..0be96cc67 100644 --- a/tests/test_cli_seal.py +++ b/tests/test_cli_seal.py @@ -21,15 +21,27 @@ def test_seal_forwards_to_agent(monkeypatch): assert calls == [["seal", "verify", "report.json", "--json"]] +def _old_agent(monkeypatch): + # openadapt-agent 2.2+ is an alias of openadapt_flow.agent. Stand in for an + # agent without Seal checks under both names, and drop the submodule + # attribute an earlier import left on the (shared) package. + for name in ("openadapt_agent.seal_cli", "openadapt_flow.agent.seal_cli"): + monkeypatch.setitem(sys.modules, name, None) + for package in ("openadapt_agent", "openadapt_flow.agent"): + module = sys.modules.get(package) + if module is not None: + monkeypatch.delattr(module, "seal_cli", raising=False) + + def test_seal_explains_an_old_agent(monkeypatch): - monkeypatch.setitem(sys.modules, "openadapt_agent.seal_cli", None) + _old_agent(monkeypatch) result = CliRunner().invoke(main, ["seal", "verify", "x.json"]) assert result.exit_code == 1 assert "pip install --upgrade openadapt-agent" in result.output def test_seal_help_works_with_an_old_agent(monkeypatch): - monkeypatch.setitem(sys.modules, "openadapt_agent.seal_cli", None) + _old_agent(monkeypatch) result = CliRunner().invoke(main, ["seal", "--help"]) assert result.exit_code == 0 assert "openadapt seal verify" in result.output