[abrichr]

For review only. openadapt-flow is frozen for the monorepo cutover: do not merge, no release. Base is the freeze SHA f21281fa (origin/main).

What this adds

A bundle can now declare more than one surface:

• the primary surface;
• browser pages a step opens (window.open popups, target=_blank tabs);
• other backends (web → Windows/macOS/RDP/Citrix).

Each step names its surface. Moving between surfaces takes an explicit, typed SWITCH_SURFACE step, which has its own identity check. The design, the rejected alternatives, and the file:line audit of origin/main are in docs/MULTI_SURFACE.md.

Vision first: a surface is a name plus an identity check (new)

A bundle-level surface exists only at an actuator boundary: a browser driven through its pages, a local desktop, or an RDP/Citrix session. Tabs, popups, native windows and dialogs inside one actuator are screen states, handled by per-step visual identity checks. No surface kind was added.

• Binding by identity. Binding finds the one live target that passes the check. Zero or several matches halt. The check is vision first: the step's anchor template, or the switch's screen text. Structured attributes (origin, URL/title pattern, opener link) are conjoined only where the actuator exposes them. Handles are only a cache, never trusted across processes or a closed page.
• Routing. Inside the browser backend, each step goes to the page that passes its own visual check, falling back to the most recently used live page. There is no per-step screenshot of every page.
• Resume. Checkpoints store surface, active_surface and per-surface surface_observations: kind, origin, perceptual hash, and the step last seen. They hold no handles, URL paths, titles or screen text. Resume re-binds the checkpoint's surface by identity before the approval is consumed. An ambiguous or missing match raises StateDiverged with nothing driven. "Check the record" (RECONCILIATION_REQUIRED) still comes first after a save that may have gone through.
• Cloud. The handle-only refusal of managed dispatch was removed, because the bundle digest seals the surface map and the hashed deployment profile carries surfaces:. One refusal is kept because it guards something real: a managed run will not start with an unbound backend surface.
• Explicit switch back. A popup that closes itself shows a blank frame. Both recorders write the return to the opener as an explicit switch with its identity check, and nothing returns automatically.
• Attach mode (--cdp) follows popups and tabs that a followed page opened. It uses window.opener, or the browser target's opener id for a noopener tab. The binding goes on that page only, with exact per-page viewport geometry and the same origin, mask and guard contract. A tab the person opens by hand is still refused.
• JS dialogs. Step.dialog = {response, message} is set on the step whose action raises the dialog. The browser backend answers it only when the message matches. Any other message is dismissed (never accepted) and the step halts. Screen-driven backends refuse a dialog step, because there a dialog is just pixels.
• Native windows overlapping on one desktop are reached with an explicit, checkable focus step (a click or hotkey with a postcondition). This needs no window enumeration.
• Rejected: surface lifetimes (long-lived/transient types, automatic return to the opener), OS window enumeration and set-diffing in the recorder, and screenshotting every page on every step. The reasons are in the doc.

Safety invariants

• Every existing gate (identity ladder, effect gates, halt-not-guess, remote identity/lease checks) acts on Replayer.backend. The active surface is routed through it, and backend-bound caches are reset on each switch.
• Any page a step did not declare halts the run before the next step. So does a declared page that does not open, opens from the wrong opener, or opens on the wrong origin, URL, title or opener relation. Halt messages never quote a URL or title.
• A switch identity check fails closed, and VLM rescue cannot override it. Recorded on the new surface, it covers the page origin, the backend kind, and screen text.
• A healthy replay still makes zero model calls.

Governed, qualified, and durable runs (new)

A multi-surface run can reach VERIFIED under the same rules as a single-surface run. No gate is weakened.

• The surface map is in workflow.json. That file is sealed in the bundle content digest, which the governed authorization already binds, so there is no second binding that could drift.
• In a multi-surface run, each StepResult records the surface it ran on. The outcome classifier returns COMPLETED_UNVERIFIED unless every result matches its step's declared surface. A single-surface report never carries the field.
• Effects stay keyed by step. VERIFIED still requires independent_readback, meaning a non-UI read of the system of record. A persisted re-read or a screen read never reaches it, on any surface.
• Qualification adds QualificationProject.surface_environments (one EnvironmentBoundary per non-primary surface).
  • A case must cover every surface, with the right kind.
  • At each switch, the runtime observes the target surface and records report.observed_surface_environments. The input guard moves to the active backend.
  • A mismatch halts at the switch. Case evidence requires every used surface's recomputed observation.
• Durable checkpoints work, and resume re-binds surfaces by identity (above).
• Still refused, with a reason: resume of a multi-surface program (graph) run, a linear checkpoint without a surface observation, and a managed run with an unbound backend surface.

CLI backend binding (new)

replay/run take a repeatable --surface NAME=KIND[:TARGET]. A deployment config takes a surfaces: map of full backend configs.

• KIND defaults from the bundle, and TARGET fills that kind's usual field.
• Everything is built by the existing backend factory. A web surface opens as a page in the run's browser.
• These are refused: an undeclared name, the primary, a page surface, or a kind mismatch.
• An unbound backend surface is named before the run. The run then halts at its switch with a hint naming the flag. Nothing is guessed.

Interactive record --url (new)

In a browser Flow launched, popups and tabs are followed as surfaces automatically.

• The listener and binding are on the browser context.
• Pages are claimed through the shared Recorder.
• Acting in another page records a switch.
• This produces the same bundle shape as the scripted Recorder.
• The listener re-arms per document, because Chromium reuses a popup's initial about:blank Window and skips the init script on its first navigation.

Compatibility: SCHEMA_VERSION stays 2.

• New fields are omitted when empty, so single-surface bundles, reports, and qualification projects serialize and hash exactly as before.
• Version-scoped sealed-digest omission rules keep existing bundles byte- and digest-identical.
• An older runtime refuses the unknown switch_surface action, so it fails closed.

Tests

• test_multi_surface_e2e.py: popup + tab, recorded, compiled, and replayed with model_calls == 0, plus refusals.
• test_multi_surface_backends.py: web then MockWaa Windows; wrong, missing, or mismatched backends halt at the switch.
• test_multi_surface_contract.py: IR contract, plus the remaining resume and managed-dispatch refusals.
• test_multi_surface_identity.py (new): durable resume across surfaces. It pauses at "Check the record" on a tab, the checkpoint holds the name and observation, the resume needs approval, re-binds by identity, and makes 0 model calls. Also: an ambiguous re-bind halts, a page-routing mismatch halts, and router-level routing works.
• test_multi_surface_selfclose.py (new): a self-closing popup followed by an explicit switch back, in the scripted and interactive recorders. It replays with 0 model calls, and without the switch back the bundle is invalid.
• test_multi_surface_attach.py (new): CDP attach with a popup and a noopener tab replays in the same browser with 0 model calls. A hand-opened tab is refused.
• test_multi_surface_dialog.py (new): a confirm() answer replays. An identity mismatch is dismissed and halts. A screen-driven backend refuses the step.
• test_multi_surface_governed.py (new):
  • a governed, durable, Standard run (web → Windows) reaches VERIFIED;
  • a tampered surface record, or a persisted-tier verifier, does not;
  • per-surface qualification boundaries and the guard;
  • unchanged single-surface bytes.
• test_surface_bindings.py (new): flag and config resolution, plus a CLI replay that binds a backend surface (0 model calls) or halts with a hint.
• test_multi_surface_interactive.py (new): an owned-browser interactive recording through a popup and a tab gives the scripted bundle shape and replays with 0 model calls.
• Locally, 1181 passed and 0 failed across 49 related test files, with the durable authority DB in a fresh, resolved temp path. The multi-surface suites alone are 56 tests. The files cover:
  • multi-surface, durable, qualification, replayer, execution profiles, and bundle schema (single-surface bundles still hash-identical);
  • recorder, interactive recorder, browser attach, attached run, and submit binding;
  • CLI deployment, surface bindings, and action evidence.
• test_qualification_gate_campaign.py passes alone (38/38). In the combined run, its 470 s campaign fixture went over the 600 s pytest timeout (8 errors) under load (scrypt-bound checkpoint reads), so those 8 are counted from that separate run.
• ruff is clean on the package and the changed tests. mypy is clean on the whole package and on the CI strict safety set.
• The earlier 8 test_qualification_gate_campaign.py errors were environmental. A macOS mktemp path goes through the /var → /private/var symlink, and the durable authority path refuses links. With a resolved path, all 38 pass.

Limits

• Dialogs are recorded only by the scripted recorder (expect_dialog). The interactive recorder does not capture them yet. Qualification does not treat a dialog answer as separate evidence, and prompt() text answers are not supported.
• The recorder does not insert the focus step for a hidden native window. The author or agent records it as an ordinary click or hotkey.
• Interactive recording refuses a popup whose first action closes it before any frame was captured. Act in it once first, or use the scripted recorder.
• Attach mode follows only same-origin pages opened by a followed page. A cross-origin OAuth popup is refused by attach mode's origin contract.
• Two pages that look identical and share every structured attribute make a re-bind ambiguous and halt. This is by design.
• Program (graph) resume of a multi-surface run is refused.
• Qualification pin/propose does not yet propose surface_environments; the operator writes them.
• A popup that opens and closes before the step's check halts. A click in a new page within the first poll tick may not be recorded; replay then halts rather than skipping it.

[abrichr]

Superseded: this work landed in the OpenAdapt monorepo as OpenAdaptAI/OpenAdapt#1197. openadapt-flow is now developed in packages/flow there.