Repository navigation
feat: multi-tab, popup, and backend-switching surfaces (carry-over of openadapt-flow#533) - #1197
Merged
Merged
Conversation
…ce switch (WIP) Adds Workflow.surfaces, Step.surface/opens/surface_origin, ActionKind.SWITCH_SURFACE, the surface contract check, reviewed v2 sealed-digest omissions, and the design doc docs/MULTI_SURFACE.md. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…lt on undeclared ones (WIP) SurfaceRouter, Replayer surface_backends, SWITCH_SURFACE identity check, recorder page/switch capture, compiler surfaces. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
… and contract refusals Also detect popup vs tab from window.opener rather than Playwright's opener(). Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…es for mypy Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…urface bundles (WIP) Each step result records the surface it ran on; the outcome classifier requires that record to match the sealed contract. A qualification case observes every surface it switches to against that surface's boundary (QualificationProject.surface_environments) and moves the input-edge guard with it. Resume and managed dispatch stay refused. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…[:TARGET] and a config surfaces: map (WIP) Kind defaults to the kind the bundle declares; TARGET fills that kind's one target field; a config entry carries any other field. Bindings the bundle cannot use are refused before anything runs; an unbound backend surface is named up front and halts at its switch. A web backend surface opens as a page in the run's own browser. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
Owned browser: the listener script and event binding are bound on the context, new pages are claimed by the shared Recorder when the action that opened them is written, and acting in another page records a surface switch. The listener re-arms per document (Chromium reuses a popup's initial about:blank Window). Attach mode stays one tab. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
Record the chosen design and rejected alternatives for each gap, replace the stale refusal text, and update the limits in MULTI_SURFACE, LIMITS, PRODUCTION, and BROWSER_RECORDING. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
A surface is a name plus an identity check, never a live handle. The router binds a page or backend surface to the one live target that passes the step's vision-first identity check (anchor template or recorded screen text, plus origin/URL/title/opener where exposed); zero or several halt. Checkpoints carry each surface's last-seen observation and the active surface (omitted when unset), so a linear resume re-binds by identity. Leaving a self-closed page needs an explicit switch. Managed dispatch is admitted when every backend surface is bound; program resume stays refused. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…tests (WIP) Resume revalidation binds the checkpoint's active surface to the one live target that shows its retained contract before approval is consumed. A surface switch's armed identity is its surface check, not an entity identity row, so compiled switch steps can be checkpointed. Tests: resume across surfaces behind the Check-the-record approval, an ambiguous re-bind, and page-routing mismatches. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…h back A page-surface backend shows a blank frame once its page closes, so postconditions on a closed popup fail rather than reading another window. Both recorders write the return to the opener as an explicit SWITCH_SURFACE with that page's identity check; nothing returns on its own. The interactive recorder refuses, with a clear reason, a popup whose first action closes it before any frame of it was captured. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…answered by identity; docs Attach mode (record --cdp) follows a page only when a followed page opened it (window.opener, or the browser target's opener id for a noopener tab), installs the session binding on that page alone, and rebinds the exact CSS-viewport geometry at each switch. A tab the person opens by hand, even one that closes before the next poll, is still refused. Step.dialog carries the answer to a JavaScript dialog the step's action raises. The browser backend answers it only when its message matches; a different message is dismissed, never accepted, and the step halts. A backend that cannot see browser dialogs refuses the step before any input. The field is omitted when empty, so existing sealed bundles are unchanged. docs/MULTI_SURFACE.md documents the vision-first design, resume and Cloud, routing, explicit switches back, dialogs, the native focus step, rejected alternatives, and the remaining limits.
The 3-way replay of #533 onto main put main's _omit_empty_field_checks serializer on ObservedSurfaceEnvironment, which has no effect_field_checks (mypy attr-defined). StepResult already has #533's wrap serializer and pydantic allows one, so it now drops an empty effect_field_checks as well as a None surface.
…h test to a fresh tab test_multi_surface_attach.py failed about 4 runs in 10 here: the replay's "Request Approval" click navigated the tab instead of opening the popup. The tab-reset helper kept context.pages[0], and CDP lists pages in no fixed order, so it sometimes kept the recording test's leftover popup, whose window.name is still "approval"; window.open(url, "approval") then targets that tab. The helper now opens a fresh tab, closes every other one, and checks window.name is empty. Runtime: with the sync Playwright API a time.sleep dispatches nothing, so polling context.pages or page.url between sleeps can miss a popup until the next Playwright call. The surface router now waits on the context's "page" event (and pumps through page.wait_for_timeout while matching a claimed page's URL), the new_tab_opened postcondition waits on the page event, and page_count no longer counts closed pages. 10/10 runs pass, and 10/10 with six CPU-bound processes running alongside.
…olerate stub workflows in surface binding SWITCH_SURFACE is a flow-only control step with no canonical ActionType, so step_to_action refuses it the way it refuses SELECT_OPTION. The CLI's surface-binding check reads workflow.surfaces with getattr, so tests that stub Workflow.load with a bare object keep working.
This was referenced Oct 17, 2026
…its frame Attach mode now follows a popup instead of refusing it, so the opener's after-frame is captured while the popup tab holds the foreground. Chromium draws no new frames for a background tab, and a person's browser reached over CDP lacks the launch flags Playwright uses to keep background tabs rendering, so Page.screenshot (masked, or repeated in the settle loop) waited until its 30 s timeout. That failed the live CDP attach test on Linux. Attach-mode backends now bring their page to front before each capture (a no-op for the tab already in front). Re-showing a tab makes Chromium fire a resize that leaves the viewport unchanged; a viewport event that reports the bound geometry, or comes from a page other than the active one (that page is rebound when Flow switches to it), no longer marks the coordinate space dirty. A real resize still refuses the recording. The new test reproduces the stall in a separately started Chromium: it times out without the front-of-tab step and passes with it.
Resolves the one conflict in packages/flow/openadapt_flow/runtime/replayer.py: both sides added Replayer.__init__ keyword arguments (surface_backends here, attended_draft_by and draft_key_dir on main). Both are kept.
Keeps both sides: Workflow.origin and Workflow.surfaces in ir.py, the layout-drift fields and surface_backends in Replayer.__init__, and both deployment_surface and surface_backends in the CLI wiring. The layout-drift check now follows the active surface in a multi-surface run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… evidence Routing: - A non-switch step on a closed active page is refused, whatever the surface kind; only an explicit switch moves on. - A page surface rebinds only to a page claimed on this run. A resumed run may bind a page from the starting inventory, but only before that surface is first bound again. In attach mode this keeps the person's own tabs out. - Rebinding needs the step's visual evidence on the page; structured attributes alone never rebind. A held page that fails its check while no other candidate passes now halts instead of being used. - A switch with neither recorded screen text nor an origin halts for every target kind, including page surfaces. Evidence: - Dialog halts name a code (dialog_mismatch, dialog_missing), never the live or recorded dialog text, which lands in report.json and REPORT.md. - The compiler seals a dialog as a full-match message_pattern when its message carries digits or a demonstrated parameter value, and refuses a message the privacy pass still flags. - url_pattern generalizes identifier-like path segments (any digit, or a long opaque token). - The recorder writes page URLs to meta.json without the query or fragment, and no titles. Also types the layout-drift kind for mypy. Each rule has a test in tests/test_multi_surface_fail_closed.py (or the e2e/dialog files) that fails without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
openadapt-agent 2.2.1 (on PyPI since 2026-10-18 15:58 UTC) is a shim whose openadapt_agent package is the openadapt_flow.agent module. Blocking only openadapt_agent.seal_cli no longer simulates an agent without Seal checks: after test_seal_forwards_to_agent imports the real submodule, the shared package keeps a seal_cli attribute and the old-agent tests reach the real agent CLI. Python CI (run-ci) fails on every branch since that upload. Block the submodule under both names and drop the package attribute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 30dd9b1)
Member
Author
|
Review fixes pushed (8ed5a33 merge of main, ac603e4 fixes, f64f8ee CI test fix). Merge of main. Both sides are kept in Fail-opens closed (
PHI:
Tests. Each change has a test that fails without it: CI fix (f64f8ee). This is the same change as #1259. Since openadapt-agent 2.2.1 reached PyPI, 🤖 Generated with Claude Code |
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carry-over of OpenAdaptAI/openadapt-flow#533 (
feat/multi-surface, head8529d8e7, includingea6985ce,ea2d3e12,a8972c71,8529d8e7). To merge last of the carry-overs.Replayed with
git format-patch --no-mergesover the 13 commits after flowmainandgit am -3 --directory=packages/flowonto OpenAdaptmain.Conflicts and adaptations, all checked against
git merge-fileof the same three versions:compiler/compile.py: main's default fix-up passes (Implement Instructor #532) and Explore MkDocs #533's switch-surface risk check both kept, fix-ups first.interactive_recorder.py: main'slaunch_chromium(Investigate Video-LLaVA #528) with Explore MkDocs #533's explicit context so popups share it.ir.py: the 3-way merge put main's_omit_empty_field_checksserializer (Explore Outlines #534) onObservedSurfaceEnvironment(mypy attr-defined).StepResultalready has Explore MkDocs #533's wrap serializer and pydantic allows one, so that one now drops an emptyeffect_field_checkstoo (4caad5e5).Flake fix (
b3d5644e):test_multi_surface_attach.pyfailed about 4 runs in 10 locally. The replay's "Request Approval" click navigated the tab instead of opening a popup. The tab-reset helper keptcontext.pages[0], and CDP lists pages in no fixed order, so it sometimes kept the recording test's leftover popup, whosewindow.namewas stillapproval. The helper now opens a fresh tab and closes the rest. In the runtime, the surface router and thenew_tab_openedpostcondition now wait on the context'spageevent instead oftime.sleeppolling (a sleep dispatches no Playwright events).page_countskips closed pages. Result: 10/10 runs pass, and 10/10 with six CPU-bound processes running alongside.test_qualification_gate_campaign.py(about 470 s of module setup against the 600 s pytest-timeout): the monorepo Packages workflow leaves it out of the unit shards, just as Flow's PR gate did, so it can't hit the timeout there. Flow used to run it on schedule and dispatch; the monorepo has no such job yet (follow-up).Local: ruff, ruff format and mypy are clean on
packages/flow, and the multi-surface, risk and attached-browser tests pass (217 passed).