Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
4fdb7da
feat(ir): declare surfaces, name a step's surface, and type the surfa…
abrichr Oct 17, 2026
5ce5b84
feat(runtime): route steps to their surface, claim declared pages, ha…
abrichr Oct 17, 2026
771c0c4
test: multi-surface e2e (popup + tab), web-to-Windows backend switch,…
abrichr Oct 17, 2026
7d198cb
docs: point the one-surface boundary at MULTI_SURFACE.md; tighten typ…
abrichr Oct 17, 2026
6246e25
feat(runtime): admit governed, durable, and qualified runs of multi-s…
abrichr Oct 17, 2026
cc2227f
feat(cli): bind backends to backend surfaces with --surface NAME=KIND…
abrichr Oct 17, 2026
4068e78
Interactive record --url follows popups and tabs as surfaces
abrichr Oct 17, 2026
4af1b90
docs: multi-surface governance, CLI binding, and interactive recording
abrichr Oct 17, 2026
a96df37
fix(types): satisfy mypy for surface boundaries and binding builder
abrichr Oct 17, 2026
3e26292
feat(runtime): bind surfaces by identity; resume re-binds them (WIP)
abrichr Oct 17, 2026
db766e2
feat(durable): resume re-binds the checkpoint's surface by identity; …
abrichr Oct 17, 2026
b85d98d
feat(surfaces): a self-closing popup is followed by an explicit switc…
abrichr Oct 17, 2026
1f623d7
feat(surfaces): attach mode follows popups and tabs; browser dialogs …
abrichr Oct 17, 2026
4caad5e
fix(ir): one result serializer drops both empty surface and field checks
abrichr Oct 17, 2026
b3d5644
fix(surfaces): wait on the browser's page events, and reset the attac…
abrichr Oct 17, 2026
1fc0648
fix(interop): keep SWITCH_SURFACE out of the shared ActionType map; t…
abrichr Oct 17, 2026
4823550
fix(attach): bring the captured tab to front so a popup cannot stall …
abrichr Oct 18, 2026
b501cb1
Merge origin/main into carry/flow-533
abrichr Oct 18, 2026
8ed5a33
Merge origin/main into carry/flow-533
abrichr Oct 18, 2026
ac603e4
fix(surfaces): close the multi-surface fail-opens and keep PHI out of…
abrichr Oct 18, 2026
f64f8ee
test(cli): fake an old agent under the openadapt_flow.agent alias too
abrichr Oct 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions packages/flow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,9 +289,12 @@ Install the extras for the surface that records and replays the workflow:
each surface, and [docs/PRODUCT_STATUS.md](docs/PRODUCT_STATUS.md) lists the
evidence for each one. A bundle is bound to the surface it was recorded on.

One bundle drives one surface: one browser tab, one desktop app, or one
remote desktop session. For a task that crosses surfaces, record one bundle
per surface and chain them with `openadapt-flow compose`. Each child starts
A bundle drives one surface by default: one browser tab, one desktop app, or
one remote desktop session. A bundle can also declare more than one surface:
popups and new tabs it opens, or another backend it switches to with an
explicit, identity-checked step. See [docs/MULTI_SURFACE.md](docs/MULTI_SURFACE.md)
for this and its limits. Otherwise, record one bundle per surface and chain
them with `openadapt-flow compose`. Each child starts
only after the one before it ends with a checked save, unless you allow a
specific stop with `--allow-halt`.

Expand Down
21 changes: 16 additions & 5 deletions packages/flow/docs/BROWSER_RECORDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,12 @@ Flow selects the sole open HTTP or HTTPS tab on the `--url` origin. It does not
navigate the tab. It also does not close the tab or browser when recording
finishes.

Attach mode follows a popup or tab that the selected tab (or a page it already
follows) opened, on the declared origin. These pages are recorded as surfaces,
as they are in a browser Flow launched (`record --url`); see
[MULTI_SURFACE.md](MULTI_SURFACE.md). A tab the person opens by hand is
refused.

If two or more open tabs have that origin, Flow refuses to guess. Supply the
exact current URL:

Expand Down Expand Up @@ -133,8 +139,10 @@ silently acting in a new, signed-out one.
- The selected tab must stay on that origin for the full recording. A
cross-origin navigation stops the recording and does not produce complete
metadata.
- Do not open a popup or a new tab in the selected tab's browser context while
recording. Flow currently binds one page. It installs a new-page latch on
- Do not open a new tab by hand in the selected tab's browser context while
recording. A popup or tab that a followed page opens is followed, judged by
`window.opener` or the browser target's opener id. Any other new page is
refused. Flow installs a new-page latch on
every candidate context before it reads the accepted page baseline or selects
the recording tab. Listener registration does not replay pre-existing tabs,
so those tabs remain allowed. The selected-context latch records every later
Expand Down Expand Up @@ -371,7 +379,7 @@ Attach mode supports local Chromium-family browsers that expose a CDP
endpoint. It requires a browser process started with remote debugging and a
separate user-data directory. It does not claim Firefox, WebKit, arbitrary
Chrome extensions, an ordinary browser process that was not started for local
debugging, cross-origin tab selection, multi-page/popup recording, or direct
debugging, cross-origin tab selection, cross-origin popups, or direct
extension replay.

Same-origin iframes are recorded: every emitted coordinate is composed into
Expand All @@ -395,8 +403,11 @@ Refused loudly, never recorded silently wrong:
dropdown UI that produces no recordable action events, so the choice would
be silently absent from the compiled workflow. The recording stops with an
explicit error at the moment of selection.
- A new tab or popup. The recording is bound to one tab and stops when
another page appears.
- In attach mode, a new tab that no followed page opened, or a popup on
another origin. Popups and tabs that a followed page opens are recorded as
surfaces ([MULTI_SURFACE.md](MULTI_SURFACE.md)). A popup that closes itself
is followed by an explicit switch back. A popup whose first action closes
it, before any frame of it was captured, stops the recording.
- An action in a frame whose page-space position cannot be proven (see
above).

Expand Down
2 changes: 1 addition & 1 deletion packages/flow/docs/LIMITS.md
Original file line number Diff line number Diff line change
Expand Up @@ -463,7 +463,7 @@ and exposes independently verifiable outcomes.
| --- | --- |
| Zoom, DPI, font, layout, viewport, or monitor changes | Browser attach recording accepts a stable viewport or device-scale change between actions, starts a new exact geometry epoch, and refuses an action that overlaps the transition. Native offline conversion can normalize a stable source-window move, resize, monitor change, or scale change into one fixed output viewport when every timeline row carries valid mapping metadata. It refuses a malformed timeline or a changed output viewport. RDP and Citrix live backends can rebaseline geometry between actions and refuse a change during the exact-frame lease. Other rescale or reflow behavior remains workflow-specific. |
| Native select menus, file choosers, permission prompts, and secure desktops | OS or browser chrome may not appear in the captured surface and may be unrecordable or undrivable. Prefer an application-level or keyboard/API path. |
| New tabs and windows | Opening a tab can be observed structurally where the backend supports it; interaction inside additional windows and multi-window coordination are not a general supported path. |
| New tabs and windows | A bundle recorded with `Recorder` can declare the popups and tabs a step opens, act in them after an explicit surface switch, and halt on any page it did not declare ([MULTI_SURFACE.md](MULTI_SURFACE.md)). Such bundles run governed, qualified, and durable. Resume re-binds surfaces by identity, and managed Cloud dispatch needs only the `surfaces:` map. A program (graph) resume is refused. `record --url` and attach mode (`--cdp`, only pages a followed page opened) capture popups and tabs, and `--surface NAME=KIND[:TARGET]` binds backend surfaces at replay. Native windows and JavaScript dialogs are screen states, not surfaces. A dialog is answered by the step that raises it (browser backend, scripted recorder), and a hidden native window is reached by an explicit focus step. |
| Drag and drop or gesture-heavy controls | Not a general supported primitive. Use a structured/API alternative or validate a purpose-built workflow. |
| Dates, locale-sensitive fields, and auto-formatting | The same keystrokes can produce different values across platforms or locales. Require an explicit read-back or system effect and test the deployment locale. |
| Slow or asynchronous applications | A state that outlasts configured waits can halt. Increasing a timeout does not establish correctness. |
Expand Down
Loading
Loading