Repository navigation
feat(authoring): let an agent drive recording on native Windows, RDP, and Citrix - #527
Merged
Merged
Conversation
…trix AuthoringSession refused windows, rdp, and citrix with COACH_ONLY, so an agent could only drive the recording in a browser or a macOS or Linux window. It now accepts every substrate Flow has a backend for. On native Windows the session drives WindowsBackend, the same synthetic-input path replay uses; RDP and Citrix are screen-only and drive the local client window from pixels and keys. The Recorder that sends the input writes the demonstration, as with record_desktop_demo; Capture's observers drop injected input on purpose. Each event now carries actor (agent or person), and meta.json carries demonstrated_by, surface, and how the input reached the app. rdp/citrix sessions stamp window_pixels and optional backend_hints for replay. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
… session run_qualification takes agent_authored, which records the canvas stand-in through AuthoringSession(backend_kind="citrix"), the path an agent drives, instead of a bare Recorder. The opt-in stand-in e2e gains a test that runs it and checks the recording says the agent did it. Also types the screen- only observe branch for mypy. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AuthoringSession refused windows and rdp/citrix with CoachOnlyError (authoring.py:65-66, :428-430 on main), even though record_desktop_demo already records over WindowsBackend.
Changes:
screenshot()gives the frame.citrix. Backend hints are limited to rdp_window / rdp_window_title / rdp_readiness_text.actor(agent/person). meta.json getsdemonstrated_by,surface,authoring.input, andcoordinate_space: window_pixelsfor remote displays.agent_authored.OAFLOW_CITRIX_STANDIN_E2E=1, not run here because Docker isn't available).Related: OpenAdaptAI/openadapt-types#47. None of this has run against real Windows, RDP, or Citrix yet; only the mock tests ran.