Repository navigation
flow: check a click target's hover response, and guard pixel clicks from check to click - #1334
Merged
Merged
Conversation
On a pixel surface, an OCR identity read that abstains only because the record ID has a look-alike glyph (1/l, 0/O) now takes extra looks before the run stops: two fresh frames, the 2x re-read, the known-ID list, and, where configured, a select-and-copy of the identifier's text box. Only the list or a copy equal to a run parameter can settle the record; any look that names another record is final. Each look is recorded on the step's identity check. The copy look is input, so only the step-entry check takes it; the actuation check reuses it only on matching pixels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a pixel surface, an icon-only click has no text that names its control. The recorder can now rest the pointer on each click target and keep that frame; the compiler stores what changed near the target (the control's highlight or tooltip) as hover evidence for clicks with no other identity evidence. At replay, on the pixel rungs, the runtime rests the pointer on the resolved target and compares. A different response stops the run before the click; a missing or uncomparable one is recorded and changes nothing. Hover works on the browser, remote-display and FreeRDP backends. Agent-driven RDP and Citrix recordings capture it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An absent Step.hover is omitted from the content digest, like other additive v2 step fields; a recorded hover response is sealed content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The copy drag now runs to the edges of the identifier's text box; it stopped inside the last glyph and copied one character short. - A split identifier (103 045 against 103045) is matched as one string, against the read and the known-ID list. - The check just before the click reuses the step-entry verdict when the identifier shows the same pixels, instead of re-reading the selected text. - A run-parameter mismatch that is only a look-alike misread (C0X1234 read as COX1234) gets the same looks. The mismatch stands unless the list or an exact copy of the run value settles the record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # packages/flow/openadapt_flow/runtime/replayer.py
Consequential identity-gated clicks were already bound to the exact checked frame. Every other pixel click now compares the target, identity and hover regions on a fresh frame taken right before the click with the frame the step was checked on (the hover frame after a hover look). On a change it does not click: it re-resolves the target and re-runs the identity check on the fresh frame with no input, and clicks only when the next frame matches. Two re-observations that still see a change, or a target or identity that cannot be confirmed again, refuse the click. Each change is recorded on the step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr
changed the title
- identity_looks_dense: pass a typed click point and ActionKind (mypy). - LIMITS.md: name identity_looks.py by its package path, which the docs consistency check resolves. - The look-alike twin test asserts the stop; it accepts a read that is already final, as Linux OCR builds read it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- replayer: assert the anchor before the identity check so mypy can narrow it when a hover check stands in for the OCR check. - LIMITS.md: name hover_evidence.py by its package path. - test_state_dependency: the never-settling fixture now animates a strip clear of the click target. An animation over the target itself is a change between check and click, which the new guard refuses; the test is about the readiness default, not about clicking under a moving bar. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1333 (its commits show here until it merges). This PR's own change is the last commit.
What
On pixel surfaces, an icon-only click carries no text that names its control, so the right-record check had nothing to compare. Most apps answer a pointer resting on a control (highlight, tooltip, status text). Now:
Recorder(capture_hover=True)rests the pointer on each click target, waits 0.8 s, and keeps the frame (frames/{i}_hover.png). Agent-driven RDP and Citrix authoring sessions turn it on.Step.hover(HoverEvidence: the box relative to the click, and its pixels undertemplates/hover/, sealed with the other crops).hover, the runtime rests the pointer on the resolved target, waits the same time, and compares (runtime/hover_evidence.py). The verdict is bound into the step's identity check (mode="hover").Fail-closed rules
Step.hoveris a reviewed v2 omission; a recorded response is sealed content.Check-to-click guard (pixel clicks)
A pixel step is checked on one frame and clicked a moment later (the gap the "visual confused deputy" work calls TOCTOU). Consequential identity-gated clicks were already bound to the exact checked frame by hash. Every other pixel click (click, double-click, right-click) now gets
Replayer._pixel_recheck:safety_halt);StepResult.check_to_click, omitted when empty). Changes under 4 pixels or 24 levels per channel (encoder noise) don't count. Structured targets keep their element tokens and aren't rechecked.Tests (
tests/test_check_to_click.py): a dialog opening between check and click aborts with nothing clicked; a moved target is re-observed and clicked where it is now; a screen that keeps changing is refused; a clock elsewhere and encoder noise don't stop the click; the hover frame is the baseline after a hover look; structured targets aren't rechecked.Benchmark
benchmark/hover_identity/run_hover_sweep.py: the appointment kiosk with Save as a toolbar icon, plus an identical-lookingClear formicon that differs only by tooltip; the kiosk draws Tk-style hover feedback (kiosk_render.py, off by default so the display-scale sweep is unchanged). Then the two icons swap places. On that kiosk the look-alike icon was clicked in 2 of 3 runs before and 0 of 3 after; numbers and docs in #1340.Tests
tests/test_hover_evidence.py(19): response measurement, compare verdicts, look-alike icon with other tooltip stops, rasterizer change with same text verifies, no hover at the pre-click check, structural rung takes no hover, backend without hover keeps old behaviour, compiler keeps the response, recorder saves the hover frame only when asked.No new commands.
🤖 Generated with Claude Code