[abrichr]

Stacked on #1504 (base: its branch). Merge #1504 first; GitHub then retargets this PR to main.

Before

After #1504, a step that picks a row before a write halts with identity_uncounted on any surface that can't count look-alike rows. That covered native Windows, macOS and Linux apps, plus pixel identities with no OCR band. Those workflows stopped instead of failing open, but they couldn't run.

After

Each of those surfaces now counts the rows that read the same as the target's row. The rules from #1504 then apply: one matching row runs; several halt unless the recorded record id picks one; a row list that can't be fully enumerated halts.

Surface	Rows counted	Record id	Incomplete (halts)
Windows (UIA agent, /uia/identity-candidates-at)	DataItem, ListItem, TreeItem and TableRow siblings, read at the same child index as /uia/text-at-point	AutomationId, unless it only encodes the row's position (ListViewItem-0)	Grid pattern RowCount above the exposed rows; a scrollable list with the ItemContainer pattern
macOS (AX)	rows of AXTable / AXOutline; items of AXList	AXIdentifier, AXURL	more AXRows than exposed rows; a row whose cells aren't built
Linux (AT-SPI)	sibling list items, table rows, tree items, read exactly as structured_text reads them	AT-SPI accessible id	Table interface n_rows above the exposed rows
Pixel only, no OCR band	the places on the frame where the identifier crop verifies (vision.match.template_peaks, then the ladder's pixel verifier on each place)	none	16 or more candidate places

Pixel screens that have an OCR band already used #1504's OCR census; that's unchanged.

The tree walk is one function, identity_census.tree_row_census, used by macOS and Linux. The Windows agent is copied into the guest without this package, so it carries a mirror of the same rule.

Tests

• tests/test_native_identity_census.py: the generic census (two and three identical rows, distinct ids picking the row, a unique row, controls outside rows, position-only ids, a short or virtualized table, an unbuilt row, AXList items). It also covers the Linux client over fake AT-SPI nodes, the Windows agent over fake UIA controls (including the served operation), the macOS backend's point-to-pixel conversion, and the replayer on a native backend: identical rows halt, the recorded id moves the click, a unique row runs.
• tests/test_native_census_real_apps.py (opt-in): a WPF ListBox read through real UI Automation, and a Swift NSTableView read through real AX. The new CI job flow-native-census runs both. The Windows leg fails if any WPF test is skipped. The macOS leg skips when the hosted runner gives no Accessibility access. On my Mac, AX returns application elements in place of windows in this session, so those tests skipped here too.
• tests/test_identity_evidence_capability.py: a crop-only identity now runs when the crop verifies in one place and halts when it verifies in two.
• Locally (2 workers): ruff, mypy (including strict), and 98 targeted test files, 3,291 passed.

Not covered

• Linux: GTK 3 list boxes (one child per row) and tree views (cells without a row) give no row identity under the current structured reading, so identity there rests on OCR. No real-app Linux test yet.
• macOS real-app: the test only proves something on a runner with Accessibility access. If the CI leg skips, the macOS census is covered by fake-tree tests only.

🤖 Generated with Claude Code

[abrichr]

Coordinator review (AI-assisted). Approved at a08534d.

• Earlier fail-open findings, all fixed:
  • The pixel census counts anything that isn't a clear mismatch as a look-alike; an uncountable place halts.
  • A list with no declared row count reads incomplete unless it definitely doesn't scroll.
  • Windows provider errors read incomplete.
• positional_id: the token rule drops 0 of 24,000 real ids across 12 families (UUIDs, ULIDs, encounter and visit ids, MRNs) and still drops positional ones (row-3, r3c2, item_12). Parity holds in the browser and the Windows agent.
• Write verbs: "Sign out patient", "Discharge patient" and similar are consequential; session sign-out stays non-consequential.
• Real-app CI: the Windows WPF and macOS AX legs ran with zero skips and passed, and the job fails on any skip.

Land together with #1504 (#1518 contains it), after #1467.

[abrichr]

Real-app check, run by the production-lifecycle session: at ee707b8, the OpenEMR representative admission case is VERIFIED (exact_record read-back, 0 model calls, 105 s engine). It was tested at d21779a68 (measure/production-lifecycle-final 8bfd7f6 = main 78ff5ad + #1516 + harness, merged with ee707b8) on a fresh stack. step_016 is pixel verified, then the census gives not_a_row; every other step through the save (step_040) is structured verified. Only the representative case ran, not the 5 fault cases.

[abrichr]

AI review (coordinator)

Scope: the commits after the #1557 merge at head ee707b80d (33bc0ac, 152d69a, ee707b8). I've turned auto-merge off because of finding 1, which lets the census click the wrong row.

1. Blocker: an OCR misread makes two identical rows count as different (152d69a)

The new tie-break treats any difference in the squashed text as distinguishable (other_text != target_text). It doesn't check OCR confidence or how close the texts are. Two renders of the same label often read slightly differently.

To test this, I rendered a patient list with two "Smith, John" rows that have different DOBs (Arial 13 px). I ran the real Replayer._pixel_census with real RapidOCR and pixel_verify_enabled=True. I counted a case only when the target's own pixel tier returned verified, so the census actually runs. A sample of results with a tight identifier crop (the size of a web field rect or an operator-marked box):

live frame	target crop OCR	other "Smith, John" crop OCR	census
JPEG q75 (lossy remote display), no offset	Smith, John	Smith, 10(0.71) lohn(0.87)	unique
zebra rows, 1 px offset	Smith, John	Smith, S(0.77) John	unique
no JPEG, 1 px vertical offset	Smith, John 19(0.65)	Smith, John	unique

Overall, 18 of 34 tight-crop cases returned unique while two identical rows were on screen. The ambiguous halt then never fires, and the resolver can click either Smith before the save. Padded crops (1 to 4 px) were not affected in this grid. Every misread had a fragment below 0.9 confidence, and every pair of reads had a similarity ratio of 0.82 or more.

Fix. A place counts as distinguishable only when all of these hold:

• every OCR line in both crops has confidence of at least 0.8;
• neither text contains the other;
• difflib ratio < 0.6.

Sibling tab labels measure 0.5 or less ("Contact"/"Stats" 0.5, "Contact"/"Who" 0.2).

2. Bug: crop_is_featureless returns 1.0 when one shifted window is flat (ee707b8)

shifted_score returns 1.0 when denom == 0. That includes the case where only one of the two shifted windows is flat. So a crop whose content sits entirely in one edge quarter counts as "featureless". Examples: a short left-aligned label in a wide cell, such as "Li, Al" in a 200 px name cell, or "17" in a 120 px cell. I measured both as featureless=True.

The pixel census then returns cannot_count:

• for a row or a target with no role, the run halts (identity_uncounted). This is a false halt.
• for a teach-time control, the target verifies with no count at all.

Fix. Return 1.0 only when both windows are flat, and 0.0 when exactly one is.

3. Featureless crop plus control role (question 2): acceptable, not a regression

A featureless crop returns cannot_count, and a control then gets not_a_row. That is exactly main's behaviour for every control (main has no pixel census), so this change doesn't widen anything compared with main. It narrows only the stricter pre-ee707 state of this PR, and it's consistent with #1557's rule. The remaining exposure is a per-row button whose crop really is self-similar, such as a row of identical icons spaced at a quarter of the width (measured 1.0).

Common per-row controls aren't flagged:

control	measured
kebab menu	0.79
ellipsis	0.79
empty checkbox	0.43
radio button	0.05
empty input	0.82
button outline	0.77
text crops	at most 0.5

Those crops still get counted and halt when they're ambiguous. The 0.9 threshold leaves margin above real crops. The flat-window case in finding 2 is the only false positive I found.

Exact fix (verified locally)

diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py
index 48bc241f7..6bdad2cd9 100644
--- a/packages/flow/openadapt_flow/runtime/replayer.py
+++ b/packages/flow/openadapt_flow/runtime/replayer.py
@@ -736,6 +736,17 @@ class _WebFormApprovalView:
     ref: Any
 
 
+
+# The pixel census's OCR tie-break: a place the pixels do not rule out is
+# distinguishable only when OCR reads both crops with every line at or above
+# this confidence, neither text contains the other, and the two texts are less
+# similar than this ratio. One label read two ways measured 0.82-0.95
+# ("Smith, John" / "Smith, S John" / "Smith, 10 lohn"); sibling tab labels
+# ("Contact" / "Stats") measure 0.5 or less.
+PIXEL_CENSUS_OCR_MIN_CONFIDENCE = 0.8
+PIXEL_CENSUS_OCR_MAX_SIMILARITY = 0.6
+
+
 class Replayer:
     """Replays a Workflow against a Backend using injected vision.
 
@@ -15823,10 +15834,27 @@ class Replayer:
                 lines = self.vision.ocr(png, region=None)
             except Exception:
                 return ""
+            # A low-confidence fragment ("S", "10", "lohn") is a misread, not
+            # evidence: the whole crop then counts as unreadable.
+            if any(
+                float(getattr(line, "confidence", 0.0) or 0.0)
+                < PIXEL_CENSUS_OCR_MIN_CONFIDENCE
+                for line in lines
+            ):
+                return ""
             return identity_mod.squash(
                 " ".join(str(getattr(line, "text", "") or "") for line in lines)
             )
 
+        def texts_differ(a: str, b: str) -> bool:
+            """Clearly different labels, not one label read two ways."""
+            import difflib
+
+            if a in b or b in a:
+                return False  # a dropped or clipped word, not another label
+            ratio = difflib.SequenceMatcher(None, a, b, autojunk=False).ratio()
+            return ratio < PIXEL_CENSUS_OCR_MAX_SIMILARITY
+
         target_text: Optional[str] = None
         count = 1
         for x, y in peaks:
@@ -15847,7 +15875,7 @@ class Replayer:
             if target_text is None:
                 target_text = crop_text(live)
             other_text = crop_text(crop)
-            if target_text and other_text and other_text != target_text:
+            if target_text and other_text and texts_differ(target_text, other_text):
                 continue
             count += 1
         if count > 1:
diff --git a/packages/flow/openadapt_flow/vision/match.py b/packages/flow/openadapt_flow/vision/match.py
index 2f96557a7..933d12c7d 100644
--- a/packages/flow/openadapt_flow/vision/match.py
+++ b/packages/flow/openadapt_flow/vision/match.py
@@ -333,8 +333,15 @@ def crop_is_featureless(template_png: bytes) -> bool:
     def shifted_score(a: np.ndarray, b: np.ndarray) -> float:
         a = a - a.mean()
         b = b - b.mean()
-        denom = math.sqrt(float((a * a).sum()) * float((b * b).sum()))
-        return 1.0 if denom == 0 else float((a * b).sum()) / denom
+        energy_a = float((a * a).sum())
+        energy_b = float((b * b).sum())
+        if energy_a == 0 and energy_b == 0:
+            return 1.0  # both windows flat: nothing tells places apart
+        if energy_a == 0 or energy_b == 0:
+            # Content confined to one edge quarter (a short label in a wide
+            # cell): the shifted windows are not alike.
+            return 0.0
+        return float((a * b).sum()) / math.sqrt(energy_a * energy_b)
 
     dx = max(1, width // 4)
     dy = max(1, height // 4)
diff --git a/packages/flow/tests/test_native_identity_census.py b/packages/flow/tests/test_native_identity_census.py
index e3ae7facf..8d068091c 100644
--- a/packages/flow/tests/test_native_identity_census.py
+++ b/packages/flow/tests/test_native_identity_census.py
@@ -855,7 +855,7 @@ class _TabOcr:
             _TABS + ["Contact"],
             key=lambda name: float(np.abs(crop.astype(int) - _tab(name)).mean()),
         )
-        return [types.SimpleNamespace(text=label)]
+        return [types.SimpleNamespace(text=label, confidence=1.0)]
 
 
 def _tab_census(tmp_path, labels: list[str], target: int, *, readable: bool = True):

With this diff:

• the Smith grid has 0 of 288 unique results at 13 px and 0 of 288 at 11 px. On the head commit it had 77 of 288 at 13 px.
• the motivating OpenEMR "Contact" tab is still unique with real RapidOCR on the rendered tabs.
• two "Contact" tabs still halt.
• test_native_identity_census, test_census_teach_role and test_native_census_real_apps pass (80 passed, 6 skipped). The fake _TabOcr needs confidence=1.0 because a line with no confidence now counts as unreadable.
• ruff is clean.

Please also add a regression test: two identical rows, where the other crop's OCR returns a low-confidence fragment, should be ambiguous.

Auto-merge stays off until the fix lands.

[abrichr]

Real-app check, run by the production-lifecycle session at the pushed head 68e79e7: the OpenEMR representative admission case is VERIFIED (exact_record read-back, 0 model calls, 105 s engine). It was tested at 0cedc0c7d (measure/production-lifecycle-final 07742b8 = main 78ff5ad + #1516 + harness, merged with 68e79e7) on a fresh stack. step_016 is pixel verified, then the census gives not_a_row, and all 20 census calls stayed verified. Only the representative case ran, not the 5 fault cases.