Skip to content

flow: the compatibility matrix, checked against the runtime in CI - #1364

Merged
abrichr merged 3 commits into
mainfrom
flow/compatibility-matrix
Oct 20, 2026
Merged

abrichr merged 3 commits into
mainfrom
flow/compatibility-matrix

Conversation

@abrichr

@abrichr abrichr commented Oct 19, 2026

Copy link
Copy Markdown
Member

Stacked on #1362 (then #1361, #1358, #1355). Also carries the durable-authority fix from #1363, which is against main and can merge first.

Fifth of the small PRs that put the API step, pipelined read-back and self-repair in one production profile. This one adds the matrix.

What it is

  • openadapt_flow/compatibility.py holds every feature (API step, pipelined read-back, self-repair, standard changes, parallel workers, multi-surface) against every run profile (default gated, durable, governed, pinned, qualified, managed), plus the API step combined with each other feature. Each cell either works or is refused with its reason verbatim, and names the tests behind it. I split the brief's "admitted" profile into pinned (--pin-digest) and qualified (signed production admission) because they behave differently.
  • packages/flow/docs/COMPATIBILITY.md is generated from that module by python scripts/build_compatibility_doc.py and linked from CAPABILITIES.md. It has an at-a-glance grid, per-feature tables with the reasons, the API-step combinations, and an evidence list.

How CI checks it

tests/test_compatibility_matrix.py:

  • Completeness. Every feature has exactly one cell per profile. A refused cell names a reason. Every evidence test exists (parsed with ast).
  • API step. The probes run the real replayer under default, durable, governed and pinned, and require api_folded/api, one request, an api_request approval and no api_fallback. Under qualified and managed, the probes require exactly the matrix's reason. Inside a multi-surface run, the API step must give its multi-surface reason.
  • Pipelined read-back. Under each profile the probe requires either a real hand-off that settles the saved report, or inline_reason equal to the matrix's reason. The test double is a RestRecordVerifier subclass, not a monkeypatch.
  • Workers and self-repair. These cells call the CLI's own refusals. The pool's refusal is now cli_worklist_pool.pool_profile_refusal, used by run; self-repair's is cli_self_repair._refusal.
  • Workers with the API step. Two workers each run a Regulated, durable, gated API-step item, and every item reads VERIFIED.
  • Freshness. The published page must equal the generated one.

What the matrix says today

The widest combination is Regulated, durable, governed, gated per item, on parallel workers, with the API step. Pipelined read-back runs inline there (durable), and self-repair runs through the Python API. Under a signed qualification admission, the API step and self-repair are still refused, each with its reason. Qualifying an API step's path is the next PR.

Found on the way

The first run of the workers probe failed on a real race in the durable authority database's first creation. That is fixed separately in #1363 and carried here so the probe passes.

Locally: 87 passed (matrix, capabilities map, CLI worklist pool) and 148 passed with the durable authority suites. ruff passes.

🤖 Generated with Claude Code

abrichr and others added 3 commits October 19, 2026 13:08
openadapt_flow/compatibility.py holds every feature (API step, pipelined
read-back, self-repair, standard changes, parallel workers, multi-surface)
against every run profile (default gated, durable, governed, pinned,
qualified, managed), plus the API step with each other feature. Each cell
works or is refused with its reason verbatim, and names the tests behind
it. docs/COMPATIBILITY.md is generated from it
(scripts/build_compatibility_doc.py) and linked from CAPABILITIES.md.

tests/test_compatibility_matrix.py runs the API-step and read-back cells
through the real replayer under each profile and fails when a cell that
should work falls back, or a refusal gives another reason. The pool and
self-repair cells call the refusals the CLI uses (the pool's is now
cli_worklist_pool.pool_profile_refusal). Two workers each running a
Regulated, durable, gated API-step item read VERIFIED.

Building that probe surfaced a race in the durable authority database's
first creation (fixed separately in #1363 and carried here).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr

abrichr commented Oct 19, 2026

Copy link
Copy Markdown
Member Author

Included in batch #1383, which runs CI once for several reviewed PRs. This PR is marked merged when #1383 merges. Please don't push here unless asked; a new commit leaves it out of the batch.

@abrichr
abrichr merged commit 13ac7dd into main Oct 20, 2026
38 of 70 checks passed
abrichr added a commit that referenced this pull request Oct 20, 2026
batch: API-step fixes, compatibility matrix and local-edit classification (#1370 #1371 #1372 #1364 #1374 #1366)
@abrichr
abrichr deleted the flow/compatibility-matrix branch October 20, 2026 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant