diff --git a/docs/build-plan.md b/docs/build-plan.md index 35ae69a..26fb4c1 100644 --- a/docs/build-plan.md +++ b/docs/build-plan.md @@ -55,7 +55,7 @@ Tasks: Owner: - [x] Owner: create a scratch source repo in the org for the V tests, separate from claude-review-test ([claude-review-p0-source](https://github.com/OperationCode/claude-review-p0-source)). -- [ ] Owner: Anthropic Console: a dedicated workspace with a hard monthly spend limit and a service account. Create the production federation rule as in spec "Auth and cost": pattern mode, subject prefix `repo:OperationCode@10479364/*`, scope `workspace:developer`, the review workspace only, and the placeholder SHA `0000000000000000000000000000000000000000` in `job_workflow_ref`. Read it back; if the Console rejects the placeholder, stop and resolve the order of rule ID and SHA first (spec V26 a). Its ID goes into the central code; phase 3 sets the candidate SHA. No API key. Done 2026-10-27: workspace `claude-review`, service account `claude-review`, rule R1 `claude-review-prod-r1` created; the Console accepted the placeholder SHA (V26 a partial). The read-back output is not recorded, so this item stays open; the later V26 b exchange does not replace it. +- [ ] Owner: Anthropic Console: a dedicated workspace with a hard monthly spend limit and a service account. Create the production federation rule as in spec "Auth and cost": pattern mode, subject prefix `repo:OperationCode@10479364/*`, scope `workspace:developer`, the review workspace only, and the placeholder SHA `0000000000000000000000000000000000000000` in `job_workflow_ref`. Read it back; if the Console rejects the placeholder, stop and resolve the order of rule ID and SHA first (spec V26 a). Its ID goes into the central code; phase 3 sets the candidate SHA. No API key. Done 2026-10-27: workspace `claude-review`, service account `claude-review`, rule R1 `claude-review-prod-r1` created; the Console accepted the placeholder SHA (V26 a partial). The owner read R1 back at M1 and R2 at M2 on 2026-10-27 (spec P3-6). Still open: the service account and the workspace were not read back. - [ ] Owner: the test workspace (reused for evals, spec P2-1), and a federation rule for `workflow_dispatch` runs in `OperationCode/.github` whose job uses the environment `eval`: pattern mode, the same subject prefix, exact claims `repository_owner_id`, `repository_id`, `event_name` and `environment`, scope `workspace:developer`, that workspace only (spec "Auth and cost"). Its IDs are in `evals.py`. The rule cannot name `eval.yml`: any `.github` workflow that uses `eval` can match, so the owner's approval and the workflow unit test (1c) hold it to `eval.yml`. Test it as in spec V27 before phase 2. - [ ] Owner: on `.github`, create the GitHub environment `eval`: the owner is a required reviewer for every run, and any branch may deploy to it. "Prevent self-review" is off temporarily (owner, 2026-10-26), so that the coordinator, which runs with the owner's token, can approve eval runs it started from reviewed branches; the owner turns it on again in phase 5. Admin bypass stays on. During phase 2 the coordinator's approval stands in for the owner's under the owner's delegation. Before approving a run, check the run that waits for approval: the workflow file and commit it runs, every job in it that uses `eval`, and their effective permissions (no GitHub write). Also check that the PR does not change `eval.yml`, any other workflow file, the eval code, the prompts, the fixtures or the federation setup in a harmful way, and that `test.yml` passed. The approval is the control; the residual risk is an approver who approves without reading. Exception to the no-execute rule (K-4): evals of PRs to this repo may run that PR's code, only after an owner approves the `eval` environment run. Needed before phase 2. - [x] Owner: a test workspace with a very small spend limit for V9. @@ -151,29 +151,29 @@ Exit criteria: Run the spec's workflow acceptance tests under a test org ruleset that requires the central workflow from `OperationCode/.github` at the candidate release SHA and targets only claude-review-test. Only one ruleset requires the workflow there at a time. - [x] Production IDs in `analyze.py` and the release tests in CI (the candidate S1). R1 `claude-review-prod-r1` (`fdrl_01N2ZktTiVKqUvvXwywBFfys`) and the production service account are set; `test.yml` runs `pytest tests/unit -m release`. -- [ ] V26 a (partial): the owner created R1 at the placeholder SHA and the Console accepted it (2026-10-27); R2 `claude-review-p3-r2` was created the same way, inert until the release simulation. No read-back output is recorded for R1 or R2. Remaining: read R1 and R2 back. +- [x] V26 a: the owner created R1 at the placeholder SHA and the Console accepted it (2026-10-27); R2 `claude-review-p3-r2` was created the same way. The owner read back R1 at M1 `590a387` and R2 at M2 `1cefe42` before the M3 edit (spec P3-6). The service account and the workspace were not read back. - [x] P3-2: an over-long string no longer fails a pass. Release eval run 40517202830 at S1 matched 12 of 13: one supply-chain quote went over 300 characters, which the API cannot enforce. Analyze now cuts every string over its schema `maxLength` (summary, file, quote, title, explanation) to that limit and keeps the finding (spec P3-2). Fix round 1: a cut file path ends in "/", so it names no changed file; when the file, title or quote is at its limit, the inline comment key covers every shown field, so a cut never hides a distinct explanation. Fix round 2: a redaction can make two titles equal far below the limit, so the key of every finding covers every shown field, at any length. Fix round 3: each inline comment carries the D14 key and the full key, both of the redacted shown text; one run dedupes by full key, a later run skips a D14 key already on the PR and shows the finding in full in the summary when its full key is new. Fix round 4: an earlier comment counts only when its text checks out (the D14 skip needs the inline layout, path, side and title; "see the earlier comment" needs the exact body at the same line), and an unverified high or critical finding is never cut from the summary. A candidate with this change needs its own eval. - [x] V26 b, before the acceptance tests: after the owner sets the production rule to the candidate SHA, a run at that SHA gets a token and a run at another SHA is denied. Run 40560632129 at S1 got a token (2026-10-27, 18:31Z); run 40561015054 at the probe SHA was denied, and the Console shows 6 rows `Failure` `match_claim_value_mismatch` on R1 at 18:33:49Z (owner). - [ ] V26 c, before V22 b: release steps 1 to 6 with a scratch rule R2 on the test workspace, R1 unchanged. Include the second exchange at the release SHA, and check that the workflows rule accepts a scratch ruleset pinned to a SHA that is only on a non-default branch of `.github`. S2 `9752ba7` (PR 10 into `p3-sim-base`) and M2 `1cefe42` both got a token (runs 40585724589 and 40587964524); the rule accepted SHAs that are only on non-default branches. Partial: accepted at S2 and M2 (token inferred through R2); R2 Console rows were not read, no R2 read-back output is recorded, and step 5a was not run for R2/M2. Remaining: R2 read-backs at the placeholder, S2 and M2, and a denied exchange at M2 before the R2 edit. - [x] Release eval at S1 (`3aa7ae9`): run 40528717338, 13 of 13 matched. - [x] M1 tree check: M1 `590a387` and S1 have the same tree, and the R1 exchange works at M1 (run 40585413003) after the denied run at M1 (run 40583290922). -- [ ] Release simulation (R2, S2, M2): steps 1 to 6 of "Each release", never merged to main. Partial: steps ran and S2 and M2 got tokens; no R2 read-backs and no step 5a deny for R2/M2. Remaining: those two parts. -- [ ] A fork PR from an account outside the org, including a draft that later becomes ready. The review starts with no approval click (V8 b). NOT YET RUN, pending account X. -- [ ] V8 a: a fork PR whose own workflow has a job named `oc-claude-review` that starts before the required run. NOT YET RUN, pending account X. A same-repo version by an org admin passed (PR 36, run 40572853203 started first; the box stayed blocked). +- [ ] Release simulation (R2, S2, M2): steps 1 to 6 of "Each release", never merged to main. Partial: steps ran and S2 and M2 got tokens; the R2 read-back at M2 is done. NOT RUN, stays open: the step 5a deny at M2 before the R2 edit. +- [x] A fork PR from an account outside the org, including a draft that later becomes ready. The review starts with no approval click (V8 b). Pass 2026-10-28: PR 38, run 40621436811 (spec P3-6). +- [x] V8 a: a fork PR whose own workflow has a job named `oc-claude-review`. Pass 2026-10-28 (PR 39): from a first-time fork GitHub holds the fake run (`action_required`), so it cannot start first; after approval it passed and the box followed the central run. The fake-first case stays covered by the same-repo evidence (PR 36, run 40572853203 started first; the box stayed blocked). - [x] V11: retarget from a non-default branch. The retarget starts no run and the box stays blocked until a reopen run passes (PR 29, run 40561903948); an old pass on the same head counts (PR 30). - [x] V12: repo-level Actions settings. With Actions disabled, and with a restrictive allowed-actions list, the required run still ran (runs 40562372022 and 40562688524). Settings restored. - [x] V22 a and b: pin behind main, then a pin move with PRs open. `job.workflow_sha` equaled the pin in 33 of 33 attempts. After a forward pin move the old passes do not count; after a move back they count again (spec "Pinned SHA only"; PRs 28 and 37, 21:32 to 21:40Z). -- [ ] V23: a same-path workflow file in the target repo. NOT YET RUN, pending account X. A same-repo version by an org admin passed (PR 35, run 40572849014). -- [ ] V25 (NOT YET RUN, pending account X): the pilot bypass. Maintain and Admin can merge past a failed run, Write cannot, and GitHub records each bypass. Record the role `actor_id` values, and whether a direct push to the default branch is blocked. +- [x] V23: a same-path workflow file in the target repo. Pass 2026-10-28 (PR 40): the central run failed on the bidi hard block and was not replaced; the fork run stayed `action_required` and never ran (the running case is covered by PR 35). Same-repo evidence: PR 35, run 40572849014. +- [x] V25: the pilot bypass. Pass 2026-10-28 for Maintain (PR 42) and Admin (PR 43): each merged a failed PR with the bypass. Write bypass NOT RUN (a plain merge was refused, which does not test the bypass; Write is not on the bypass list). GitHub records `bypass` with the user ID as actor (no role ID), and a direct push to main by Maintain is refused (GH013). RepositoryRole 2 is Maintain and 5 is Admin (built-in role IDs). Residual: the Write `--admin` attempt. The TR bypass list is test-only and goes back to empty after phase 3 (owner's choice pending). - [x] The rest of the acceptance list in the spec's "Testing" section that needs no account X, except the four unit-test-only cases listed in the next item (approver removed, edited override, artifact removed, gate cancel after inline comments), including the outage process (spec "Phase 3 results"). Design change (spec P3-1): the invalid federation rule case ran live with two denied exchanges (runs 40561015054 and 40583290922); the exhausted budget case is covered by V9 (phase 0) and the unit tests, with no Console limit edit (ruling Q7). -- [ ] The acceptance cases that need account X: V6 f as a Write user, and the web UI "Re-run all jobs" button on a ruleset-started run (NOT YET RUN; the override cases ran with the REST re-run). Also not run, covered by unit tests: gate cancelled after inline comments with a refused summary write, approver role removed, edited override, artifact upload removed. -- [x] Sonnet 5.5 on every pass (spec P3-4). Eval at 302882f: runs 40589761270 and 40590079545 each matched 13 of 13 with no failed pass; the Opus reference run 40528717338 also matched 13 of 13. Total tokens 0.97x and 0.98x of Opus, output tokens 0.71x and 0.76x, eval wall time 84 s and 82 s against 138 s. The switch takes effect at the next release: the production rule pins a SHA, so merging does not change production. +- [ ] Acceptance cases that needed account X: V6 f as a Write user is done (residual CONFIRMED, PR 41). The web UI "Re-run all jobs" button on a ruleset-started run passed (run 40628870689, PR 43: new attempt on the same run ID, live state re-read, summary comment updated; merge box not observed because the PR was merged); the override cases ran with the REST re-run. Also not run, covered by unit tests: gate cancelled after inline comments with a refused summary write, approver role removed, edited override, artifact upload removed. +- [x] Sonnet 5.5 on every pass (spec P3-4). Release evals 13 of 13 at M3 (run 40592861233); live run 40593729924 passed with `claude-sonnet-5-5` (spec P3-6). Eval at 302882f: runs 40589761270 and 40590079545 each matched 13 of 13 with no failed pass; the Opus reference run 40528717338 also matched 13 of 13. Total tokens 0.97x and 0.98x of Opus, output tokens 0.71x and 0.76x, eval wall time 84 s and 82 s against 138 s. The switch takes effect at the next release: the production rule pins a SHA, so merging does not change production. Owner: - [x] Owner: set `job_workflow_ref` of the production federation rule to the candidate release SHA. Done 2026-10-27 (S1, then M1 `590a387`; the run at M1 got a token; no read-back recorded). The candidate's code must name that rule ID (spec "Auth and cost", "Each release"). - [ ] Owner: an account outside the org opens the fork PRs. -- [ ] Owner: test users with Write, Maintain and Admin on claude-review-test for V25. +- [x] Owner: test users with Write, Maintain and Admin on claude-review-test for V25. Done with account X `irvingpop2` (2026-10-28). Remove its access and empty the TR bypass list after phase 3 (owner's choice pending). Depends on: phases 1 and 2; V8, V11, V12, V22, V23, V25, V26. @@ -206,7 +206,7 @@ Exit criteria: ## Phase 5: before org-wide enforcement -- [ ] Owner: accept in writing the residuals in the spec's "Which run counts": an earlier pass on the same SHA, and run deletion if V6 f did not pass. +- [ ] Owner: accept in writing the residuals in the spec's "Which run counts": an earlier pass on the same SHA, and run deletion (V6 f confirmed it as a Write user). - [ ] Owner: fix the existing disabled org ruleset "All checks must pass" before it is enabled: - [ ] dismiss stale approvals on push: on - [ ] `OrganizationAdmin` bypass mode: `pull_request`, not `always` diff --git a/docs/claude-review-design.md b/docs/claude-review-design.md index dad8f59..286fa57 100644 --- a/docs/claude-review-design.md +++ b/docs/claude-review-design.md @@ -104,14 +104,14 @@ Facts about ruleset-required workflows ([troubleshooting rules](https://docs.git | Role | What it can still do against the gate | |---|---| -| Outside contributor (fork PR) | Nothing found. The rule ignores jobs from the PR's own workflows. A fork from a non-member account is still to test (V8). Cannot cancel, re-run or delete runs. | -| Org member without write access to the repo | Same as an outside contributor (V8). | -| Repo writer | Cannot forge or remove the rule. Can approve another writer's PR, which is the 1 approval a merge needs. Can cancel, re-run and delete workflow runs ([workflow runs API](https://docs.github.com/en/rest/actions/workflow-runs#delete-a-workflow-run)). A cancel only blocks. A re-run reads the live PR state. Deleting a newer run on a head SHA can expose an older pass on that SHA (V6 f). | +| Outside contributor (fork PR) | Nothing found. The rule ignores jobs from the PR's own workflows. A fork from a non-member account was tested: the review starts with no approval click and a fake or same-path workflow does not replace it (V8, V23). Cannot cancel, re-run or delete runs. | +| Org member without write access to the repo | Same as an outside contributor. Not tested with a separate account. | +| Repo writer | Cannot forge or remove the rule. Can approve another writer's PR, which is the 1 approval a merge needs. Can cancel, re-run and delete workflow runs ([workflow runs API](https://docs.github.com/en/rest/actions/workflow-runs#delete-a-workflow-run)). A cancel only blocks. A re-run reads the live PR state. Deleting a newer run on a head SHA can expose an older pass on that SHA. Confirmed as a Write user (V6 f, PR 41). | | Repo maintainer | Same as a repo writer under the org ruleset. On a pilot repo, can merge past a failed run with the pilot bypass, also on their own PR, which GitHub records. | | Repo admin | Cannot remove or change the rule. Same run powers as a writer. Can change repo-level Actions settings, which do not affect the required run (V12, phase 3: with Actions disabled, and with an allowed-actions list that excludes the actions the workflow uses, the required run still started and ran). On a pilot repo, can merge past a failed run with the pilot bypass. | | Org admin | Bypasses the ruleset through a pull request, and changes it. | -- **Why a same-name job cannot stand in for the gate.** In [claude-review-test](https://github.com/OperationCode/claude-review-test), a repo ruleset that required a check by name was satisfied by a passing job from a fork PR's own workflow (the positive control). Under the workflows rule, the same fake job did not satisfy the rule when it passed, and did not block it when it failed. This held on fork and same-repo PRs, when the fake finished after the required run, and when it finished 179 seconds before it (T6, T7). In these tests the scratch repo was both the workflow source and the target, and the pin was the tip of main. Not yet tested: a fake job that starts before the required run, and a fork from a non-member account (V8); a separate source repo with a pin behind main (V22); a same-path workflow in the target (V23). +- **Why a same-name job cannot stand in for the gate.** In [claude-review-test](https://github.com/OperationCode/claude-review-test), a repo ruleset that required a check by name was satisfied by a passing job from a fork PR's own workflow (the positive control). Under the workflows rule, the same fake job did not satisfy the rule when it passed, and did not block it when it failed. This held on fork and same-repo PRs, when the fake finished after the required run, and when it finished 179 seconds before it (T6, T7). In these tests the scratch repo was both the workflow source and the target, and the pin was the tip of main. A fork from a non-member account with no repo access (account X, before it became a collaborator) did the same (V8 a, V23). From a first-time fork, GitHub holds a fake `pull_request` job for approval, so it cannot start before the required run; after a maintainer approved it, the box still followed the required run (V8 a, PR 39). The fake-first case is also covered by the same-repo test (PR 36). A same-path workflow in the target did not replace the required run (V23, PR 40). Tested before: a separate source repo with a pin behind main (V22). - Every merge still needs 1 human approval, except a merge by an org admin through the bypass. - An earlier pass on a head SHA can still count after a revoke, after the head returns to that SHA, or after a newer run is deleted, until a newer run on that SHA exists. With no newer run there is no time limit (see "Which run counts"). @@ -531,7 +531,7 @@ The gate record is a run of the central workflow on the PR head SHA. GitHub crea ``` - **Pin rollback (V22 b).** A move of the pin back to an earlier release SHA revives the passes at that SHA (see "Pinned SHA only"). This is a residual of the same kind as the list above. -- **Exact residual.** Invalidating an override (revoke, or the grant author's role removal) takes effect only when a newer run on that head exists. If nobody starts one, it never takes effect. The same holds for an H to B to H return pushed with a `GITHUB_TOKEN`, a branch restore that starts no run, and a deleted newer run if V6 (f) shows the older pass decides. Between a reopen and the creation of its run, the old pass can also count (V6 a records the gap). This is a reduction of the gate guarantee. The owner accepts it in writing before org-wide enforcement (rollout step 7). Strict invalidation needs another gate design: a completed run cannot react to an event that starts no run. +- **Exact residual.** Invalidating an override (revoke, or the grant author's role removal) takes effect only when a newer run on that head exists. If nobody starts one, it never takes effect. The same holds for an H to B to H return pushed with a `GITHUB_TOKEN`, a branch restore that starts no run, and a deleted newer run (V6 f: the older pass decides, confirmed as a Write user). Between a reopen and the creation of its run, the old pass can also count (V6 a records the gap). This is a reduction of the gate guarantee. The owner accepts it in writing before org-wide enforcement (rollout step 7). Strict invalidation needs another gate design: a completed run cannot react to an event that starts no run. - The ruleset blocks force pushes on the base branch only, not on the head branch, so H to B to H is possible. - In each case the code that can merge is code that an earlier run of the pinned workflow passed. Human approval is a separate control. The push of B dismissed every earlier approval ("dismiss stale approvals on push"), so a merge needs a fresh approval. Role removal on an unchanged head is not a push and dismisses nothing. A new approval does not re-check the old override. - **Cross-PR effect (V6 g2).** Two PRs with one head H share the runs on H. A failing or pending run on PR 2 blocks PR 1. A pass on PR 2 clears PR 1, also when PR 1 has no override. An override covers the code pair (head and merge base), not one PR, so an override on PR 2 can clear PR 1. Anyone who can open a PR with head H (a fork can fetch the commit) can start a run on H, so they can block PR 1 or start a new run on it. @@ -733,7 +733,7 @@ token lifetime: 10 minutes - override: fail, then override, then re-run the newest run; fail, then override, then re-run the newest run (accepted); fail, then reopen, then override, then re-run (accepted); fail, then override, then reopen (rejected: override older than the `reopened` event); push invalidates; delete before and during publish; approver role removed; non-waivable reason ignores it - revoke: pass by override, delete the comment, close and reopen: the box blocks from the new run's creation (V6) - recovery: fail on a hard block or registry outage, then pass on a later run at the same head - - pilot bypass: under a ruleset shaped like the pilot ruleset, a failed run shows the red X; a user with Maintain and a user with Admin can merge with the bypass, a user with Write cannot, and GitHub records the bypass (V25) + - pilot bypass: under a ruleset shaped like the pilot ruleset, a failed run shows the red X; a user with Maintain and a user with Admin can merge with the bypass, and GitHub records the bypass (V25: Maintain and Admin pass; the Write bypass was not run, a plain merge was refused and Write is not on the bypass list; a direct push to the default branch is refused for Maintain) - gate job failure at each point: a normal run cancel before the gate job starts does not stop it (the run is `cancelled`, the gate job still runs, the box blocks); a force-cancel stops the gate job, and is the way to cancel it after inline comments and after the summary; summary write refused; each must leave the run failed or cancelled and the box blocked - whole run cancelled during collect or analyze: the run fails and the box blocks - two PRs to the default branch with the same head SHA (from two branches): record which run decides each (V6 g) @@ -751,23 +751,26 @@ token lifetime: 10 minutes ## Phase 3 results (smoke test on claude-review-test, 2026-10-27) -Target: [claude-review-test](https://github.com/OperationCode/claude-review-test), test ruleset TR (org ruleset 25835887) pinned to the candidate or release SHA, bypass list empty. S1 = `3aa7ae9ec072ca5d77b895dac12665f2370983de`. M1 = `590a387333c999a5d30dad8022a00383251ce16a`, the squash merge of S1 (PR 9). Times are UTC. The V items above hold the detail for V6 f, V8, V11, V12, V22, V23, V25 and V26. Cases that need a non-member fork author (account X, being arranged by the owner) are marked NOT YET RUN. +Target: [claude-review-test](https://github.com/OperationCode/claude-review-test), test ruleset TR (org ruleset 25835887) pinned to the candidate or release SHA, bypass list empty. S1 = `3aa7ae9ec072ca5d77b895dac12665f2370983de`. M1 = `590a387333c999a5d30dad8022a00383251ce16a`, the squash merge of S1 (PR 9). Times are UTC. The V items above hold the detail for V6 f, V8, V11, V12, V22, V23, V25 and V26. Account X is `irvingpop2`, a separate GitHub account outside the org that the owner made as a test identity; it became an outside collaborator when it got repo access. Its steps ran from an isolated container, with Write, then Maintain, then Admin on claude-review-test. The owner ran the bypass merges and the direct push by hand, because the automation's safety classifier blocks bypass merges; the evidence comes from the GitHub API. Cases still not run are marked NOT RUN. | Item | Result | Evidence | |---|---|---| | Release eval at S1 | Pass: 13 of 13 matched, 0 missed | [`.github` run 40528717338](https://github.com/OperationCode/.github/actions/runs/40528717338), approved 2026-10-27 15:13:14Z; tokens: input 39399, output 19901, cache write 56524, cache read 113048 | | M1 tree check | Pass: S1 and M1 have tree `6ace36e90f776f1fd9e821851b42dfc79dad13c0`; `git diff S1 M1` empty | `origin/main` = M1 | -| V26 a, b, c | a partial (R1 and R2 created at the placeholder and accepted by the Console; no read-back recorded); b pass; c partial (see V26 c: no R2 read-backs, step 5a not run for R2 and M2) | The owner created R1 and R2 at the placeholder and the Console accepted them; runs [40560632129](https://github.com/OperationCode/claude-review-test/actions/runs/40560632129) (accept), [40561015054](https://github.com/OperationCode/claude-review-test/actions/runs/40561015054) (deny), [40585724589](https://github.com/OperationCode/claude-review-test/actions/runs/40585724589) (S2 via R2), [40587964524](https://github.com/OperationCode/claude-review-test/actions/runs/40587964524) (M2 via R2) | +| V26 a, b, c | a done for `job_workflow_ref` (R1 and R2 created at the placeholder and accepted by the Console; read back at M1 and M2, not at the placeholder: the owner read back R1 at M1 `590a387` and R2 at M2 `1cefe42` on 2026-10-27, before the M3 edit; the service account and the workspace were not read back); b pass; c partial (see V26 c: deny at M2 before the R2 edit NOT RUN) | The owner created R1 and R2 at the placeholder and the Console accepted them; runs [40560632129](https://github.com/OperationCode/claude-review-test/actions/runs/40560632129) (accept), [40561015054](https://github.com/OperationCode/claude-review-test/actions/runs/40561015054) (deny), [40585724589](https://github.com/OperationCode/claude-review-test/actions/runs/40585724589) (S2 via R2), [40587964524](https://github.com/OperationCode/claude-review-test/actions/runs/40587964524) (M2 via R2) | | Deny at the merge SHA, before the second edit | Pass: run [40583290922](https://github.com/OperationCode/claude-review-test/actions/runs/40583290922) at M1, PR 28 reopened 21:02:02 to 21:02:08Z; 3 passes `failed (authentication failed)`; gate fail; box blocked from 21:02:01Z | Exchange window 21:02:36 to 21:02:57Z, R1 then at S1. Console reason not read | | Accept at M1 after the second edit | Pass: run [40585413003](https://github.com/OperationCode/claude-review-test/actions/runs/40585413003), PR 28 reopened 21:16:44Z, 3 `ok`, box clear 21:18:12Z; PR 37: run [40586089942](https://github.com/OperationCode/claude-review-test/actions/runs/40586089942), clear 21:22:46Z | R1 at M1 (no read-back or Console evidence recorded, only the working token); M1 exchange passed; phase 3 exit criteria remain open | -| Simulation (R2, S2, M2) | Partial pass: see V26 c. Remaining: R2 read-backs at the placeholder, S2 and M2; a denied exchange at M2 before the R2 edit to M2 | S2 `9752ba7c04d796822c64c670415d1046607c3c03`, M2 `1cefe4259bc26197ec4b063e8399ef856efdd4f2`, PR 10 into `p3-sim-base`, never main | +| Simulation (R2, S2, M2) | Partial pass: see V26 c. Remaining: R2 read-backs at the placeholder and at S2; a denied exchange at M2 before the R2 edit to M2 | S2 `9752ba7c04d796822c64c670415d1046607c3c03`, M2 `1cefe4259bc26197ec4b063e8399ef856efdd4f2`, PR 10 into `p3-sim-base`, never main | | V22 a, V22 b | Pass, with a rollback residual (see "Which run counts") | V22 | | V11 | Pass | V11 | | V12 | Pass; Actions settings do not affect the required run | V12 | -| V6 f | Residual confirmed as an org admin; NOT YET RUN as a Write user | V6 | -| V8, V23 | NOT YET RUN (pending account X); partial same-repo evidence | V8, V23 | -| V25 | NOT YET RUN (pending account X) | V25 | -| Web UI "Re-run all jobs" | NOT YET RUN (pending account X) | "Verify later" | +| V6 f | Residual confirmed as an org admin and as a Write user: deleting the newer run reopened the box on the older pass (PR 41, run 40623844194, deleted in the test; box BLOCKED then CLEAN at once and for 3 readings) | V6 | +| V8 a | Pass: from a first-time fork the fake `pull_request` run waited as `action_required`; after a maintainer approved only that run it passed, and the box stayed BLOCKED and followed the central run (PR [39](https://github.com/OperationCode/claude-review-test/pull/39)). Fake-first stays covered by the same-repo evidence (PR 36) | V8 | +| V8 b | Pass: the central `pull_request_target` run started on a first-time-contributor fork PR with no approval click, and marking the draft ready started no new run (PR [38](https://github.com/OperationCode/claude-review-test/pull/38), run [40621436811](https://github.com/OperationCode/claude-review-test/actions/runs/40621436811)) | V8 | +| V23 | Pass for the central run: a same-path `claude-review.yml` in a fork PR did not replace the central run, which failed on the bidi hard block; the fork run stayed `action_required` and never ran, so the running case is covered by PR 35 (PR [40](https://github.com/OperationCode/claude-review-test/pull/40)) | V23 | +| V25 | Pass for Maintain and Admin; Write bypass NOT RUN: with the TR bypass list, Maintain (PR [42](https://github.com/OperationCode/claude-review-test/pull/42)) and Admin (PR [43](https://github.com/OperationCode/claude-review-test/pull/43)) merged a failed PR with the bypass; Write: a plain merge was refused (PR 40, "base branch policy prohibits the merge"), which does not test the bypass, and Write is not on the bypass list; a direct push to main by Maintain is refused (GH013) | V25 | +| Write user, `--admin` merge (Write bypass) | NOT RUN: the automation's classifier blocked it | V25 | +| Web UI "Re-run all jobs" | Pass: the owner's click made attempt 2 of the same run ID (run [40628870689](https://github.com/OperationCode/claude-review-test/actions/runs/40628870689), PR 43, `pull_request_target`, head 423e04c; actor `irvingpop2`, because the browser was signed in as the test account, which had Admin). Attempt 1 ended `failure` (03:50:56Z, bidi hard block); attempt 2 ended `success` (04:23:46Z) with `collect`, `analyze`, `dependency-review` and `oc-claude-review` all `success`, and result `pass` ("No reason applies", 0 hard blocks, 1 signal `changed-files-differ`, no dependency delta). PR 43 had been merged with a bypass first, so the live comparison no longer held the bidi file: the re-run re-read live state. The one summary comment was updated in place (created 03:50:51Z, updated 04:23:40Z) and names attempt 2. The merge box was not observed (PR already merged) | "Verify later" | | Outage: invalid federation rule | Pass, live: two denied exchanges | runs 40561015054 and 40583290922 | | Outage: exhausted budget | Covered by V9 and the unit tests; no Console limit edit (ruling Q7, P3-1) | V9 | @@ -793,7 +796,7 @@ Acceptance list, run without account X (V6 gives 33 run attempts and 6,153 obser | Re-run posts no duplicate inline comment | Pass on PR 34 (0 new). PR 23 got 3 new comments on the re-run, because the model re-worded the quote: recorded | runs [40567379756](https://github.com/OperationCode/claude-review-test/actions/runs/40567379756) and [40568067319](https://github.com/OperationCode/claude-review-test/actions/runs/40568067319) | | Egress audit mode | Pass: `egress_policy: audit` and the V10 lists in every job log | run logs | | Approver role removed; edited override; artifact upload removed | NOT RUN: need a second account or a workflow change at S1; unit tests cover them | none | -| Fork PR, draft then ready (V8 b); V8 a and V23 from a fork; V6 f as Write; V25 | NOT YET RUN (pending account X) | none | +| Fork PR, draft then ready (V8 b); V8 a and V23 from a fork; V6 f as Write; V25 | Run with account X (2026-10-28): V8 b, V8 a and V23 pass; V25 passes for Maintain and Admin (Write bypass NOT RUN); V6 f residual confirmed | PRs 38 to 43, see the table above | ## Rollout @@ -805,7 +808,7 @@ The tasks, owner actions and exit criteria for each step are in [docs/build-plan 4. **Fixture evals and acceptance:** the fixture eval passes, and the workflow acceptance tests pass in the scratch repo, including a fork PR from an account outside the org. 5. **Pilot:** the "blocking" items must pass first. Check that the federation rule named in the release code has the release SHA. An org owner gives each pilot repo its own approval protection, then creates the pilot ruleset (see "Pilot and enforcement") with the release SHA and the pilot repos (first `back-end`), reads it back, and enables it. Then list open PRs and start a run on each (push, or close and reopen), because the rule does not start on PRs that were already open. Watch the real results on real PRs: false positives, overrides, bypasses, egress and cost per run. The owner decides when the pilot is done. 6. **Each release** (from the pilot on): follow "Each release" in "Auth and cost". Create a new federation rule at the placeholder SHA, put its ID in the candidate code, and set the rule to the candidate SHA. Validate the candidate before any pin moves: unit tests, the owner-approved fixture eval and a token exchange under the scratch test ruleset. If the merge makes a new SHA, its tree must equal the tested candidate. Move the pin of every ruleset that requires the workflow and read each back. Start a new run on each open PR, and delete the old rule only when each has completed at the release SHA (V22). -7. **Before org-wide enforcement:** the owner accepts in writing the residuals in "Which run counts" (an earlier pass on the same SHA, the cross-PR effect of one head SHA, a pin rollback, and run deletion if V6 f did not pass). An org owner fixes the org ruleset (see "Admin checklist"). Build the supported repo inventory (see "Supported repos"), and add CODEOWNERS to each supported repo. Revisit harden-runner block mode (K-14). Re-run the key acceptance tests. +7. **Before org-wide enforcement:** the owner accepts in writing the residuals in "Which run counts" (an earlier pass on the same SHA, the cross-PR effect of one head SHA, a pin rollback, and run deletion, which V6 f confirmed for a Write user). An org owner fixes the org ruleset (see "Admin checklist"). Build the supported repo inventory (see "Supported repos"), and add CODEOWNERS to each supported repo. Revisit harden-runner block mode (K-14). Re-run the key acceptance tests. 8. **Org-wide enforcement:** enable the org ruleset with the supported repos that are not in the pilot, in small groups. Then move each pilot repo from the pilot ruleset to the org ruleset (see "Pilot and enforcement"), and delete the pilot ruleset. After each group, start a run on each open PR, and check that each shows the required workflow passing or failing on its own result. ## Admin checklist (Irving) @@ -834,13 +837,13 @@ These are assumptions I could not confirm from docs. Each has a concrete test. I 3. **Answered: federation rule (build).** 2026-10-26 (UTC), pass after one fix. The final rule text is in "Auth and cost". The test rule on the test workspace used pattern mode, the subject prefix `repo:OperationCode@10479364/*`, and exact claims `event_name`, `repository_owner`, `repository_owner_id` and `job_workflow_ref` (the scratch `p0-required.yml` at one SHA). The first rule had the prefix `repo:OperationCode/*` and denied every token with reason `match_subject_prefix`, because `sub` uses the ID form. CEL mode refused the planned compound condition, so the rule has no `startsWith` and no SHA list. With the fixed prefix, the ruleset-required run of the pinned workflow got HTTP 200: a token with scope `workspace:developer` for the test workspace, valid for 10 minutes. Four denies got HTTP 401 with the generic `authentication_error`; the Console reason for each was `match_claim_value_mismatch`: another repo and workflow on `pull_request_target`; a `push` run; a `workflow_dispatch` run of the pinned file (its `job_workflow_ref` uses `@refs/heads/main`); and the same workflow at an SHA that is not in the rule. `pull_request_target` ran the file from the default branch of the target repo, not from the PR base branch (see "Workflow details"). Evidence: accept [40258763277](https://github.com/OperationCode/claude-review-test/actions/runs/40258763277) (PR [14](https://github.com/OperationCode/claude-review-test/pull/14)); other repo [40258766911](https://github.com/OperationCode/claude-review-test/actions/runs/40258766911) (PR [24](https://github.com/OperationCode/claude-review-test/pull/24)); push [40258769681](https://github.com/OperationCode/claude-review-test/actions/runs/40258769681); dispatch [40259047420](https://github.com/OperationCode/claude-review-p0-source/actions/runs/40259047420) in [claude-review-p0-source](https://github.com/OperationCode/claude-review-p0-source); unapproved SHA [40259088062](https://github.com/OperationCode/claude-review-test/actions/runs/40259088062) (PR [14](https://github.com/OperationCode/claude-review-test/pull/14)). No log held a JWT or an access token. 4. **Answered: Dependabot PRs (build).** 2026-10-25, pass. Dependabot PR [23](https://github.com/OperationCode/claude-review-test/pull/23) got the ruleset-required run as `dependabot[bot]` (event `pull_request_target`). Analyze minted an OIDC token; no secret was needed. `job_workflow_sha` equaled the pinned SHA. The gate failed as set, and the box blocked (read once with `gh pr view`, not with the observer). An override and a reopen gave a passing run, and the box cleared. `sub` does not name the actor. The federation exchange on a Dependabot run is not tested. V3 passed on other PRs, and the rule matches no actor claim. Evidence: runs [40243816279](https://github.com/OperationCode/claude-review-test/actions/runs/40243816279) (fail) and [40243930238](https://github.com/OperationCode/claude-review-test/actions/runs/40243930238) (pass after override). The docs say nothing about `id-token` for Dependabot ([troubleshooting Dependabot](https://docs.github.com/en/code-security/dependabot/troubleshooting-dependabot/troubleshooting-dependabot-on-github-actions)). 5. **Answered: mid-conversation system message plus structured output plus server-side fallback** in one request (build). 2026-10-26 (UTC), pass. One request through the federated token: `claude-opus-5-5`, top-level `system` with ``, the nonce-delimited bundle with `cache_control`, a mid-conversation `system` message as the last entry, `output_config.format` with a stripped findings schema, `fallbacks: "default"` with the beta header, effort `medium`. HTTP 200, `stop_reason` `end_turn`, top-level `model` `claude-opus-5-5`. The output parsed, and it matched both the sent schema and the full schema in a client check. The model followed the mid-conversation message. It returned one correct, grounded finding (the `if (user.role = "admin")` assignment). The cache write covered the system and the bundle, so the prefix before the pass message can be shared. The fallback did not trigger, so a real fallback attempt is not tested. The request left out the schema keywords that structured outputs do not support (see "Request shape"). The fallback in this item (pass instructions in the top-level `system`) is not needed. Evidence: run [40261749299](https://github.com/OperationCode/claude-review-test/actions/runs/40261749299) (PR [26](https://github.com/OperationCode/claude-review-test/pull/26)). -6. **Which run counts (build, first).** The docs do not say which run decides when a head SHA has several runs of the required workflow. The design requires newest-run selection (see "Which run counts"). Run on fork and same-repo PRs, with a separate source and target repo. A scratch-only build of the gate job may pause for a set time after its final check, before it exits. Criteria: with no earlier pass on the head, the box blocks unless the newest run completed with `success`; with an earlier pass, the box blocks from the moment a newer run exists, in every state but `success`. **Phase 3 (2026-10-27), (f) as an org admin, not a Write user: residual confirmed. NOT YET RUN as a Write user (pending account X).** PR 32 had an older pass (run 40564294168) and 4 newer cancelled runs. Deleting them newest first (19:49:16 to 19:50:06Z) gave blocked, blocked, blocked, then clear after the last delete, and still clear 20 s later. This is the residual the spec states; a Write user can delete runs and is expected to see the same. +6. **Which run counts (build, first).** The docs do not say which run decides when a head SHA has several runs of the required workflow. The design requires newest-run selection (see "Which run counts"). Run on fork and same-repo PRs, with a separate source and target repo. A scratch-only build of the gate job may pause for a set time after its final check, before it exits. Criteria: with no earlier pass on the head, the box blocks unless the newest run completed with `success`; with an earlier pass, the box blocks from the moment a newer run exists, in every state but `success`. **Phase 3 (2026-10-27), (f) as an org admin, not a Write user: residual confirmed. Also confirmed as a Write user on 2026-10-28 (see the Phase 3 result below).** PR 32 had an older pass (run 40564294168) and 4 newer cancelled runs. Deleting them newest first (19:49:16 to 19:50:06Z) gave blocked, blocked, blocked, then clear after the last delete, and still clear 20 s later. This is the residual the spec states; a Write user can delete runs and is expected to see the same. - (a) Replacement states: H passes by override; revoke; a reopen starts a new run, slowed on purpose. Pass when the box blocks while the new run is queued and in progress, after it is cancelled, after it times out and after it fails, and clears when it passes. Record the gap between the reopen and the run's creation. With no reopen (absent replacement), record that the old pass stays usable (the stated residual). - (b) Completion order: run A passes its final check and pauses; the override is revoked; run B starts and fails; A then exits 0 and finishes last. Pass when the box blocks after A completes. Reverse: B passes; A, which saw B, exits `superseded` and finishes last. Record whether the box follows B (liveness). - (c) H to B to H: after H passed by override, force-push the head to B and back to H. Pass when the box blocks while the new run on H is pending and after it fails. Repeat with the return push made with a `GITHUB_TOKEN`: record that no run starts and that the box shows the old pass. Check that the push of B dismissed approvals. - (d) Cancel a run during collect. Pass when the run ends failed or cancelled and the box blocks. - (e) Re-runs: re-run all jobs on the newest run after an override; pass when the box follows the new attempt. With an older passed run A and a newer failed run B, re-run A; pass when the box stays blocked. - - (f) Run deletion (as a repo writer): with an older pass on the head, delete a newer failed, cancelled or timed-out run, and a pending one where GitHub allows it. Pass when the box still blocks. Check both merge eligibility and the run list. + - (f) Run deletion (as a repo writer): with an older pass on the head, delete a newer failed, cancelled or timed-out run, and a pending one where GitHub allows it. Pass when the box still blocks. Check both merge eligibility and the run list. **Phase 3 (2026-10-28): residual CONFIRMED as a Write user** (PR 41, account X). With an older pass A (run 40623697490) and a newer cancelled run B (run 40623844194, check FAILURE) on the head, the box was BLOCKED. The Write user deleted B with the workflow runs API (exit 0, allowed). The box turned CLEAN at once and stayed CLEAN for 3 readings on the older pass. Keep it in the residuals the owner accepts before org-wide enforcement. B ended cancelled, not failed, as in PR 32. - (g) Delete and restore the head branch at H; and open a second PR to the default branch with head H. Record which run decides each PR. If (a), (b), (c) with a user push, or (e) shows that an older or the last completed run decides, building stops and the gate design changes. If (f) does not pass, repo writers can restore an older pass on the same SHA; the owner accepts that trust boundary before org-wide enforcement or the design changes. - **Result (2026-10-25): stop rule not triggered.** The newest run on the head SHA decided the box on same-repo PR [14](https://github.com/OperationCode/claude-review-test/pull/14) and fork PR [15](https://github.com/OperationCode/claude-review-test/pull/15). In 4,168 observer readings, no valid reading was mergeable while the newest run was not `success`. @@ -849,7 +852,7 @@ These are assumptions I could not confirm from docs. Each has a concrete test. I - **Still open:** the queued cells and n1 (need an owner repo variable); (f) run deletion by a writer account; the approval-dismissal part of (c) (needs a second account); (g1) with the real Restore button, and on the same-repo PR; the default-branch check (`~DEFAULT_BRANCH`). "Which run counts" now has the cross-PR result (g2). It changes again when (f) has a result. - Evidence, same-repo run then fork run: (a) in progress and failed [40235495107](https://github.com/OperationCode/claude-review-test/actions/runs/40235495107) and [40235493548](https://github.com/OperationCode/claude-review-test/actions/runs/40235493548); (a) cancelled [40235689574](https://github.com/OperationCode/claude-review-test/actions/runs/40235689574) and [40235689417](https://github.com/OperationCode/claude-review-test/actions/runs/40235689417); (a) timed out [40235846948](https://github.com/OperationCode/claude-review-test/actions/runs/40235846948) and [40235918049](https://github.com/OperationCode/claude-review-test/actions/runs/40235918049); (a) cleared [40236150113](https://github.com/OperationCode/claude-review-test/actions/runs/40236150113) and [40236222191](https://github.com/OperationCode/claude-review-test/actions/runs/40236222191); (b1) A [40236289458](https://github.com/OperationCode/claude-review-test/actions/runs/40236289458) and [40236352621](https://github.com/OperationCode/claude-review-test/actions/runs/40236352621), B [40236386756](https://github.com/OperationCode/claude-review-test/actions/runs/40236386756) and [40236446523](https://github.com/OperationCode/claude-review-test/actions/runs/40236446523); (b2) A [40236783304](https://github.com/OperationCode/claude-review-test/actions/runs/40236783304) and [40236844122](https://github.com/OperationCode/claude-review-test/actions/runs/40236844122), B [40236801503](https://github.com/OperationCode/claude-review-test/actions/runs/40236801503) and [40236861667](https://github.com/OperationCode/claude-review-test/actions/runs/40236861667); (c) user push [40239816420](https://github.com/OperationCode/claude-review-test/actions/runs/40239816420) and [40239867888](https://github.com/OperationCode/claude-review-test/actions/runs/40239867888); (c) token push [40240144869](https://github.com/OperationCode/claude-review-test/actions/runs/40240144869) (same-repo only); (d) [40241159046](https://github.com/OperationCode/claude-review-test/actions/runs/40241159046) and [40243374015](https://github.com/OperationCode/claude-review-test/actions/runs/40243374015); (e) re-run newest [40237033260](https://github.com/OperationCode/claude-review-test/actions/runs/40237033260) and [40237091065](https://github.com/OperationCode/claude-review-test/actions/runs/40237091065); (e) re-run older [40237142925](https://github.com/OperationCode/claude-review-test/actions/runs/40237142925) and [40237192544](https://github.com/OperationCode/claude-review-test/actions/runs/40237192544); (g1) fork only [40242670876](https://github.com/OperationCode/claude-review-test/actions/runs/40242670876); (g2) second PRs [21](https://github.com/OperationCode/claude-review-test/pull/21) run [40240449144](https://github.com/OperationCode/claude-review-test/actions/runs/40240449144) and [20](https://github.com/OperationCode/claude-review-test/pull/20) run [40240149876](https://github.com/OperationCode/claude-review-test/actions/runs/40240149876). 7. **Answered: head SHA.** Job checks and required runs of a `pull_request_target` workflow land on the PR head SHA, on fork and same-repo PRs, although `GITHUB_SHA` is the default branch commit (T4, T6). -8. **Forging (blocking).** Answered in part: a rule bound to a check name was satisfied by a fake job from a fork PR's own workflow; the workflows rule ignored the same fake job when it passed or failed, when it finished after the required run, and when it finished 179 seconds before it (T6, T7). The fake always started after the required run, the forks belonged to org admins, and the scratch repo was both workflow source and target. Use a source repo separate from the target. Still to test with an account that is not an org member, and with an org member without write access: (a) a fork PR whose own workflow has a same-name job that starts before the required run; pass when the box follows only the required run; (b) the real `pull_request_target` review starts with no approval click; if it waits, the goal "no maintainer step" fails for outside contributors and the owner decides. **Phase 3 (2026-10-27): NOT YET RUN (pending account X).** The fork cases need a non-member author. Partial evidence only, from a same-repo PR by an org admin (PR 36, V8 a): a fake job `oc-claude-review` in `.github/workflows/p3-fake-gate.yml` (`on: pull_request`) started first (run 40572853203, lower ID than the central run 40572853562) and passed at 19:51:35Z; the central run failed at 19:53:01Z; the box stayed blocked throughout. +8. **Forging (blocking).** Answered in part: a rule bound to a check name was satisfied by a fake job from a fork PR's own workflow; the workflows rule ignored the same fake job when it passed or failed, when it finished after the required run, and when it finished 179 seconds before it (T6, T7). The fake always started after the required run, the forks belonged to org admins, and the scratch repo was both workflow source and target. Use a source repo separate from the target. Still to test with an account that is not an org member, and with an org member without write access: (a) a fork PR whose own workflow has a same-name job that starts before the required run; pass when the box follows only the required run; (b) the real `pull_request_target` review starts with no approval click; if it waits, the goal "no maintainer step" fails for outside contributors and the owner decides. **Phase 3 (2026-10-28): PASS, with account X (`irvingpop2`, outside the org).** (b) PR 38 (fork, draft then ready): the central `pull_request_target` run 40621436811 started 3 seconds after the PR opened as a draft, with no approval click, and passed; marking the PR ready started no new run, and no `pull_request` run exists for the head SHA. (a) PR 39 (fork): the central run 40622846242 and the fake `pull_request` run 40622846337 were created at the same second (the central run has the lower ID). GitHub held the fake run for approval (`action_required`; the author is a first-time contributor), so the fake job cannot start before the central run from a first-time fork. The central review failed (a waivable blocking finding on the fake workflow file), and the box was BLOCKED. After a maintainer approved only the fake run, it passed, the rollup showed two `oc-claude-review` checks (fake SUCCESS, central FAILURE), and the box stayed BLOCKED and followed the central run. Record (a) as pass with this note: the fake-first case stays covered by the earlier same-repo evidence. Not tested: an org member without write access. Earlier evidence from a same-repo PR by an org admin (PR 36, V8 a): a fake job `oc-claude-review` in `.github/workflows/p3-fake-gate.yml` (`on: pull_request`) started first (run 40572853203, lower ID than the central run 40572853562) and passed at 19:51:35Z; the central run failed at 19:53:01Z; the box stayed blocked throughout. 9. **Answered: spend limit behavior (build).** 2026-10-26 (UTC), pass. Test workspace with a deliberately tiny hard monthly limit, sequential calls of about 61k input tokens each, through the federated token. Calls 1 to 7 got HTTP 200. Call 8 got HTTP 400 `invalid_request_error` "You have reached your specified workspace API usage limits. You will regain access on 2026-11-01 at 00:00 UTC." A fresh token exchange after that still got HTTP 200, and a call with the new token got the same 400. Analyze failed, the gate job failed with `needs.analyze=failure` (not waivable) even with a forced pass, and the merge box was `BLOCKED`. Overshoot: the estimate from `usage` passed the limit after call 4, and the first deny came about 39 s and 3 calls later. The overshoot equals the cost of the calls that started during the lag: spend reached 87 percent over the limit. The owner confirmed that the Console spend matched the estimate. This repo records no currency amounts (P0-17). Billing of cancelled calls is not tested. The design changes are in "Auth and cost". Evidence: run [40261904117](https://github.com/OperationCode/claude-review-test/actions/runs/40261904117) (PR [26](https://github.com/OperationCode/claude-review-test/pull/26)). 10. **Answered: egress allowlist (build).** 2026-10-27 (UTC), pass, with a harden-runner hang. Kits: the built workflow and `review/` staged in the scratch source repo as K1 `5a7a569` (audit), K2 `7399e8b` (block, `disable-telemetry: true`, `disable-sudo-and-containers: true`, per-job lists) and K3 `df8ff51` (K2 without `disable-sudo-and-containers`), with `setup-uv` v10.3.2 from K2 on. Each kit ran on same-repo PR 27 (clean), fork PR 19 (clean) and fork PR 18 (vulnerable dependencies). The audit run gave the endpoints per job (see "Workflow details"). In block mode no job needed a host off its list, every analyze job did the token exchange, and all 27 Claude passes were `ok`. A clean PR passed the gate in block mode (PR 19 in K2 and K3, PR 27 in K3 after a re-run), and PR 18 failed it through dependency review in every round. Hang: 2 of 24 block-mode jobs (one in K2, one in K3, both on PR 27; 2 of 6 block-mode runs) ran every step, then hung in "Complete job" until GitHub cancelled them about 15 minutes later. The gate failed closed. The K3 run passed on a REST re-run; the K2 run was not re-run. Only those 2 of the 36 jobs logged dropped IPs: Azure DNS and IPs of allowlisted hosts, after the agent restarted systemd-resolved, and then an Azure IP that the runner likely needs to finish the job. The symptoms match the open harden-runner [issue 708](https://github.com/step-security/harden-runner/issues/708), which reports that in block mode on GitHub-hosted runners with a GitHub DNS change ([runner-images issue 14925](https://github.com/actions/runner-images/issues/14925)), the agent drops Azure DNS `168.63.129.16` and the job hangs in cleanup. The runner region of the two hung jobs was not recorded, so the cause is not confirmed. v2.22.2, the latest release, has no fix, and audit mode is the known workaround. It is not a missing list entry. `disable-sudo-and-containers` is not the cause (the K3 hang had it off), but it adds a slow docker purge. Ruling: audit mode with the measured lists (K-14). Evidence: runs in claude-review-test; audit [40433731499](https://github.com/OperationCode/claude-review-test/actions/runs/40433731499), [40433735510](https://github.com/OperationCode/claude-review-test/actions/runs/40433735510), [40433740291](https://github.com/OperationCode/claude-review-test/actions/runs/40433740291); K2 [40434470537](https://github.com/OperationCode/claude-review-test/actions/runs/40434470537) (hang), [40434476879](https://github.com/OperationCode/claude-review-test/actions/runs/40434476879), [40434482049](https://github.com/OperationCode/claude-review-test/actions/runs/40434482049); K3 [40436332281](https://github.com/OperationCode/claude-review-test/actions/runs/40436332281) (hang, attempt 2 is the re-run), [40436337518](https://github.com/OperationCode/claude-review-test/actions/runs/40436337518), [40436343610](https://github.com/OperationCode/claude-review-test/actions/runs/40436343610). The endpoints are in each job's harden-runner post-step log; the kit diffs are in the owner's V10 test notes, outside this repo. 11. **Retarget (blocking).** The ruleset targets only the default branch, so a PR into another branch should start no required run. Test (a): open a PR to a staging branch that already holds a change not on the default branch; confirm that no required run starts; retarget to the default branch, and record whether the retarget starts a required run. Pass when the merge stays blocked until a review against the default branch passes (from a run that the retarget starts, or from a push or reopen), and that review shows the staging change in its diff. (b): the same, but the head SHA already has a passing required run from a PR to the default branch; record whether that pass counts after the retarget. **Result (phase 3, 2026-10-27): pass.** (a) PR 29 to a staging branch (staging = main plus one file) started no run in 60 s. The retarget to main at 18:37:27Z started no run and the box stayed blocked. A reopen at 18:38:43Z started run [40561903948](https://github.com/OperationCode/claude-review-test/actions/runs/40561903948): coverage "files in comparison 2" (the staging file and the PR file), 3 passes `ok`, pass; the box cleared only after it. (b) PR 30 (same head `1c432d5` as PR 28, which had an S1 pass) retargeted at 18:37:29Z started no run, and the box read mergeable: the pass from the other PR counted (per head SHA, V6 g2). @@ -864,15 +867,15 @@ These are assumptions I could not confirm from docs. Each has a concrete test. I 20. **Merged into V6.** On a new head SHA, the box blocked while the required run was pending (T7). 21. **Merged into V6.** 22. **Pin after `.github` main moves (blocking).** The rule accepted a SHA pin, and the run used it (T6), but the pin was still the head of main, and the scratch repo was both source and target. Use a source repo separate from the target. Before each step, read the ruleset back and check `repository_id`, path and `sha`, with no `ref`. (a) Pin S1, push a changed workflow to the source main, open a PR in the target; pass when the run uses S1 (`job.workflow_sha`). (b) Move the pin to a new SHA with the release steps in "Auth and cost" (a new federation rule for the new SHA); record whether runs from the old pin still satisfy the rule on open PRs. (c) Removed (K-2): there is no advisory release. **Result (phase 3, 2026-10-27): (a) pass.** `job.workflow_sha` equaled the TR pin at creation in all 33 run attempts (1 at `f808dd0`, 1 at the probe `13a1e48`, 31 at S1 `3aa7ae9`), including re-run attempts. The run at `f808dd0` equaled main then; the 31 at S1 ran while main was still `f808dd0`. **(b) pass, with a rollback residual.** Forward move M1 `590a387` to M2 `1cefe42` at 21:32:15Z: PRs 28 and 37, clear on their M1 passes at 21:32:04Z, read blocked at 21:32:17Z and 21:34:18Z (the M1 passes did not satisfy the rule). New runs at M2 ([40587964524](https://github.com/OperationCode/claude-review-test/actions/runs/40587964524), [40587977659](https://github.com/OperationCode/claude-review-test/actions/runs/40587977659); `job.workflow_sha` M2, 3 `ok`) cleared both at 21:38:23Z. In the probe case the newer run had failed and the box still read clear, so the newer run did not decide there. After the pin moved back to M1 at 21:38:43Z both read clear again at 21:38:44Z, 21:39:15Z and 21:40:16Z on the older M1 passes. The first answer came from S1 to the probe SHA and back (18:32:57Z to 18:34:53Z). See "Pinned SHA only" in "Which run counts". -23. **Same-path workflow in the target repo (blocking).** T6 was inconclusive: the PR's copy never ran, and the scratch repo was both source and target. Use a source repo separate from the target. Test: a fork PR adds `.github/workflows/claude-review.yml` to the target repo with a passing job `oc-claude-review`, while the central run fails. Pass when the box blocks. **Phase 3 (2026-10-27): NOT YET RUN (pending account X).** Partial evidence only, from a same-repo PR by an org admin (PR 35): the PR added `.github/workflows/claude-review.yml` with a passing job `oc-claude-review` (run 40572849151, higher ID, success at 19:51:34Z) and a bidi file. The central run [40572849014](https://github.com/OperationCode/claude-review-test/actions/runs/40572849014) failed (blocking-finding, prefilter-hard-block, `superseded`: the final check counted the same-path run). The box stayed blocked throughout and after. +23. **Same-path workflow in the target repo (blocking).** T6 was inconclusive: the PR's copy never ran, and the scratch repo was both source and target. Use a source repo separate from the target. Test: a fork PR adds `.github/workflows/claude-review.yml` to the target repo with a passing job `oc-claude-review`, while the central run fails. Pass when the box blocks. **Phase 3 (2026-10-28): PASS, with account X.** PR 40 (fork) added `.github/workflows/claude-review.yml`. The central `pull_request_target` run 40623048150 still ran and failed: a bidi hard block (`bidi-control`, not waivable) and two waivable blocking findings on the workflow file. The fork's same-path `pull_request` run 40623048638 stayed `action_required` and was not approved, so the fork job never ran; the running case is covered by PR 35 below. The box was BLOCKED, and the rollup showed only the central checks. Earlier evidence from a same-repo PR by an org admin (PR 35): the PR added `.github/workflows/claude-review.yml` with a passing job `oc-claude-review` (run 40572849151, higher ID, success at 19:51:34Z) and a bidi file. The central run [40572849014](https://github.com/OperationCode/claude-review-test/actions/runs/40572849014) failed (blocking-finding, prefilter-hard-block, `superseded`: the final check counted the same-path run). The box stayed blocked throughout and after. 24. **Answered: run list for the final check (build).** 2026-10-25, pass. The job's `GITHUB_TOKEN` with `actions: read` listed runs by head SHA on same-repo `pull_request` runs and on `pull_request_target` runs of fork and same-repo PRs, including ruleset-started runs, with the correct `path`. A new run first appeared in the list within 5 s of its creation in every measurement (poll step about 3 s). The final check stays defense in depth. Evidence: [40227607202](https://github.com/OperationCode/claude-review-test/actions/runs/40227607202) and [40227659578](https://github.com/OperationCode/claude-review-test/actions/runs/40227659578) (same-repo); [40230848821](https://github.com/OperationCode/claude-review-test/actions/runs/40230848821) (ruleset-started runs, PR [17](https://github.com/OperationCode/claude-review-test/pull/17)); b1 run A [40236352621](https://github.com/OperationCode/claude-review-test/actions/runs/40236352621) listed run B (fork PR [15](https://github.com/OperationCode/claude-review-test/pull/15)). -25. **Pilot bypass (blocking).** The docs list the `RepositoryRole` actor type but not the `actor_id` values of the built-in roles. Test: in the scratch repo, an org ruleset with only the workflows rule and a bypass for the Maintain and Admin repository roles in `pull_request` mode. Read it back and record the `actor_id` values. Pass when, on a failed run, a user with Maintain and a user with Admin can merge with the bypass, a user with Write cannot, and GitHub records each bypass. Also record whether a direct push to the default branch is blocked; before a repo joins the pilot, check it for people or bots that push to the default branch directly. **Phase 3 (2026-10-27): NOT YET RUN (pending account X).** It needs accounts with Write, Maintain and Admin on claude-review-test, and the bypass list on the test ruleset, which stays empty until then. The role `actor_id` values are not recorded yet. -26. **Placeholder SHA in a federation rule (verify in phase 3; part a partial 2026-10-27: creation accepted, read-back not recorded).** The setup and every release create a rule at the placeholder SHA `0000000000000000000000000000000000000000` and later edit it to a real SHA ("Each release" in "Auth and cost"). V5 and V9 edited a rule from one real SHA to another; no test created a rule with a SHA that matches no commit. Test: (a) at setup, create the production rule at the placeholder and read it back. If the Console rejects it, stop, and resolve the order of rule ID and SHA before the production setup goes on. Result of part a (2026-10-27): the owner created R1 and the Console accepted it; no read-back output is recorded, so part a is partial: placeholder creation accepted, read-back not recorded. Read R1 back to close it. R1 `claude-review-prod-r1` (`fdrl_01N2ZktTiVKqUvvXwywBFfys`) was created with `job_workflow_ref` at the placeholder SHA. The scratch rule R2 `claude-review-p3-r2` (`fdrl_01AfACzupW3U4Bx361ZUbsyR`; service account `claude-review-p0`, workspace `claude-review-p0-test`) was created the same way and stays at the placeholder until the release simulation of part c. (b) In phase 3, before the acceptance tests, edit the rule to the candidate SHA and read it back. Pass when a run under the scratch test ruleset at the candidate SHA gets a token, and a run at another SHA is denied. (c) Before the first release (V22 b), do steps 1 to 6 of "Each release" with a scratch rule R2 on the test workspace, including the re-edit of R2 to the merge SHA and the second exchange. Do not change R1 during this check. Pass when R2 reads back at the placeholder, then at the candidate SHA, then at the release SHA, and the exchange succeeds at both real SHAs. The scratch ruleset is pinned to a SHA that is on a non-default branch of `.github` at step 4 (a fork branch counts if the fork is the test path); pass only if the workflows rule accepts and runs it. If it does not, validate after the merge, at the main SHA, and the order of the steps changes. **Results (phase 3, 2026-10-27).** (b) pass: with R1 at S1 `3aa7ae9` the run [40560632129](https://github.com/OperationCode/claude-review-test/actions/runs/40560632129) (PR 28, reopened 18:30:32Z) got a token, 3 passes `ok`, gate pass, box clear at 18:32:12Z (exchange window 18:31:01 to 18:31:40Z). The run [40561015054](https://github.com/OperationCode/claude-review-test/actions/runs/40561015054) at the probe `13a1e48` (S1 plus one empty commit, on branch `p3-deny-probe`; TR pinned to it at 18:32:57.9Z) was denied: 3 passes `failed (authentication failed)`, gate fail, box blocked (window 18:33:32 to 18:33:56Z). Console, read by the owner (2026-10-27): the deny run at 18:33:49Z shows 6 rows `Failure`, reason `match_claim_value_mismatch`, rule `claude-review-prod-r1`, service account `claude-review`. Before the merge, the owner reported `Success` for the R1 rows in the windows 18:31:01 to 18:31:40Z and 19:16:09 to 19:16:46Z. The windows from 21:02Z on (M1 deny, M1 accept, PR 37) and all R2 rows were not checked. There is no screenshot, and the token scope and workspace are not separately confirmed. The read-back output of R1 at the placeholder is still not recorded. (c) partial pass: accepted at S2 and M2, token through R2 inferred; no R2 read-backs, and step 5a (a denied exchange at the merge SHA before the second edit) was not run for R2 and M2 (the deny at the merge SHA ran only for R1 at M1). Not done, and still to do: R2 read-backs at the placeholder, S2 and M2, and a denied run at M2 before the R2 edit. S2 `9752ba7` (M1 plus R2 and the test service account; three assertions in `test_evals.py` adjusted, simulation only; unit tests 2563 passed, `-m release` 2 passed) on branch `p3-sim`, PR 10 into `p3-sim-base` (branched from M1), never main. TR accepted a SHA that is only on a non-default branch (S1, and S2 and the probe) and started runs at it. Step 4: TR pinned to S2 at 21:18:49Z, run [40585724589](https://github.com/OperationCode/claude-review-test/actions/runs/40585724589) at S2: 3 `ok`, box clear 21:20:18Z (window 21:19:19 to 21:19:45Z). Step 5: PR 10 squash-merged into `p3-sim-base` at 21:20:59Z gave M2 `1cefe4259bc26197ec4b063e8399ef856efdd4f2`; `git diff S2 M2` is empty. Step 6: runs [40587964524](https://github.com/OperationCode/claude-review-test/actions/runs/40587964524) and [40587977659](https://github.com/OperationCode/claude-review-test/actions/runs/40587977659) at M2, 3 `ok` each (window 21:35:22 to 21:35:54Z). R2 rows were not read in the Console, so the token through R2 is inferred: R1 names M1 and cannot match a run at M2. The reports record no read-back output for R2 at the placeholder, S2 or M2. R1 was not changed during the simulation, and not deleted. +25. **Pilot bypass (blocking).** The docs list the `RepositoryRole` actor type but not the `actor_id` values of the built-in roles. Test: in the scratch repo, an org ruleset with only the workflows rule and a bypass for the Maintain and Admin repository roles in `pull_request` mode. Read it back and record the `actor_id` values. Pass when, on a failed run, a user with Maintain and a user with Admin can merge with the bypass, a user with Write cannot, and GitHub records each bypass. (Phase 3 could show the Maintain and Admin parts only; see the result.) Also record whether a direct push to the default branch is blocked; before a repo joins the pilot, check it for people or bots that push to the default branch directly. **Phase 3 (2026-10-28): PASS for Maintain and Admin; the Write bypass part is NOT RUN.** Account X had Write, then Maintain, then Admin on claude-review-test. The TR bypass list held the Maintain role, the Admin role and org admin, all in `pull_request` mode. Write: a plain merge of the failed PR 40 was refused ("the base branch policy prohibits the merge"). A plain merge fails for any user without a bypass, so this does not test the bypass. The `--admin` attempt as Write was NOT RUN (the automation's classifier blocked it). The only support that Write cannot bypass is the setting: the TR bypass list does not hold the Write role. Maintain: PR 42 (failed run 40624895110, bidi file) merged with `--admin` (merge commit `a5ec202`). Admin: PR 43 (failed run 40628870689) merged with `--admin` (merge commit `bb8c937`). GitHub recorded rule suite result `bypass` for each (repo endpoint `repos/OWNER/REPO/rulesets/rule-suites`, suites 4761914803 and 4762319933), with the rule evaluation still `fail` ("Required workflows 'claude-review' failed"). That pair is the bypass signature. The record names the user, not a role: `actor_id` is the user ID (349107140), and no role ID or role name is stored, so it does not say which bypass entry was used. The org endpoint `orgs/OWNER/rulesets/rule-suites` returns 403 on this plan (Enterprise only), so use the repo endpoint. A direct push to main by Maintain was refused (GH013, "Required workflow 'claude-review' is not satisfied"; suite 4761955192, result `fail`; main stayed at `a5ec202`): the `pull_request` bypass mode covers the PR merge only. The TR bypass list read back through the API holds OrganizationAdmin, RepositoryRole 2 and RepositoryRole 5; report step 7 names Admin as role 5. RepositoryRole 2 is Maintain and 5 is Admin (built-in role IDs). The bypass merges and the direct push were run by the owner by hand, because the automation's classifier blocks them. The TR bypass list is a test-only setting and goes back to empty after phase 3 (the owner has not yet decided). +26. **Placeholder SHA in a federation rule (verify in phase 3; part a 2026-10-27: creation accepted, `job_workflow_ref` read back at M1 and M2, not at the placeholder).** The setup and every release create a rule at the placeholder SHA `0000000000000000000000000000000000000000` and later edit it to a real SHA ("Each release" in "Auth and cost"). V5 and V9 edited a rule from one real SHA to another; no test created a rule with a SHA that matches no commit. Test: (a) at setup, create the production rule at the placeholder and read it back. If the Console rejects it, stop, and resolve the order of rule ID and SHA before the production setup goes on. Result of part a (2026-10-27): the owner created R1 and the Console accepted it; the owner read R1 back at M1 `590a387` and R2 at M2 `1cefe42` on 2026-10-27, before the M3 edit; R1 was then moved to M3. The service account and the workspace were not read back. So part a is done for `job_workflow_ref`, with read-backs at M1 and M2 and not at the placeholder. R1 `claude-review-prod-r1` (`fdrl_01N2ZktTiVKqUvvXwywBFfys`) was created with `job_workflow_ref` at the placeholder SHA. The scratch rule R2 `claude-review-p3-r2` (`fdrl_01AfACzupW3U4Bx361ZUbsyR`; service account `claude-review-p0`, workspace `claude-review-p0-test`) was created the same way and stays at the placeholder until the release simulation of part c. (b) In phase 3, before the acceptance tests, edit the rule to the candidate SHA and read it back. Pass when a run under the scratch test ruleset at the candidate SHA gets a token, and a run at another SHA is denied. (c) Before the first release (V22 b), do steps 1 to 6 of "Each release" with a scratch rule R2 on the test workspace, including the re-edit of R2 to the merge SHA and the second exchange. Do not change R1 during this check. Pass when R2 reads back at the placeholder, then at the candidate SHA, then at the release SHA, and the exchange succeeds at both real SHAs. The scratch ruleset is pinned to a SHA that is on a non-default branch of `.github` at step 4 (a fork branch counts if the fork is the test path); pass only if the workflows rule accepts and runs it. If it does not, validate after the merge, at the main SHA, and the order of the steps changes. **Results (phase 3, 2026-10-27).** (b) pass: with R1 at S1 `3aa7ae9` the run [40560632129](https://github.com/OperationCode/claude-review-test/actions/runs/40560632129) (PR 28, reopened 18:30:32Z) got a token, 3 passes `ok`, gate pass, box clear at 18:32:12Z (exchange window 18:31:01 to 18:31:40Z). The run [40561015054](https://github.com/OperationCode/claude-review-test/actions/runs/40561015054) at the probe `13a1e48` (S1 plus one empty commit, on branch `p3-deny-probe`; TR pinned to it at 18:32:57.9Z) was denied: 3 passes `failed (authentication failed)`, gate fail, box blocked (window 18:33:32 to 18:33:56Z). Console, read by the owner (2026-10-27): the deny run at 18:33:49Z shows 6 rows `Failure`, reason `match_claim_value_mismatch`, rule `claude-review-prod-r1`, service account `claude-review`. Before the merge, the owner reported `Success` for the R1 rows in the windows 18:31:01 to 18:31:40Z and 19:16:09 to 19:16:46Z. The windows from 21:02Z on (M1 deny, M1 accept, PR 37) and all R2 rows were not checked. There is no screenshot, and the token scope and workspace are not separately confirmed. The read-back of R1 and R2 at the placeholder is still not recorded (the read-backs were at M1 and M2). (c) partial pass: accepted at S2 and M2, token through R2 inferred; no R2 read-back except at M2 (see P3-6), and step 5a (a denied exchange at the merge SHA before the second edit) was not run for R2 and M2 (the deny at the merge SHA ran only for R1 at M1). NOT RUN, stays open: a denied run at M2 before the R2 edit to M2. S2 `9752ba7` (M1 plus R2 and the test service account; three assertions in `test_evals.py` adjusted, simulation only; unit tests 2563 passed, `-m release` 2 passed) on branch `p3-sim`, PR 10 into `p3-sim-base` (branched from M1), never main. TR accepted a SHA that is only on a non-default branch (S1, and S2 and the probe) and started runs at it. Step 4: TR pinned to S2 at 21:18:49Z, run [40585724589](https://github.com/OperationCode/claude-review-test/actions/runs/40585724589) at S2: 3 `ok`, box clear 21:20:18Z (window 21:19:19 to 21:19:45Z). Step 5: PR 10 squash-merged into `p3-sim-base` at 21:20:59Z gave M2 `1cefe4259bc26197ec4b063e8399ef856efdd4f2`; `git diff S2 M2` is empty. Step 6: runs [40587964524](https://github.com/OperationCode/claude-review-test/actions/runs/40587964524) and [40587977659](https://github.com/OperationCode/claude-review-test/actions/runs/40587977659) at M2, 3 `ok` each (window 21:35:22 to 21:35:54Z). R2 rows were not read in the Console, so the token through R2 is inferred: R1 names M1 and cannot match a run at M2. R1 was not changed during the simulation, and not deleted. 27. **Eval federation rule (verify in phase 2, before the first eval).** The rule cannot name `eval.yml`, because a dispatch run's `job_workflow_ref` ends in the branch ref (V3). It binds `repository_id`, `event_name` and `environment: eval` instead ("Fixture eval credential"). Passed 2026-10-27 (UTC) with one substitute, one case not run, and the P1 scope not confirmed. The other-repo case ran as `push` in claude-review-test with no environment (N4, 401), so `repository_id` is not tested separately. The "another environment" case was not run; only the no-environment case ran (N1). The Console reason for each deny is recorded in "Fixture eval credential" (`match_claim_absent` for N1 and the no-environment job of the rejected-approval run; `match_claim_value_mismatch` for the no-environment job of the push run, N3 and N4); claim name and P1 scope not shown in the list view, so the P1 scope `workspace:developer` is not confirmed. See "Fixture eval credential". Test on the test workspace `claude-review-p0-test`, and record the Console reason for each deny. Pass when an approved `eval.yml` dispatch run in `.github` gets a token with scope `workspace:developer` for that workspace only, and each of these gets no token: a run whose approval is rejected (the job never starts, and the Console shows no exchange); a dispatch run in `.github` with no environment or another environment; a dispatch run with an `eval` environment in another repo (claude-review-test, with its own `eval` environment; this case is required, P2-2); a run in `.github` with the `eval` environment on another event (for example `push`). Also pass when the workflow unit test (see "Testing") fails on a fixture workflow other than `eval.yml` that uses `eval`, and on a job that uses `eval` with a GitHub write permission. The residual stays: an approver who approves without reading. Verify later (phase 3): -- **NOT YET RUN (pending account X):** the web UI "Re-run all jobs" button on a ruleset-started run. Phase 3 asked for the owner to click it on run 40568865128 and got no click, so the override cases ran with the REST re-run call. Until the button is tested, the documented way to apply an override is the REST re-run call (see "Override"). +- **PASS (2026-10-28), box not observed:** the web UI "Re-run all jobs" button on a ruleset-started run. Phase 3 first asked for the owner to click it on run 40568865128 and got no click, so the override cases ran with the REST re-run call. The owner then clicked it on run 40628870689 (claude-review-test, PR 43): it made attempt 2 of the same run ID, attempt 2 re-read live state (result `pass` after the merge removed the bidi file from the live comparison), and the single summary comment followed the newest attempt. The merge box after the re-run was not observed because PR 43 was already merged. The REST re-run call stays the documented way to apply an override (see "Override"). Answered since the last version: @@ -977,6 +980,7 @@ P2: phase 2 rulings, 2026-10-27 (UTC). Where a P2- line changes an earlier line, - P2-5. Expectation format v2 and the fixture builder ("Testing"). An exact severity failed a "high" case on a `critical` answer, and there was no way to state "no high findings", a citation side, verification, alternative categories or a redacted summary. Now severity means "or higher", and `max_severity`, `file`, `side`, `verified`, category lists and `summary_excludes` exist. A hand-written bundle made "hard block regardless of Claude" true by construction, so every fixture bundle is built by the real `collect()` and a unit test checks it against a rebuild. - P3-1. Phase 3 rulings (owner and coordinator, 2026-10-27). Q1: a new production service account `claude-review` (`svac_01PiMgfb6qJgNpLZUV8hprCC`), separate from the test account, so test teardown cannot break production. Q2: the coordinator's approval of `eval` runs extends through phase 4; "prevent self-review" stays off until phase 5. Q3: test work in claude-review-test and `p3-*` branches in `.github` is approved; the non-member fork author (account X) is pending, and such an account becomes an outside collaborator, not an org member, when given repo access. Q4: `test.yml` runs `pytest tests/unit -m release` on every PR, in the `unit-tests` job. Q5: no workflow change on main for V22 a; the evidence is `job.workflow_sha` against the pin. Q6: one eval, at the candidate commit only. Q7 (design change): no Console edit lowers the test workspace spend limit; the exhausted-budget outage case is covered by V9 (phase 0) and the unit tests; the invalid federation rule case still runs live (denied exchange, V26 b). V26 a: R1 was created at the placeholder SHA and the Console accepted it (2026-10-27); R2 was created the same way. - P3-2. Over-long strings (design change). Release eval [run 40517202830](https://github.com/OperationCode/.github/actions/runs/40517202830) at the candidate S1 (ccbae51) matched 12 of 13 cases: in `obfuscated-test-payload` the supply-chain pass failed with "response does not match the findings schema: 'findings'/1/'quote': [maxLength]". Phase 2 run 40445885087 matched the same case with the same prompts, so this is model variance. Cause: structured outputs cannot enforce `maxLength` (P0-12), so the API accepts a string of any length, and one long string made the whole pass `failed` and the gate fail on a real PR. Fix (coordinator ruling, 2026-10-27): before the full schema check, analyze cuts every string over its schema `maxLength` to that limit and keeps the finding. This covers `summary` (1500), `file` (400), `quote` (300), `title` (120) and `explanation` (1200), and any limit added to the schema later; a unit test checks that the cut reaches every `maxLength`. A file path is an identifier, not prose, so its cut differs: the first 399 characters and a final "/" (fix round 1, review r-p3-cut Important 1). A plain prefix of a 401-character path can be the path of another changed file, which could then verify the finding and place its inline comment on the wrong file. No file path ends in "/", and the file lookup never matches one, so a cut path names no file. The finding stays and still counts toward the gate. The system prompt also tells the model to copy the part of a longer line that shows the problem. Why it does not lower risk: a finding is never dropped, and its category, severity, confidence, side and line never change. Blocking never depends on the text, the quote or verification ("Citation check"). Whitespace normalization keeps a prefix of a quote inside the normalized text wherever the whole quote is, so a quote that verified still verifies. A cut quote that is not at that place, a cut quote that is only whitespace or holds a redacted credential, or a cut file path, stays unverified and still blocks. A cut can make two findings that differed past the cut share file, title and quote (review r-p3-cut Important 2). A redaction before the cut can do the same far below the limit: two titles that differ only in a credential both become `Token [REDACTED:jwt]` (fix round 2, review rr-p3-cut-fix1 Important A). Two findings below every limit can also share file, side, title and quote and differ in explanation or severity. So each inline comment carries two keys ("Publishing (code)", fix round 3, review rr-p3-cut-fix2): the D14 key (file, side, title and quote) and a full key of every field that the reader sees, both of the shown, redacted text. In one run, findings are deduped by full key, so each distinct finding gets its own inline comment. Across runs, a finding whose D14 key is already on the PR is not posted again; when its full key matches none of the earlier comments with that D14 key, the summary shows it in full. So none hides the explanation of another, and a re-run or a push does not post every finding again (fix round 2 keyed only on every shown field, which re-posted almost every finding on each run, because the explanation and the line change). Fix round 4 (review r-p3-cut-final, coordinator rulings 2026-10-27): the marker alone is not proof that a comment shows a finding, because any Actions workflow in the repo writes as `github-actions[bot]` and the keys are public hashes. The D14 skip needs the inline layout, path, side and title, and "see the earlier comment" needs the exact body at the same line; a forged comment can at most move a finding to its full summary block. A cut file path makes a finding unverified, and an unverified high or critical finding now keeps its never-cut rank in the summary, so the summary shows every such explanation or the run fails closed. The residual of one summary comment (a low or medium explanation can be cut on a changed-key re-run at the largest sizes; capacity can fail a run closed) is stated in "Publishing (code)"; continuation comments are planned for phase 4. Findings that are equal in every shown field after the cut show one inline comment; the publisher cannot know the text past the cut, and that comment holds all the text of each. A credential-like string in an over-long string is redacted before the cut, so the cut cannot leave part of a secret that the publisher would then not recognize. The publisher escapes every character of a cut string, so a cut inside Markdown, HTML or an escape still shows as text (unit test). A cut title, explanation or summary has no truncation mark, because the publisher shows one only for its own cuts. A new candidate with this change needs its own eval (P0-15). -- P3-3. Phase 3 results and spec corrections (2026-10-27). Four spec claims did not survive the live tests, and each is now corrected in place. (1) The merge box counts only runs at the pinned SHA, not the newest run on the head SHA; a pin move back revives older passes at that SHA (PR 28 on 2026-10-27 at 18:34:53Z; PRs 28 and 37 at 21:38:44Z). A forward move fails closed (the box blocks at once). The rollback is a residual, covered by the rule to start a new run on each open PR after any pin move; it joins the residuals in rollout step 7. (2) Repo Actions settings do not affect the required run (V12 pass); the role table no longer lists them as an admin power against the gate. (3) A normal cancel before the gate job starts does not stop the `always()` gate job (run 40565538379), which then writes a `pass` summary next to a `cancelled` run; a normal cancel while the gate job runs was not tested; the run conclusion still blocks the box, and a force-cancel stopped the gate job after the summary (run 40565910355). (4) The artifacts of earlier attempts are gone after a re-run; the gate reads only its own attempt, so it still fails closed, and the residual is that earlier pass records cannot be read back. Release procedure: two Console edits per release (candidate SHA, then merge SHA), a tree check of the merge SHA, and a denied exchange at the merge SHA before the second edit. NOT YET RUN, pending account X: V8, V23, V6 f as a Write user, V25 and the web UI "Re-run all jobs" case. The exhausted-budget outage case stays covered by V9 and the unit tests (Q7). +- P3-3. Phase 3 results and spec corrections (2026-10-27). Four spec claims did not survive the live tests, and each is now corrected in place. (1) The merge box counts only runs at the pinned SHA, not the newest run on the head SHA; a pin move back revives older passes at that SHA (PR 28 on 2026-10-27 at 18:34:53Z; PRs 28 and 37 at 21:38:44Z). A forward move fails closed (the box blocks at once). The rollback is a residual, covered by the rule to start a new run on each open PR after any pin move; it joins the residuals in rollout step 7. (2) Repo Actions settings do not affect the required run (V12 pass); the role table no longer lists them as an admin power against the gate. (3) A normal cancel before the gate job starts does not stop the `always()` gate job (run 40565538379), which then writes a `pass` summary next to a `cancelled` run; a normal cancel while the gate job runs was not tested; the run conclusion still blocks the box, and a force-cancel stopped the gate job after the summary (run 40565910355). (4) The artifacts of earlier attempts are gone after a re-run; the gate reads only its own attempt, so it still fails closed, and the residual is that earlier pass records cannot be read back. Release procedure: two Console edits per release (candidate SHA, then merge SHA), a tree check of the merge SHA, and a denied exchange at the merge SHA before the second edit. Account X results came later (P3-6); the web UI "Re-run all jobs" case stays NOT RUN. The exhausted-budget outage case stays covered by V9 and the unit tests (Q7). - P3-4. Model per pass, Sonnet 5.5 (2026-10-27). The owner proposed `claude-sonnet-5-5` for general use. The model is now a per-pass setting, and every pass uses Sonnet 5.5. The switch needed a full match of the 13 cases (P2-4 bar); Opus 5.5 would stay on any pass where Sonnet missed. Result: two eval runs at 302882f, [40589761270](https://github.com/OperationCode/.github/actions/runs/40589761270) and [40590079545](https://github.com/OperationCode/.github/actions/runs/40590079545), each matched 13 of 13 cases with no failed pass (Opus reference [run 40528717338](https://github.com/OperationCode/.github/actions/runs/40528717338), also 13 of 13). Against Opus, total tokens were 0.97x and 0.98x, output tokens 0.71x and 0.76x, and eval wall time 84 s and 82 s against 138 s. No currency figures were recorded. The switch takes effect at the next release: the production rule pins a SHA, so merging does not change production. - P3-5. Summary comment redesigned for fast human judgment (owner and coordinator, 2026-10-27). The comment now opens with a result heading and one colored callout, then a findings table, a checks strip and a provenance line, with the waiver and the full record collapsed ("Summary comment" under "Publishing (code)"). The owner chose the Fable design with these rulings: every fail uses the red `[!CAUTION]` callout and the red heading, waivable or not (a waivable fail on a fake check was misread as a mild warning under `[!WARNING]`, so `[!WARNING]` is not used for a fail); the waiver line stays in the collapsed Waiver block; model summaries and token counts stay in the collapsed record; duplicate findings merge only at the same file, side, line and gate effect; a medium finding in a blocking category reads "Does not block"; the visible line shows the 7-character head SHA and the record shows the full SHAs; emoji are allowed, each next to its word. Machine markers, the override line format, the size limit and the overflow path are unchanged. +- P3-6. Account X results and Sonnet release (owner and coordinator, 2026-10-28). Account X is `irvingpop2`, a separate GitHub account outside the org that the owner made as a test identity; it became an outside collaborator when it got repo access. Steps ran from an isolated container. V8 b pass (PR 38): the central run started on a first-time-contributor fork PR with no approval click, and marking the draft ready started no new run. V8 a pass (PR 39): from a first-time fork, GitHub holds the fake `pull_request` job (`action_required`), so it cannot start before the central run; after a maintainer approved only that run it passed, and the box stayed BLOCKED and followed the central run; the fake-first case stays covered by PR 36. V23 pass (PR 40): a same-path `claude-review.yml` in the fork PR did not replace the central run, which failed on the bidi hard block; the fork run stayed `action_required` and never ran, so the running case is covered by PR 35. V6 f as a Write user: CONFIRMED residual (PR 41); deleting the newer run reopened the box on an older pass, and it stays in the residuals the owner must accept before org-wide enforcement. A Write user's plain merge of a failed PR was refused; this does not test the bypass, and the `--admin` attempt as Write was not run. V25 pass for Maintain and Admin, Write bypass NOT RUN: with the TR bypass list, a Maintain user (PR 42) and an Admin user (PR 43) merged a failed PR; GitHub records rule suite result `bypass` with the user ID as actor and no role ID, and the org rule-suites endpoint returns 403 on this plan, so use the repo endpoint; a direct push to main by Maintain is refused (GH013). The TR bypass list is a test-only setting and goes back to empty after phase 3 (owner's choice pending). Process: the owner ran the bypass merges and the direct push by hand, because the automation's safety classifier blocks bypass merges; the evidence comes from the GitHub API. Sonnet release: release evals 13 of 13 at M3 ([run 40592861233](https://github.com/OperationCode/.github/actions/runs/40592861233)); the live run [40593729924](https://github.com/OperationCode/claude-review-test/actions/runs/40593729924) passed with `claude-sonnet-5-5`. Console read-back: the owner read back R1 at 590a387 and R2 at 1cefe42 on 2026-10-27, before the M3 edit; R1 was then moved to M3; the service account and the workspace were not read back. V26 a is done for `job_workflow_ref`. NOT RUN, stays open: the Write `--admin` bypass attempt and V26 c deny at M2 before the R2 edit. Web UI "Re-run all jobs": pass (run 40628870689, PR 43, 2026-10-28): the click made attempt 2 of the same run ID, which re-read live state and updated the one summary comment; the merge box was not observed because the PR was already merged.