Phase 3: outside-account results (V8, V23, V6 f, V25) and closing records

docs/build-plan.md

@@ -55,7 +55,7 @@ Tasks:
 Owner:
 
 - [x] Owner: create a scratch source repo in the org for the V tests, separate from claude-review-test ([claude-review-p0-source](https://github.com/OperationCode/claude-review-p0-source)).
-- [ ] Owner: Anthropic Console: a dedicated workspace with a hard monthly spend limit and a service account. Create the production federation rule as in spec "Auth and cost": pattern mode, subject prefix `repo:OperationCode@10479364/*`, scope `workspace:developer`, the review workspace only, and the placeholder SHA `0000000000000000000000000000000000000000` in `job_workflow_ref`. Read it back; if the Console rejects the placeholder, stop and resolve the order of rule ID and SHA first (spec V26 a). Its ID goes into the central code; phase 3 sets the candidate SHA. No API key. Done 2026-10-27: workspace `claude-review`, service account `claude-review`, rule R1 `claude-review-prod-r1` created; the Console accepted the placeholder SHA (V26 a partial). The read-back output is not recorded, so this item stays open; the later V26 b exchange does not replace it.
+- [ ] Owner: Anthropic Console: a dedicated workspace with a hard monthly spend limit and a service account. Create the production federation rule as in spec "Auth and cost": pattern mode, subject prefix `repo:OperationCode@10479364/*`, scope `workspace:developer`, the review workspace only, and the placeholder SHA `0000000000000000000000000000000000000000` in `job_workflow_ref`. Read it back; if the Console rejects the placeholder, stop and resolve the order of rule ID and SHA first (spec V26 a). Its ID goes into the central code; phase 3 sets the candidate SHA. No API key. Done 2026-10-27: workspace `claude-review`, service account `claude-review`, rule R1 `claude-review-prod-r1` created; the Console accepted the placeholder SHA (V26 a partial). The owner read R1 back at M1 and R2 at M2 on 2026-10-27 (spec P3-6). Still open: the service account and the workspace were not read back.
 - [ ] Owner: the test workspace (reused for evals, spec P2-1), and a federation rule for `workflow_dispatch` runs in `OperationCode/.github` whose job uses the environment `eval`: pattern mode, the same subject prefix, exact claims `repository_owner_id`, `repository_id`, `event_name` and `environment`, scope `workspace:developer`, that workspace only (spec "Auth and cost"). Its IDs are in `evals.py`. The rule cannot name `eval.yml`: any `.github` workflow that uses `eval` can match, so the owner's approval and the workflow unit test (1c) hold it to `eval.yml`. Test it as in spec V27 before phase 2.
 - [ ] Owner: on `.github`, create the GitHub environment `eval`: the owner is a required reviewer for every run, and any branch may deploy to it. "Prevent self-review" is off temporarily (owner, 2026-10-26), so that the coordinator, which runs with the owner's token, can approve eval runs it started from reviewed branches; the owner turns it on again in phase 5. Admin bypass stays on. During phase 2 the coordinator's approval stands in for the owner's under the owner's delegation. Before approving a run, check the run that waits for approval: the workflow file and commit it runs, every job in it that uses `eval`, and their effective permissions (no GitHub write). Also check that the PR does not change `eval.yml`, any other workflow file, the eval code, the prompts, the fixtures or the federation setup in a harmful way, and that `test.yml` passed. The approval is the control; the residual risk is an approver who approves without reading. Exception to the no-execute rule (K-4): evals of PRs to this repo may run that PR's code, only after an owner approves the `eval` environment run. Needed before phase 2.
 - [x] Owner: a test workspace with a very small spend limit for V9.
@@ -151,29 +151,29 @@ Exit criteria:
 Run the spec's workflow acceptance tests under a test org ruleset that requires the central workflow from `OperationCode/.github` at the candidate release SHA and targets only claude-review-test. Only one ruleset requires the workflow there at a time.
 
 - [x] Production IDs in `analyze.py` and the release tests in CI (the candidate S1). R1 `claude-review-prod-r1` (`fdrl_01N2ZktTiVKqUvvXwywBFfys`) and the production service account are set; `test.yml` runs `pytest tests/unit -m release`.
-- [ ] V26 a (partial): the owner created R1 at the placeholder SHA and the Console accepted it (2026-10-27); R2 `claude-review-p3-r2` was created the same way, inert until the release simulation. No read-back output is recorded for R1 or R2. Remaining: read R1 and R2 back.
+- [x] V26 a: the owner created R1 at the placeholder SHA and the Console accepted it (2026-10-27); R2 `claude-review-p3-r2` was created the same way. The owner read back R1 at M1 `590a387` and R2 at M2 `1cefe42` before the M3 edit (spec P3-6). The service account and the workspace were not read back.
 - [x] P3-2: an over-long string no longer fails a pass. Release eval run 40517202830 at S1 matched 12 of 13: one supply-chain quote went over 300 characters, which the API cannot enforce. Analyze now cuts every string over its schema `maxLength` (summary, file, quote, title, explanation) to that limit and keeps the finding (spec P3-2). Fix round 1: a cut file path ends in "/", so it names no changed file; when the file, title or quote is at its limit, the inline comment key covers every shown field, so a cut never hides a distinct explanation. Fix round 2: a redaction can make two titles equal far below the limit, so the key of every finding covers every shown field, at any length. Fix round 3: each inline comment carries the D14 key and the full key, both of the redacted shown text; one run dedupes by full key, a later run skips a D14 key already on the PR and shows the finding in full in the summary when its full key is new. Fix round 4: an earlier comment counts only when its text checks out (the D14 skip needs the inline layout, path, side and title; "see the earlier comment" needs the exact body at the same line), and an unverified high or critical finding is never cut from the summary. A candidate with this change needs its own eval.
 - [x] V26 b, before the acceptance tests: after the owner sets the production rule to the candidate SHA, a run at that SHA gets a token and a run at another SHA is denied. Run 40560632129 at S1 got a token (2026-10-27, 18:31Z); run 40561015054 at the probe SHA was denied, and the Console shows 6 rows `Failure` `match_claim_value_mismatch` on R1 at 18:33:49Z (owner).
 - [ ] V26 c, before V22 b: release steps 1 to 6 with a scratch rule R2 on the test workspace, R1 unchanged. Include the second exchange at the release SHA, and check that the workflows rule accepts a scratch ruleset pinned to a SHA that is only on a non-default branch of `.github`. S2 `9752ba7` (PR 10 into `p3-sim-base`) and M2 `1cefe42` both got a token (runs 40585724589 and 40587964524); the rule accepted SHAs that are only on non-default branches. Partial: accepted at S2 and M2 (token inferred through R2); R2 Console rows were not read, no R2 read-back output is recorded, and step 5a was not run for R2/M2. Remaining: R2 read-backs at the placeholder, S2 and M2, and a denied exchange at M2 before the R2 edit.
 - [x] Release eval at S1 (`3aa7ae9`): run 40528717338, 13 of 13 matched.
 - [x] M1 tree check: M1 `590a387` and S1 have the same tree, and the R1 exchange works at M1 (run 40585413003) after the denied run at M1 (run 40583290922).
-- [ ] Release simulation (R2, S2, M2): steps 1 to 6 of "Each release", never merged to main. Partial: steps ran and S2 and M2 got tokens; no R2 read-backs and no step 5a deny for R2/M2. Remaining: those two parts.
-- [ ] A fork PR from an account outside the org, including a draft that later becomes ready. The review starts with no approval click (V8 b). NOT YET RUN, pending account X.
-- [ ] V8 a: a fork PR whose own workflow has a job named `oc-claude-review` that starts before the required run. NOT YET RUN, pending account X. A same-repo version by an org admin passed (PR 36, run 40572853203 started first; the box stayed blocked).
+- [ ] Release simulation (R2, S2, M2): steps 1 to 6 of "Each release", never merged to main. Partial: steps ran and S2 and M2 got tokens; the R2 read-back at M2 is done. NOT RUN, stays open: the step 5a deny at M2 before the R2 edit.
+- [x] A fork PR from an account outside the org, including a draft that later becomes ready. The review starts with no approval click (V8 b). Pass 2026-10-28: PR 38, run 40621436811 (spec P3-6).
+- [x] V8 a: a fork PR whose own workflow has a job named `oc-claude-review`. Pass 2026-10-28 (PR 39): from a first-time fork GitHub holds the fake run (`action_required`), so it cannot start first; after approval it passed and the box followed the central run. The fake-first case stays covered by the same-repo evidence (PR 36, run 40572853203 started first; the box stayed blocked).
 - [x] V11: retarget from a non-default branch. The retarget starts no run and the box stays blocked until a reopen run passes (PR 29, run 40561903948); an old pass on the same head counts (PR 30).
 - [x] V12: repo-level Actions settings. With Actions disabled, and with a restrictive allowed-actions list, the required run still ran (runs 40562372022 and 40562688524). Settings restored.
 - [x] V22 a and b: pin behind main, then a pin move with PRs open. `job.workflow_sha` equaled the pin in 33 of 33 attempts. After a forward pin move the old passes do not count; after a move back they count again (spec "Pinned SHA only"; PRs 28 and 37, 21:32 to 21:40Z).
-- [ ] V23: a same-path workflow file in the target repo. NOT YET RUN, pending account X. A same-repo version by an org admin passed (PR 35, run 40572849014).
-- [ ] V25 (NOT YET RUN, pending account X): the pilot bypass. Maintain and Admin can merge past a failed run, Write cannot, and GitHub records each bypass. Record the role `actor_id` values, and whether a direct push to the default branch is blocked.
+- [x] V23: a same-path workflow file in the target repo. Pass 2026-10-28 (PR 40): the central run failed on the bidi hard block and was not replaced; the fork run stayed `action_required` and never ran (the running case is covered by PR 35). Same-repo evidence: PR 35, run 40572849014.
+- [x] V25: the pilot bypass. Pass 2026-10-28 for Maintain (PR 42) and Admin (PR 43): each merged a failed PR with the bypass. Write bypass NOT RUN (a plain merge was refused, which does not test the bypass; Write is not on the bypass list). GitHub records `bypass` with the user ID as actor (no role ID), and a direct push to main by Maintain is refused (GH013). RepositoryRole 2 is Maintain and 5 is Admin (built-in role IDs). Residual: the Write `--admin` attempt. The TR bypass list is test-only and goes back to empty after phase 3 (owner's choice pending).
 - [x] The rest of the acceptance list in the spec's "Testing" section that needs no account X, except the four unit-test-only cases listed in the next item (approver removed, edited override, artifact removed, gate cancel after inline comments), including the outage process (spec "Phase 3 results"). Design change (spec P3-1): the invalid federation rule case ran live with two denied exchanges (runs 40561015054 and 40583290922); the exhausted budget case is covered by V9 (phase 0) and the unit tests, with no Console limit edit (ruling Q7).
-- [ ] The acceptance cases that need account X: V6 f as a Write user, and the web UI "Re-run all jobs" button on a ruleset-started run (NOT YET RUN; the override cases ran with the REST re-run). Also not run, covered by unit tests: gate cancelled after inline comments with a refused summary write, approver role removed, edited override, artifact upload removed.
-- [x] Sonnet 5.5 on every pass (spec P3-4). Eval at 302882f: runs 40589761270 and 40590079545 each matched 13 of 13 with no failed pass; the Opus reference run 40528717338 also matched 13 of 13. Total tokens 0.97x and 0.98x of Opus, output tokens 0.71x and 0.76x, eval wall time 84 s and 82 s against 138 s. The switch takes effect at the next release: the production rule pins a SHA, so merging does not change production.
+- [ ] Acceptance cases that needed account X: V6 f as a Write user is done (residual CONFIRMED, PR 41). The web UI "Re-run all jobs" button on a ruleset-started run passed (run 40628870689, PR 43: new attempt on the same run ID, live state re-read, summary comment updated; merge box not observed because the PR was merged); the override cases ran with the REST re-run. Also not run, covered by unit tests: gate cancelled after inline comments with a refused summary write, approver role removed, edited override, artifact upload removed.
+- [x] Sonnet 5.5 on every pass (spec P3-4). Release evals 13 of 13 at M3 (run 40592861233); live run 40593729924 passed with `claude-sonnet-5-5` (spec P3-6). Eval at 302882f: runs 40589761270 and 40590079545 each matched 13 of 13 with no failed pass; the Opus reference run 40528717338 also matched 13 of 13. Total tokens 0.97x and 0.98x of Opus, output tokens 0.71x and 0.76x, eval wall time 84 s and 82 s against 138 s. The switch takes effect at the next release: the production rule pins a SHA, so merging does not change production.
 
 Owner:
 
 - [x] Owner: set `job_workflow_ref` of the production federation rule to the candidate release SHA. Done 2026-10-27 (S1, then M1 `590a387`; the run at M1 got a token; no read-back recorded). The candidate's code must name that rule ID (spec "Auth and cost", "Each release").
 - [ ] Owner: an account outside the org opens the fork PRs.
-- [ ] Owner: test users with Write, Maintain and Admin on claude-review-test for V25.
+- [x] Owner: test users with Write, Maintain and Admin on claude-review-test for V25. Done with account X `irvingpop2` (2026-10-28). Remove its access and empty the TR bypass list after phase 3 (owner's choice pending).
 
 Depends on: phases 1 and 2; V8, V11, V12, V22, V23, V25, V26.
 
@@ -206,7 +206,7 @@ Exit criteria:
 
 ## Phase 5: before org-wide enforcement
 
-- [ ] Owner: accept in writing the residuals in the spec's "Which run counts": an earlier pass on the same SHA, and run deletion if V6 f did not pass.
+- [ ] Owner: accept in writing the residuals in the spec's "Which run counts": an earlier pass on the same SHA, and run deletion (V6 f confirmed it as a Write user).
 - [ ] Owner: fix the existing disabled org ruleset "All checks must pass" before it is enabled:
   - [ ] dismiss stale approvals on push: on
   - [ ] `OrganizationAdmin` bypass mode: `pull_request`, not `always`