Phase 4: pilot fixes and records (combines #15-#19)

.github/workflows/eval.yml

@@ -43,6 +43,7 @@ jobs:
             files.pythonhosted.org:443
             api.anthropic.com:443
             *.actions.githubusercontent.com:443
+            *.blob.core.windows.net:443
       - name: Check out the dispatched branch
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1, released 2026-07-20
         with:
@@ -64,3 +65,18 @@ jobs:
           uv run --no-sync --project review python -m oc_review.evals
           --fixtures-dir review/tests/fixtures
           --out-dir "$RUNNER_TEMP/eval"
+      # results.jsonl and each case's summary.md only. The per-case data/ dirs
+      # (bundle and pass records) stay out. No file holds a credential: the
+      # Anthropic token lives in memory only (spec "Fixture eval credential").
+      - name: Upload the eval results
+        if: always()
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2, released 2026-10-07
+        with:
+          name: eval-results-a${{ github.run_attempt }}
+          path: |
+            ${{ runner.temp }}/eval/results.jsonl
+            ${{ runner.temp }}/eval/*/summary.md
+          # warn, not error: if an earlier step failed before any result exists,
+          # a second red step would bury the real cause.
+          if-no-files-found: warn
+          retention-days: 14

docs/build-plan.md

@@ -185,19 +185,21 @@ Exit criteria:
 
 ## Phase 4: pilot
 
-- [ ] Owner: before a repo joins the pilot, give it its own approval protection on the default branch: a repo ruleset or branch protection with at least 1 approval, code owner review and dismiss stale approvals on push. `back-end` has none today.
-- [ ] Owner: before a repo joins the pilot, check it for people or bots that push to the default branch directly (V25).
+- [x] Owner: record the pilot scope. The plan named only `back-end`; the owner added `front-end` (135777271), `operationcode-pybot` (146210805, default branch `master`) and `operationcode_infra` (107068150), with `back-end` (182584277). Four repos (spec P4-1).
+- [x] Owner: before a repo joins the pilot, give it its own approval protection on the default branch: a repo ruleset or branch protection with at least 1 approval, code owner review and dismiss stale approvals on push. Done 2026-10-28: "Pilot approval protection" on `back-end` (26057791), `operationcode-pybot` (26057793) and `operationcode_infra` (26057794), with the Maintain, Admin and org admin bypass because pybot and infra have one active committer; `front-end`: the existing "Protect main" (25634150), only `require_code_owner_review` changed to true (spec P4-3).
+- [x] Owner: before a repo joins the pilot, check it for people or bots that push to the default branch directly (V25). Done: none in 12 months on `back-end`, `operationcode-pybot` and `operationcode_infra`; `front-end` had a force push on 2026-10-28 that rewrote three merged PRs (cleanup of a malicious dependency payload), not new unreviewed code (spec P4-4).
 - [ ] Owner: accept that during the pilot a maintainer or repo admin can merge past any failed run, also on their own PR (spec "Pilot and enforcement").
-- [ ] Owner: confirm that the federation rule named in the release code has `job_workflow_ref` at the release SHA.
-- [ ] Owner: create the pilot ruleset. Only the rule "Require workflows to pass before merging" with the central workflow `repository_id`, path and release SHA. Targets: the pilot repos by name (first `back-end`), default branch. Bypass in `pull_request` mode: the Maintain and Admin repository roles, and org admins.
-- [ ] Owner: read the pilot ruleset back through the [org rules API](https://docs.github.com/en/rest/orgs/rules#get-an-organization-repository-ruleset). Check `repository_id`, path, `sha`, no `ref`, and the role `actor_id` values from V25. Then enable it.
-- [ ] Owner: require 2FA for all org members.
-- [ ] Owner: CODEOWNERS in each pilot repo for `.github/`, manifests, lockfiles, `pyproject.toml`, `Dockerfile`.
-- [ ] Start a run on each open PR in the pilot repos (push, or close and reopen).
+- [x] Owner: confirm that the federation rule named in the release code has `job_workflow_ref` at the release SHA. Read back by the owner at M4 `9e2795b` (2026-10-29, spec P4-5).
+- [x] Owner: every pilot repo sends the immutable OIDC subject (spec "Immutable subject", P4-6). Found 2026-10-29: repos made before the GitHub default sent `repo:OperationCode/<repo>:...`, the rule rejected them (`match_subject_prefix`), and the first pilot runs failed closed as designed. Fix per repo: `PUT repos/OperationCode/<repo>/actions/oidc/customization/sub` with `use_default` true and `use_immutable_subject` true. Before it, check the other `id-token` users in the repo and their relying-party trust: `back-end` and `operationcode-pybot` use a cloud OIDC trust policy that gained the immutable subjects (owner; old values kept). After it, all four test PRs passed (all four jobs `success`).
+- [x] Owner: create the pilot ruleset. Only the rule "Require workflows to pass before merging" with the central workflow `repository_id`, path and release SHA. Targets: the pilot repos by name (`back-end`, `front-end`, `operationcode-pybot`, `operationcode_infra`), default branch. Bypass in `pull_request` mode: the Maintain and Admin repository roles, and org admins. Created disabled as `claude-review-pilot` (26132750): four repo ids, `~DEFAULT_BRANCH`, `repository_id` 1432342572, M4 `9e2795b`, bypass `RepositoryRole` 2 and 5 plus `OrganizationAdmin` (spec P4-2).
+- [x] Owner: read the pilot ruleset back through the [org rules API](https://docs.github.com/en/rest/orgs/rules#get-an-organization-repository-ruleset). Check `repository_id`, path, `sha`, no `ref`, and the role `actor_id` values from V25. Then enable it. Read back (`do_not_enforce_on_create` false, no `ref`), then enabled 2026-10-29 (created and enabled the same morning). Rollback: `PUT` with `enforcement` set to `disabled`.
+- [x] Owner: require 2FA for all org members. Done in the web UI; all 7 members already had 2FA (spec P4-5).
+- [x] Owner: CODEOWNERS in each pilot repo for `.github/`, manifests, lockfiles, `pyproject.toml`, `Dockerfile`. Added on each default branch (`back-end`: @irvingpop @kylemh; `front-end`: @kylemh @irvingpop; `operationcode-pybot` and `operationcode_infra`: @irvingpop; spec P4-3).
+- [ ] Right after enabling the pilot ruleset, list the open PRs in the pilot repos again (do not reuse a list from earlier recon) and start a run on each (close and reopen, or a new push). A required workflow starts only on a new PR event, so a PR opened between the recon and the enable gets no review run: front-end PR 2027 (opened 2026-10-29T08:06Z) was missed this way.
 - [ ] Each week, record false positives, overrides, bypasses, failed runs by reason, egress blocks and cost per run. Keep cost numbers (any currency amount) in the owner's private notes and the Anthropic Console, never in this public repo. Public text may give ratios and timing only.
 - [ ] Fix false positives in prompts or code. Each fix goes through phase 2 again, then ships as in "Each release" below.
-- [ ] Keep the fixture eval results as an artifact: `eval.yml` uploads none today, so a release eval's per-case results exist only in the job log (phase 2 run 40445885087 and phase 3 run 40528717338 were read from logs). Ships as in "Each release".
-- [ ] Plan publication before writing: bounded continuation comments, pointed to from the sticky summary, hold every summary-only finding (changed keys, unverified, outside the diff) when one comment cannot. Then no explanation is cut from the PR comments and summary capacity no longer fails a run whose computed result is `pass` (spec "Publishing (code)", size limit residual; review r-p3-cut-final Important 1). Keep the fail-closed check until this path is complete. Ships as in "Each release".
+- [x] Keep the fixture eval results as an artifact: `eval.yml` uploads `results.jsonl` and each case's `summary.md` as `eval-results-a<N>` (14 days, `if: always()`). Before this, a release eval's per-case results existed only in the job log (phase 2 run 40445885087 and phase 3 run 40528717338 were read from logs). Ships as in "Each release".
+- [ ] Plan publication before writing: bounded continuation comments, pointed to from the sticky summary, hold every summary-only finding (changed keys, unverified, outside the diff) when one comment cannot. Then no explanation is cut from the PR comments and summary capacity no longer fails a run whose computed result is `pass` (spec "Publishing (code)", size limit residual; review r-p3-cut-final Important 1). Keep the fail-closed check until this path is complete. Ships as in "Each release". Plan (planned, not built): [docs/continuation-comments-plan.md](continuation-comments-plan.md).
 
 Depends on: phase 3.
 
@@ -216,6 +218,7 @@ Exit criteria:
   - [ ] targets: never `.github`, and none of the repos still in the pilot ruleset
 - [ ] Owner: read the org ruleset back and check the workflow entry (`repository_id`, path, `sha`, no `ref`).
 - [ ] Build the supported repo inventory (spec "Supported repos") as a list in this repo.
+- [ ] Owner: every repo to be added to the org ruleset first sends the immutable OIDC subject (spec "Immutable subject"). Check `GET repos/OperationCode/<repo>/actions/oidc/customization/sub`, check the other `id-token` users in the repo and their relying-party trust, set it per repo, and run a test PR (all jobs `success`) before the repo is a target.
 - [ ] Owner: CODEOWNERS in each supported repo.
 - [ ] Owner: turn "prevent self-review" back on for the `eval` environment (it is off during phase 2), and decide whether admin bypass stays on.
 - [ ] Owner: Dependabot alerts with npm malware detection on active repos.
@@ -232,7 +235,7 @@ Exit criteria:
 
 ## Phase 6: org-wide enforcement
 
-- [ ] Owner: enable the org ruleset with the supported repos that are not in the pilot. Add them in small groups.
+- [ ] Owner: enable the org ruleset with the supported repos that are not in the pilot. Add them in small groups. Each repo in a group already sends the immutable OIDC subject (phase 5); without it the run fails closed.
 - [ ] Owner: move each pilot repo: remove it from the pilot ruleset targets, then add it to the org ruleset targets at once. Never let both rulesets require the workflow on one repo.
 - [ ] After each group or move, start a run on each open PR, and check that each shows the required workflow passing or failing on its own result.
 - [ ] Owner: delete the pilot ruleset when no repo is left in it.