From c1f8de17e51364bf7c4b2087c8401c7c1d5ad8da Mon Sep 17 00:00:00 2001 From: Irving Popovetsky Date: Thu, 29 Oct 2026 08:45:42 -0700 Subject: [PATCH 1/2] Record pilot start (four repos) and the immutable OIDC subject prerequisite (P4-1 to P4-6) Co-Authored-By: Claude Sonnet 5.5 --- docs/build-plan.md | 19 +++++++++++-------- docs/claude-review-design.md | 15 +++++++++++++-- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/docs/build-plan.md b/docs/build-plan.md index 399c62b..cd29838 100644 --- a/docs/build-plan.md +++ b/docs/build-plan.md @@ -185,14 +185,16 @@ Exit criteria: ## Phase 4: pilot -- [ ] Owner: before a repo joins the pilot, give it its own approval protection on the default branch: a repo ruleset or branch protection with at least 1 approval, code owner review and dismiss stale approvals on push. `back-end` has none today. -- [ ] Owner: before a repo joins the pilot, check it for people or bots that push to the default branch directly (V25). +- [x] Owner: record the pilot scope. The plan named only `back-end`; the owner added `front-end` (135777271), `operationcode-pybot` (146210805, default branch `master`) and `operationcode_infra` (107068150), with `back-end` (182584277). Four repos (spec P4-1). +- [x] Owner: before a repo joins the pilot, give it its own approval protection on the default branch: a repo ruleset or branch protection with at least 1 approval, code owner review and dismiss stale approvals on push. Done 2026-10-28: "Pilot approval protection" on `back-end` (26057791), `operationcode-pybot` (26057793) and `operationcode_infra` (26057794), with the Maintain, Admin and org admin bypass because pybot and infra have one active committer; `front-end`: the existing "Protect main" (25634150), only `require_code_owner_review` changed to true (spec P4-3). +- [x] Owner: before a repo joins the pilot, check it for people or bots that push to the default branch directly (V25). Done: none in 12 months on `back-end`, `operationcode-pybot` and `operationcode_infra`; `front-end` had a force push on 2026-10-28 that rewrote three merged PRs (cleanup of a malicious dependency payload), not new unreviewed code (spec P4-4). - [ ] Owner: accept that during the pilot a maintainer or repo admin can merge past any failed run, also on their own PR (spec "Pilot and enforcement"). -- [ ] Owner: confirm that the federation rule named in the release code has `job_workflow_ref` at the release SHA. -- [ ] Owner: create the pilot ruleset. Only the rule "Require workflows to pass before merging" with the central workflow `repository_id`, path and release SHA. Targets: the pilot repos by name (first `back-end`), default branch. Bypass in `pull_request` mode: the Maintain and Admin repository roles, and org admins. -- [ ] Owner: read the pilot ruleset back through the [org rules API](https://docs.github.com/en/rest/orgs/rules#get-an-organization-repository-ruleset). Check `repository_id`, path, `sha`, no `ref`, and the role `actor_id` values from V25. Then enable it. -- [ ] Owner: require 2FA for all org members. -- [ ] Owner: CODEOWNERS in each pilot repo for `.github/`, manifests, lockfiles, `pyproject.toml`, `Dockerfile`. +- [x] Owner: confirm that the federation rule named in the release code has `job_workflow_ref` at the release SHA. Read back by the owner at M4 `9e2795b` (2026-10-29, spec P4-5). +- [x] Owner: every pilot repo sends the immutable OIDC subject (spec "Immutable subject", P4-6). Found 2026-10-28: repos made before the GitHub default sent `repo:OperationCode/:...`, the rule rejected them (`match_subject_prefix`), and the first pilot runs failed closed as designed. Fix per repo: `PUT repos/OperationCode//actions/oidc/customization/sub` with `use_default` true and `use_immutable_subject` true. Before it, check the other `id-token` users in the repo and their relying-party trust: `back-end` and `operationcode-pybot` push to ECR through a role whose trust policy gained the immutable subjects (owner; old values kept). After it, all four test PRs passed (all four jobs `success`). +- [x] Owner: create the pilot ruleset. Only the rule "Require workflows to pass before merging" with the central workflow `repository_id`, path and release SHA. Targets: the pilot repos by name (first `back-end`), default branch. Bypass in `pull_request` mode: the Maintain and Admin repository roles, and org admins. Created disabled as `claude-review-pilot` (26132750): four repo ids, `~DEFAULT_BRANCH`, `repository_id` 1432342572, M4 `9e2795b`, bypass `RepositoryRole` 2 and 5 plus `OrganizationAdmin` (spec P4-2). +- [x] Owner: read the pilot ruleset back through the [org rules API](https://docs.github.com/en/rest/orgs/rules#get-an-organization-repository-ruleset). Check `repository_id`, path, `sha`, no `ref`, and the role `actor_id` values from V25. Then enable it. Read back (`do_not_enforce_on_create` false, no `ref`), then enabled 2026-10-28. Rollback: `PUT` with `enforcement` set to `disabled`. +- [x] Owner: require 2FA for all org members. Done in the web UI; all 7 members already had 2FA (spec P4-5). +- [x] Owner: CODEOWNERS in each pilot repo for `.github/`, manifests, lockfiles, `pyproject.toml`, `Dockerfile`. Added on each default branch (`back-end`: @irvingpop @kylemh; `front-end`: @kylemh @irvingpop; `operationcode-pybot` and `operationcode_infra`: @irvingpop; spec P4-3). - [ ] Start a run on each open PR in the pilot repos (push, or close and reopen). - [ ] Each week, record false positives, overrides, bypasses, failed runs by reason, egress blocks and cost per run. Keep cost numbers (any currency amount) in the owner's private notes and the Anthropic Console, never in this public repo. Public text may give ratios and timing only. - [ ] Fix false positives in prompts or code. Each fix goes through phase 2 again, then ships as in "Each release" below. @@ -216,6 +218,7 @@ Exit criteria: - [ ] targets: never `.github`, and none of the repos still in the pilot ruleset - [ ] Owner: read the org ruleset back and check the workflow entry (`repository_id`, path, `sha`, no `ref`). - [ ] Build the supported repo inventory (spec "Supported repos") as a list in this repo. +- [ ] Owner: every repo to be added to the org ruleset first sends the immutable OIDC subject (spec "Immutable subject"). Check `GET repos/OperationCode//actions/oidc/customization/sub`, check the other `id-token` users in the repo and their relying-party trust, set it per repo, and run a test PR (all jobs `success`) before the repo is a target. - [ ] Owner: CODEOWNERS in each supported repo. - [ ] Owner: turn "prevent self-review" back on for the `eval` environment (it is off during phase 2), and decide whether admin bypass stays on. - [ ] Owner: Dependabot alerts with npm malware detection on active repos. @@ -232,7 +235,7 @@ Exit criteria: ## Phase 6: org-wide enforcement -- [ ] Owner: enable the org ruleset with the supported repos that are not in the pilot. Add them in small groups. +- [ ] Owner: enable the org ruleset with the supported repos that are not in the pilot. Add them in small groups. Each repo in a group already sends the immutable OIDC subject (phase 5); without it the run fails closed. - [ ] Owner: move each pilot repo: remove it from the pilot ruleset targets, then add it to the org ruleset targets at once. Never let both rulesets require the workflow on one repo. - [ ] After each group or move, start a run on each open PR, and check that each shows the required workflow passing or failing on its own result. - [ ] Owner: delete the pilot ruleset when no repo is left in it. diff --git a/docs/claude-review-design.md b/docs/claude-review-design.md index 763e672..e1888b0 100644 --- a/docs/claude-review-design.md +++ b/docs/claude-review-design.md @@ -560,6 +560,7 @@ There is one build of the central workflow. Its gate job always applies every ru - **What the pilot does not protect.** A maintainer or repo admin can merge past any failed run, also on their own PR. The override rule "author is not the PR author" does not apply to a bypass. An author can also call a real finding a false positive and ask a maintainer to bypass. The repo's own approval rule is the only other control. The owner accepts this for the pilot repos. - **Why a bypass list and not Evaluate mode.** Ruleset Evaluate mode is on GitHub Enterprise only ([org rules API](https://docs.github.com/en/rest/orgs/rules#create-an-organization-repository-ruleset)). On Team the workflows rule is either enforced or off. - **Bypass actors.** The [org rules API](https://docs.github.com/en/rest/orgs/rules#create-an-organization-repository-ruleset) lists the actor types `OrganizationAdmin` and `RepositoryRole`, and says `actor_id` is required for `RepositoryRole`. "If actor_type is OrganizationAdmin, actor_id is ignored", so the read-back shows `null` there. The `pull_request` bypass mode "means that an actor can only bypass rules on pull requests" ([repo rules API](https://docs.github.com/en/rest/repos/rules)). The docs do not list the `actor_id` values of the built-in Maintain and Admin roles. So an org owner picks the two roles in the ruleset UI, then reads the ruleset back through the API and records the `actor_id` values. Do not assume them. +- **Immutable subject (prerequisite for every target repo).** A repo created before GitHub made the immutable OIDC subject the default sends `sub` as `repo:OperationCode/:`. The federation rule's prefix `repo:OperationCode@10479364/*` rejects it (Console reason `match_subject_prefix`), and all three passes fail closed with "authentication failed" and the gate fails. This happened on the first pilot runs. Before a repo joins any ruleset that requires the workflow, an org owner checks `GET repos/OperationCode//actions/oidc/customization/sub`, and if it does not return the immutable form, sets it per repo with `PUT repos/OperationCode//actions/oidc/customization/sub` with `use_default` true and `use_immutable_subject` true (no org template). The change applies to every OIDC workflow in that repo, not only the review. So first list the repo's other `id-token: write` users and check the trust policy of each relying party (for example the repo's existing cloud OIDC trust policies): each must accept the immutable subject before the change. Add the new subject and keep the old one. Read the setting back, then start a run on a test PR and check that all jobs succeed. - **One ruleset per repo.** No two rulesets may require the workflow on the same repo at the same time. It is not tested whether that starts one run or two; two runs would make the older one `superseded`, so a PR could never pass. When the org ruleset is enabled, its targets leave out the repos still in the pilot. - **Org-wide enforcement.** Enable the org ruleset with the supported repos that are not in the pilot. Then move each pilot repo: remove it from the pilot ruleset targets, then add it to the org ruleset targets at once. When no repo is left, delete the pilot ruleset. Only org admins can then bypass, through a pull request. An override waives false positives, and the bypass covers outages. - **Dependency review.** The dependency-review step has no `continue-on-error`. The gate does not rely on that alone: the job exports the step `outcome`, and the gate fails, not waivable, when it is missing or not `success`, even if `needs.dependency-review.result` is `success`. A unit test fails if any step in the workflow has `continue-on-error` (see "Testing"). @@ -618,7 +619,7 @@ scope: workspace:developer token lifetime: 10 minutes ``` - - **Subject in ID form.** GitHub's `sub` uses the immutable ID form `repo:OperationCode@10479364/@:`. The Console's hint `repo:YOUR_ORG/*` does not match it: a rule with the prefix `repo:OperationCode/*` denied every token (reason `match_subject_prefix`). Every rule for this org uses the prefix `repo:OperationCode@10479364/*` (V3). + - **Subject in ID form.** GitHub's `sub` uses the immutable ID form `repo:OperationCode@10479364/@:`. The Console's hint `repo:YOUR_ORG/*` does not match it: a rule with the prefix `repo:OperationCode/*` denied every token (reason `match_subject_prefix`). Every rule for this org uses the prefix `repo:OperationCode@10479364/*` (V3). This holds only for a repo that sends the immutable form: see "Immutable subject" in "Pilot and enforcement". - **Pattern mode, not CEL.** The Console rule has two match modes, and a rule uses only one. CEL mode refuses a compound condition without a subject prefix (a guard for multi-tenant issuers such as GitHub). So the rule uses pattern mode: the subject prefix plus exact claims. There is no `startsWith` and no list of SHAs. - **Claims in a ruleset-required run.** `job_workflow_ref` ends in `@` (V2, V3). A `workflow_dispatch` run of the same file uses the ref form `@refs/heads/`, so it does not match. `ref` and `sha` name the target repo's default branch, so the rule does not match them. The subject prefix binds the org only; the rule does not match the rest of `sub` (it ends in `:pull_request` and does not name the actor, V2, V4). - **One SHA per rule.** An exact claim holds one value, so one rule allows one release SHA. A release makes a new rule for the new SHA and deletes the old rule later (see "Each release" below). @@ -806,7 +807,7 @@ The tasks, owner actions and exit criteria for each step are in [docs/build-plan 2. **Admin setup** (Irving, needs org owner): see the checklist below. 3. **Build** in this repo, with unit tests passing. There is one build; its gate job fails for real on `fail` (see "Pilot and enforcement"). 4. **Fixture evals and acceptance:** the fixture eval passes, and the workflow acceptance tests pass in the scratch repo, including a fork PR from an account outside the org. -5. **Pilot:** the "blocking" items must pass first. Check that the federation rule named in the release code has the release SHA. An org owner gives each pilot repo its own approval protection, then creates the pilot ruleset (see "Pilot and enforcement") with the release SHA and the pilot repos (first `back-end`), reads it back, and enables it. Then list open PRs and start a run on each (push, or close and reopen), because the rule does not start on PRs that were already open. Watch the real results on real PRs: false positives, overrides, bypasses, egress and cost per run. The owner decides when the pilot is done. +5. **Pilot:** the "blocking" items must pass first. Check that the federation rule named in the release code has the release SHA. An org owner gives each pilot repo its own approval protection and checks that it sends the immutable OIDC subject (see "Immutable subject" in "Pilot and enforcement"), then creates the pilot ruleset (see "Pilot and enforcement") with the release SHA and the pilot repos (first `back-end`), reads it back, and enables it. Then list open PRs and start a run on each (push, or close and reopen), because the rule does not start on PRs that were already open. Watch the real results on real PRs: false positives, overrides, bypasses, egress and cost per run. The owner decides when the pilot is done. 6. **Each release** (from the pilot on): follow "Each release" in "Auth and cost". Create a new federation rule at the placeholder SHA, put its ID in the candidate code, and set the rule to the candidate SHA. Validate the candidate before any pin moves: unit tests, the owner-approved fixture eval and a token exchange under the scratch test ruleset. If the merge makes a new SHA, its tree must equal the tested candidate. Move the pin of every ruleset that requires the workflow and read each back. Start a new run on each open PR, and delete the old rule only when each has completed at the release SHA (V22). 7. **Before org-wide enforcement:** the owner accepts in writing the residuals in "Which run counts" (an earlier pass on the same SHA, the cross-PR effect of one head SHA, a pin rollback, and run deletion, which V6 f confirmed for a Write user). An org owner fixes the org ruleset (see "Admin checklist"). Build the supported repo inventory (see "Supported repos"), and add CODEOWNERS to each supported repo. Revisit harden-runner block mode (K-14). Re-run the key acceptance tests. 8. **Org-wide enforcement:** enable the org ruleset with the supported repos that are not in the pilot, in small groups. Then move each pilot repo from the pilot ruleset to the org ruleset (see "Pilot and enforcement"), and delete the pilot ruleset. After each group, start a run on each open PR, and check that each shows the required workflow passing or failing on its own result. @@ -824,6 +825,7 @@ The tasks, owner actions and exit criteria for each step are in [docs/build-plan - [ ] Add CODEOWNERS in each active repo for `.github/`, `package.json`, lockfiles, `pyproject.toml`, `Dockerfile`. - [ ] Keep the supported repo inventory current (see "Supported repos"). - [ ] Each pilot repo, before it joins the pilot: its own approval protection on the default branch (a repo ruleset or branch protection with at least 1 approval, code owner review, dismiss stale approvals on push). +- [ ] Each target repo, before it joins a ruleset that requires the workflow: the immutable OIDC subject is set and read back, and the trust policy of every other OIDC user in that repo accepts it (see "Pilot and enforcement"). - [ ] Pilot ruleset: only the workflows rule, the release SHA, the pilot repos, bypass for the Maintain and Admin repository roles and org admins in `pull_request` mode. Read it back (`repository_id`, path, `sha`, no `ref`, and the role `actor_id` values), enable it, and start runs on open PRs (rollout step 5). - [ ] Org ruleset, before it is enabled: dismiss stale approvals on push on; `OrganizationAdmin` bypass mode `pull_request`, not `always`; remove the empty required status checks rule; fill the workflows rule with the `repository_id` of `OperationCode/.github`, the path and the pinned SHA; targets leave out the repos still in the pilot ruleset. Read it back, then enable it (rollout steps 7 and 8). - [ ] Before org-wide enforcement: record the owner's acceptance of the residuals in "Which run counts", including the cross-PR effect of one head SHA. Repo writers and admins can cancel, re-run and delete runs; no setting in this design limits that. @@ -985,3 +987,12 @@ P2: phase 2 rulings, 2026-10-27 (UTC). Where a P2- line changes an earlier line, - P3-5. Summary comment redesigned for fast human judgment (owner and coordinator, 2026-10-27). The comment now opens with a result heading and one colored callout, then a findings table, a checks strip and a provenance line, with the waiver and the full record collapsed ("Summary comment" under "Publishing (code)"). The owner chose the Fable design with these rulings: every fail uses the red `[!CAUTION]` callout and the red heading, waivable or not (a waivable fail on a fake check was misread as a mild warning under `[!WARNING]`, so `[!WARNING]` is not used for a fail); the waiver line stays in the collapsed Waiver block; model summaries and token counts stay in the collapsed record; duplicate findings merge only at the same file, side, line and gate effect; a medium finding in a blocking category reads "Does not block"; the visible line shows the 7-character head SHA and the record shows the full SHAs; emoji are allowed, each next to its word. Machine markers, the override line format, the size limit and the overflow path are unchanged. - P3-6. Account X results and Sonnet release (owner and coordinator, 2026-10-28). Account X is `irvingpop2`, a separate GitHub account outside the org that the owner made as a test identity; it became an outside collaborator when it got repo access. Steps ran from an isolated container. V8 b pass (PR 38): the central run started on a first-time-contributor fork PR with no approval click, and marking the draft ready started no new run. V8 a pass (PR 39): from a first-time fork, GitHub holds the fake `pull_request` job (`action_required`), so it cannot start before the central run; after a maintainer approved only that run it passed, and the box stayed BLOCKED and followed the central run; the fake-first case stays covered by PR 36. V23 pass (PR 40): a same-path `claude-review.yml` in the fork PR did not replace the central run, which failed on the bidi hard block; the fork run stayed `action_required` and never ran, so the running case is covered by PR 35. V6 f as a Write user: CONFIRMED residual (PR 41); deleting the newer run reopened the box on an older pass, and it stays in the residuals the owner must accept before org-wide enforcement. A Write user's plain merge of a failed PR was refused; this does not test the bypass, and the `--admin` attempt as Write was not run. V25 pass for Maintain and Admin, Write bypass NOT RUN: with the TR bypass list, a Maintain user (PR 42) and an Admin user (PR 43) merged a failed PR; GitHub records rule suite result `bypass` with the user ID as actor and no role ID, and the org rule-suites endpoint returns 403 on this plan, so use the repo endpoint; a direct push to main by Maintain is refused (GH013). The TR bypass list is a test-only setting and goes back to empty after phase 3 (owner's choice pending). Process: the owner ran the bypass merges and the direct push by hand, because the automation's safety classifier blocks bypass merges; the evidence comes from the GitHub API. Sonnet release: release evals 13 of 13 at M3 ([run 40592861233](https://github.com/OperationCode/.github/actions/runs/40592861233)); the live run [40593729924](https://github.com/OperationCode/claude-review-test/actions/runs/40593729924) passed with `claude-sonnet-5-5`. Console read-back: the owner read back R1 at 590a387 and R2 at 1cefe42 on 2026-10-27, before the M3 edit; R1 was then moved to M3; the service account and the workspace were not read back. V26 a is done for `job_workflow_ref`. NOT RUN, stays open: the Write `--admin` bypass attempt and V26 c deny at M2 before the R2 edit. Web UI "Re-run all jobs": pass (run 40628870689, PR 43, 2026-10-28): the click made attempt 2 of the same run ID, which re-read live state and updated the one summary comment; the merge box was not observed because the PR was already merged. - P3-7. Release M4, the comment redesign (owner and coordinator, 2026-10-28). M4 is `9e2795b97b22869b9e5c7a777e71a2afa8366eeb` on main (code from PR 12 `a410ddd`, docs PR 13). There is no separate candidate branch: the merge SHA was validated directly. Unit tests: `test.yml` at M4, [run 40710532524](https://github.com/OperationCode/.github/actions/runs/40710532524), success. Release evals: `eval.yml` at M4, [run 40713029193](https://github.com/OperationCode/.github/actions/runs/40713029193), 13 of 13 matched, success; the coordinator approved the run under the owner's delegation through phase 4. Console: the owner edited R1 `claude-review-prod-r1` (`fdrl_01N2ZktTiVKqUvvXwywBFfys`) so that `job_workflow_ref` is `OperationCode/.github/.github/workflows/claude-review.yml@9e2795b97b22869b9e5c7a777e71a2afa8366eeb`. The owner reported "rule updated" and shared a screenshot of the rules list; the rule detail value was not read back. TR (org ruleset 25835887) moved from M3 `2c63f49b461feac74c2c70c4f86ca90040df6a0a` to M4; read back: `repository_id` 1432342572, path `.github/workflows/claude-review.yml`, `sha` `9e2795b`, no `ref`, bypass list empty, enforcement active. Live checks in claude-review-test: PR 28 reopened, [run 40713506554](https://github.com/OperationCode/claude-review-test/actions/runs/40713506554) at M4, success, all four jobs `ok`, summary comment in the new layout (green PASS heading, TIP callout, checks strip). PR 40 reopened, [run 40713856249](https://github.com/OperationCode/claude-review-test/actions/runs/40713856249), failure, red FAIL heading, CAUTION callout, the hard block row (bidi) first, merged duplicate rows with "Found by" listing both passes. Both PRs were closed again. Console rules in the owner's screenshot: `claude-review-p3-r2`, `claude-review-prod-r1`, `eval-federation-rule`, `claude-reviewer-test`; the coordinator advised archiving `claude-review-p3-r2`. Deviation 1: no new federation rule for this release; R1 was edited in place, because the merged code names R1 (same as the phase 3 release, where R1 moved from S1 to M3). Cost: M3 can no longer get a token, so a rollback to M3 needs another Console edit. Deviation 2: the denied exchange at M4 before the Console edit (release step 5a) was NOT RUN, because the owner made the edit before the check. M4 is the current production SHA and the current pin. + +P4: phase 4 pilot start, 2026-10-28 and 2026-10-29 (owner and coordinator). Where a P4- line changes an earlier line, it wins. + +- P4-1. Pilot scope (owner decision). The build plan named only `back-end`. The owner added three repos, so the pilot has four: `back-end` (182584277, `main`), `front-end` (135777271, `main`), `operationcode-pybot` (146210805, `master`) and `operationcode_infra` (107068150, `main`). +- P4-2. Pilot ruleset. Org ruleset `claude-review-pilot` (id 26132750): active; targets the four repo ids and `~DEFAULT_BRANCH`; one workflows rule with `repository_id` 1432342572, path `.github/workflows/claude-review.yml`, `sha` `9e2795b97b22869b9e5c7a777e71a2afa8366eeb` (M4), no `ref`, `do_not_enforce_on_create` false; bypass `RepositoryRole` 2 (Maintain), `RepositoryRole` 5 (Admin) and `OrganizationAdmin`, all in `pull_request` mode. It was created disabled, read back, then enabled. Rollback: `PUT` with `enforcement` set to `disabled`. +- P4-3. Approval protection and CODEOWNERS. Repo rulesets "Pilot approval protection" on `back-end` (26057791), `operationcode-pybot` (26057793) and `operationcode_infra` (26057794): 1 approval, code owner review, dismiss stale reviews, and the same bypass as the pilot ruleset. The reason for the bypass: `operationcode-pybot` and `operationcode_infra` have one active committer, so a rule without a bypass would block every merge. On `front-end` the existing ruleset "Protect main" (25634150) changed in one field only: `require_code_owner_review` false to true. CODEOWNERS was added on each default branch (`back-end`: @irvingpop @kylemh; `front-end`: @kylemh @irvingpop; `operationcode-pybot` and `operationcode_infra`: @irvingpop). +- P4-4. Direct-push check (V25), 12 months: none on `back-end`, `operationcode-pybot` and `operationcode_infra`. `front-end` had a force push on 2026-10-28 that rewrote three merged PRs, a cleanup of a malicious dependency payload; it was not new unreviewed code. +- P4-5. Org 2FA. The owner turned on the org 2FA requirement in the web UI. All 7 members already had 2FA. The federation rule R1 `job_workflow_ref` was read back by the owner at M4 `9e2795b` (this closes the read-back gap in P3-7). +- P4-6. Spec gap: the immutable subject (design change). Repos created before GitHub's immutable OIDC subject default send `sub` as `repo:OperationCode/:...`, and the federation rule's prefix `repo:OperationCode@10479364/` rejects it (Console reason `match_subject_prefix`). The first pilot runs failed closed as designed: all three passes "authentication failed", gate FAIL, exit 1. Fix, per repo: `PUT repos/OperationCode//actions/oidc/customization/sub` with `use_default` true and `use_immutable_subject` true (no org template). This changes `sub` for every OIDC workflow in the repo. `back-end` and `operationcode-pybot` push images through a cloud role whose trust policy had to gain the immutable subjects; the owner did that and kept the old values. After the change all four test PRs passed, with all four jobs `success`. The spec now lists the subject as a prerequisite for every target repo ("Pilot and enforcement"), and phases 4, 5 and 6 of the build plan have the step. From 0104d1afd98cc9f21fc52600dba46901ec98aba1 Mon Sep 17 00:00:00 2001 From: Irving Popovetsky Date: Thu, 29 Oct 2026 08:49:00 -0700 Subject: [PATCH 2/2] Fix pilot dates, immutable-subject wording and stale back-end-only text; add open-PR timing lesson (P4 fix round 1) Co-Authored-By: Claude Sonnet 5.5 --- docs/build-plan.md | 8 ++++---- docs/claude-review-design.md | 14 +++++++------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/build-plan.md b/docs/build-plan.md index cd29838..6c7b71c 100644 --- a/docs/build-plan.md +++ b/docs/build-plan.md @@ -190,12 +190,12 @@ Exit criteria: - [x] Owner: before a repo joins the pilot, check it for people or bots that push to the default branch directly (V25). Done: none in 12 months on `back-end`, `operationcode-pybot` and `operationcode_infra`; `front-end` had a force push on 2026-10-28 that rewrote three merged PRs (cleanup of a malicious dependency payload), not new unreviewed code (spec P4-4). - [ ] Owner: accept that during the pilot a maintainer or repo admin can merge past any failed run, also on their own PR (spec "Pilot and enforcement"). - [x] Owner: confirm that the federation rule named in the release code has `job_workflow_ref` at the release SHA. Read back by the owner at M4 `9e2795b` (2026-10-29, spec P4-5). -- [x] Owner: every pilot repo sends the immutable OIDC subject (spec "Immutable subject", P4-6). Found 2026-10-28: repos made before the GitHub default sent `repo:OperationCode/:...`, the rule rejected them (`match_subject_prefix`), and the first pilot runs failed closed as designed. Fix per repo: `PUT repos/OperationCode//actions/oidc/customization/sub` with `use_default` true and `use_immutable_subject` true. Before it, check the other `id-token` users in the repo and their relying-party trust: `back-end` and `operationcode-pybot` push to ECR through a role whose trust policy gained the immutable subjects (owner; old values kept). After it, all four test PRs passed (all four jobs `success`). -- [x] Owner: create the pilot ruleset. Only the rule "Require workflows to pass before merging" with the central workflow `repository_id`, path and release SHA. Targets: the pilot repos by name (first `back-end`), default branch. Bypass in `pull_request` mode: the Maintain and Admin repository roles, and org admins. Created disabled as `claude-review-pilot` (26132750): four repo ids, `~DEFAULT_BRANCH`, `repository_id` 1432342572, M4 `9e2795b`, bypass `RepositoryRole` 2 and 5 plus `OrganizationAdmin` (spec P4-2). -- [x] Owner: read the pilot ruleset back through the [org rules API](https://docs.github.com/en/rest/orgs/rules#get-an-organization-repository-ruleset). Check `repository_id`, path, `sha`, no `ref`, and the role `actor_id` values from V25. Then enable it. Read back (`do_not_enforce_on_create` false, no `ref`), then enabled 2026-10-28. Rollback: `PUT` with `enforcement` set to `disabled`. +- [x] Owner: every pilot repo sends the immutable OIDC subject (spec "Immutable subject", P4-6). Found 2026-10-29: repos made before the GitHub default sent `repo:OperationCode/:...`, the rule rejected them (`match_subject_prefix`), and the first pilot runs failed closed as designed. Fix per repo: `PUT repos/OperationCode//actions/oidc/customization/sub` with `use_default` true and `use_immutable_subject` true. Before it, check the other `id-token` users in the repo and their relying-party trust: `back-end` and `operationcode-pybot` use a cloud OIDC trust policy that gained the immutable subjects (owner; old values kept). After it, all four test PRs passed (all four jobs `success`). +- [x] Owner: create the pilot ruleset. Only the rule "Require workflows to pass before merging" with the central workflow `repository_id`, path and release SHA. Targets: the pilot repos by name (`back-end`, `front-end`, `operationcode-pybot`, `operationcode_infra`), default branch. Bypass in `pull_request` mode: the Maintain and Admin repository roles, and org admins. Created disabled as `claude-review-pilot` (26132750): four repo ids, `~DEFAULT_BRANCH`, `repository_id` 1432342572, M4 `9e2795b`, bypass `RepositoryRole` 2 and 5 plus `OrganizationAdmin` (spec P4-2). +- [x] Owner: read the pilot ruleset back through the [org rules API](https://docs.github.com/en/rest/orgs/rules#get-an-organization-repository-ruleset). Check `repository_id`, path, `sha`, no `ref`, and the role `actor_id` values from V25. Then enable it. Read back (`do_not_enforce_on_create` false, no `ref`), then enabled 2026-10-29 (created and enabled the same morning). Rollback: `PUT` with `enforcement` set to `disabled`. - [x] Owner: require 2FA for all org members. Done in the web UI; all 7 members already had 2FA (spec P4-5). - [x] Owner: CODEOWNERS in each pilot repo for `.github/`, manifests, lockfiles, `pyproject.toml`, `Dockerfile`. Added on each default branch (`back-end`: @irvingpop @kylemh; `front-end`: @kylemh @irvingpop; `operationcode-pybot` and `operationcode_infra`: @irvingpop; spec P4-3). -- [ ] Start a run on each open PR in the pilot repos (push, or close and reopen). +- [ ] Right after enabling the pilot ruleset, list the open PRs in the pilot repos again (do not reuse a list from earlier recon) and start a run on each (close and reopen, or a new push). A required workflow starts only on a new PR event, so a PR opened between the recon and the enable gets no review run: front-end PR 2027 (opened 2026-10-29T08:06Z) was missed this way. - [ ] Each week, record false positives, overrides, bypasses, failed runs by reason, egress blocks and cost per run. Keep cost numbers (any currency amount) in the owner's private notes and the Anthropic Console, never in this public repo. Public text may give ratios and timing only. - [ ] Fix false positives in prompts or code. Each fix goes through phase 2 again, then ships as in "Each release" below. - [ ] Keep the fixture eval results as an artifact: `eval.yml` uploads none today, so a release eval's per-case results exist only in the job log (phase 2 run 40445885087 and phase 3 run 40528717338 were read from logs). Ships as in "Each release". diff --git a/docs/claude-review-design.md b/docs/claude-review-design.md index e1888b0..da7d1ca 100644 --- a/docs/claude-review-design.md +++ b/docs/claude-review-design.md @@ -2,7 +2,7 @@ Status: draft, revised after several adversarial design reviews and platform test rounds (see "Decision log"). Code that does not depend on the gate may be built in parallel with the "build" items in "Verify before building"; it merges to main only after they pass. Build plan: [docs/build-plan.md](build-plan.md). Owner: Irving Popovetsky -Last updated: 2026-10-26 +Last updated: 2026-10-29 ## Summary @@ -560,7 +560,7 @@ There is one build of the central workflow. Its gate job always applies every ru - **What the pilot does not protect.** A maintainer or repo admin can merge past any failed run, also on their own PR. The override rule "author is not the PR author" does not apply to a bypass. An author can also call a real finding a false positive and ask a maintainer to bypass. The repo's own approval rule is the only other control. The owner accepts this for the pilot repos. - **Why a bypass list and not Evaluate mode.** Ruleset Evaluate mode is on GitHub Enterprise only ([org rules API](https://docs.github.com/en/rest/orgs/rules#create-an-organization-repository-ruleset)). On Team the workflows rule is either enforced or off. - **Bypass actors.** The [org rules API](https://docs.github.com/en/rest/orgs/rules#create-an-organization-repository-ruleset) lists the actor types `OrganizationAdmin` and `RepositoryRole`, and says `actor_id` is required for `RepositoryRole`. "If actor_type is OrganizationAdmin, actor_id is ignored", so the read-back shows `null` there. The `pull_request` bypass mode "means that an actor can only bypass rules on pull requests" ([repo rules API](https://docs.github.com/en/rest/repos/rules)). The docs do not list the `actor_id` values of the built-in Maintain and Admin roles. So an org owner picks the two roles in the ruleset UI, then reads the ruleset back through the API and records the `actor_id` values. Do not assume them. -- **Immutable subject (prerequisite for every target repo).** A repo created before GitHub made the immutable OIDC subject the default sends `sub` as `repo:OperationCode/:`. The federation rule's prefix `repo:OperationCode@10479364/*` rejects it (Console reason `match_subject_prefix`), and all three passes fail closed with "authentication failed" and the gate fails. This happened on the first pilot runs. Before a repo joins any ruleset that requires the workflow, an org owner checks `GET repos/OperationCode//actions/oidc/customization/sub`, and if it does not return the immutable form, sets it per repo with `PUT repos/OperationCode//actions/oidc/customization/sub` with `use_default` true and `use_immutable_subject` true (no org template). The change applies to every OIDC workflow in that repo, not only the review. So first list the repo's other `id-token: write` users and check the trust policy of each relying party (for example the repo's existing cloud OIDC trust policies): each must accept the immutable subject before the change. Add the new subject and keep the old one. Read the setting back, then start a run on a test PR and check that all jobs succeed. +- **Immutable subject (prerequisite for every target repo).** A repo created before GitHub made the immutable OIDC subject the default sends `sub` as `repo:OperationCode/:`. The federation rule's prefix `repo:OperationCode@10479364/*` rejects it (Console reason `match_subject_prefix`), and all three passes fail closed with "authentication failed" and the gate fails. This happened on the first pilot runs. Before a repo joins any ruleset that requires the workflow, an org owner checks `GET repos/OperationCode//actions/oidc/customization/sub`, and if `use_immutable_subject` is not `true` (the `sub_claim_prefix` must contain `@`, not the repo name), sets it per repo with `PUT repos/OperationCode//actions/oidc/customization/sub` with `use_default` true and `use_immutable_subject` true (no org template). The change applies to every OIDC workflow in that repo, not only the review. So first list the repo's other `id-token: write` users and check the trust policy of each relying party (for example a cloud OIDC trust policy): each must accept the immutable subject before the change. Add the new subject and keep the old one. Read the setting back, then start a run on a test PR and check that all jobs succeed. - **One ruleset per repo.** No two rulesets may require the workflow on the same repo at the same time. It is not tested whether that starts one run or two; two runs would make the older one `superseded`, so a PR could never pass. When the org ruleset is enabled, its targets leave out the repos still in the pilot. - **Org-wide enforcement.** Enable the org ruleset with the supported repos that are not in the pilot. Then move each pilot repo: remove it from the pilot ruleset targets, then add it to the org ruleset targets at once. When no repo is left, delete the pilot ruleset. Only org admins can then bypass, through a pull request. An override waives false positives, and the bypass covers outages. - **Dependency review.** The dependency-review step has no `continue-on-error`. The gate does not rely on that alone: the job exports the step `outcome`, and the gate fails, not waivable, when it is missing or not `success`, even if `needs.dependency-review.result` is `success`. A unit test fails if any step in the workflow has `continue-on-error` (see "Testing"). @@ -807,7 +807,7 @@ The tasks, owner actions and exit criteria for each step are in [docs/build-plan 2. **Admin setup** (Irving, needs org owner): see the checklist below. 3. **Build** in this repo, with unit tests passing. There is one build; its gate job fails for real on `fail` (see "Pilot and enforcement"). 4. **Fixture evals and acceptance:** the fixture eval passes, and the workflow acceptance tests pass in the scratch repo, including a fork PR from an account outside the org. -5. **Pilot:** the "blocking" items must pass first. Check that the federation rule named in the release code has the release SHA. An org owner gives each pilot repo its own approval protection and checks that it sends the immutable OIDC subject (see "Immutable subject" in "Pilot and enforcement"), then creates the pilot ruleset (see "Pilot and enforcement") with the release SHA and the pilot repos (first `back-end`), reads it back, and enables it. Then list open PRs and start a run on each (push, or close and reopen), because the rule does not start on PRs that were already open. Watch the real results on real PRs: false positives, overrides, bypasses, egress and cost per run. The owner decides when the pilot is done. +5. **Pilot:** the "blocking" items must pass first. Check that the federation rule named in the release code has the release SHA. An org owner gives each pilot repo its own approval protection and checks that it sends the immutable OIDC subject (see "Immutable subject" in "Pilot and enforcement"), then creates the pilot ruleset (see "Pilot and enforcement") with the release SHA and the pilot repos, reads it back, and enables it. Then, right after the enable, list the open PRs again (not from earlier recon) and start a run on each (close and reopen, or a new push). A required workflow starts only on a new PR event, so a PR opened between the recon and the enable gets no run (front-end PR 2027, opened 2026-10-29T08:06Z, was missed this way). Watch the real results on real PRs: false positives, overrides, bypasses, egress and cost per run. The owner decides when the pilot is done. 6. **Each release** (from the pilot on): follow "Each release" in "Auth and cost". Create a new federation rule at the placeholder SHA, put its ID in the candidate code, and set the rule to the candidate SHA. Validate the candidate before any pin moves: unit tests, the owner-approved fixture eval and a token exchange under the scratch test ruleset. If the merge makes a new SHA, its tree must equal the tested candidate. Move the pin of every ruleset that requires the workflow and read each back. Start a new run on each open PR, and delete the old rule only when each has completed at the release SHA (V22). 7. **Before org-wide enforcement:** the owner accepts in writing the residuals in "Which run counts" (an earlier pass on the same SHA, the cross-PR effect of one head SHA, a pin rollback, and run deletion, which V6 f confirmed for a Write user). An org owner fixes the org ruleset (see "Admin checklist"). Build the supported repo inventory (see "Supported repos"), and add CODEOWNERS to each supported repo. Revisit harden-runner block mode (K-14). Re-run the key acceptance tests. 8. **Org-wide enforcement:** enable the org ruleset with the supported repos that are not in the pilot, in small groups. Then move each pilot repo from the pilot ruleset to the org ruleset (see "Pilot and enforcement"), and delete the pilot ruleset. After each group, start a run on each open PR, and check that each shows the required workflow passing or failing on its own result. @@ -826,7 +826,7 @@ The tasks, owner actions and exit criteria for each step are in [docs/build-plan - [ ] Keep the supported repo inventory current (see "Supported repos"). - [ ] Each pilot repo, before it joins the pilot: its own approval protection on the default branch (a repo ruleset or branch protection with at least 1 approval, code owner review, dismiss stale approvals on push). - [ ] Each target repo, before it joins a ruleset that requires the workflow: the immutable OIDC subject is set and read back, and the trust policy of every other OIDC user in that repo accepts it (see "Pilot and enforcement"). -- [ ] Pilot ruleset: only the workflows rule, the release SHA, the pilot repos, bypass for the Maintain and Admin repository roles and org admins in `pull_request` mode. Read it back (`repository_id`, path, `sha`, no `ref`, and the role `actor_id` values), enable it, and start runs on open PRs (rollout step 5). +- [ ] Pilot ruleset: only the workflows rule, the release SHA, the pilot repos, bypass for the Maintain and Admin repository roles and org admins in `pull_request` mode. Read it back (`repository_id`, path, `sha`, no `ref`, and the role `actor_id` values), enable it, then list the open PRs again and start a run on each (rollout step 5). - [ ] Org ruleset, before it is enabled: dismiss stale approvals on push on; `OrganizationAdmin` bypass mode `pull_request`, not `always`; remove the empty required status checks rule; fill the workflows rule with the `repository_id` of `OperationCode/.github`, the path and the pinned SHA; targets leave out the repos still in the pilot ruleset. Read it back, then enable it (rollout steps 7 and 8). - [ ] Before org-wide enforcement: record the owner's acceptance of the residuals in "Which run counts", including the cross-PR effect of one head SHA. Repo writers and admins can cancel, re-run and delete runs; no setting in this design limits that. @@ -915,7 +915,7 @@ T2: decisions from the workflows-rule tests T6 and T7 in [claude-review-test](ht - T2-5. "Any newer Actions job" claim (review Important 4): **moot**. The gate no longer depends on check-name selection. V14 and V19 are moot; V20 and V21 merge into V6. - T2-6. Summary ordering (review Minor 5): **accept**. The sticky summary is best effort; run ordering, duplicate deletion and their tests are removed. - T2-7. Fail closed: the required run is `success` only when every job succeeds, so a failed, cancelled or timed-out job blocks whatever the gate job does. No job has an `if:` except `always()` on the gate job. The gate job adds `actions: read` (read only) for the final check. -- T2-8. Rollout (changed by K-2 and P0-10): no advisory mode in target repos, because the review runs there only when required. The pilot on `back-end` runs blocking, with override and bypass as relief. The ruleset is created disabled and an org owner enables it. Each release adds the new SHA to the federation rule before the pin moves (V22). +- T2-8. Rollout (changed by K-2 and P0-10): no advisory mode in target repos, because the review runs there only when required. The pilot (four repos, P4-1; was `back-end` only) runs blocking, with override and bypass as relief. The ruleset is created disabled and an org owner enables it. Each release adds the new SHA to the federation rule before the pin moves (V22). - T2-9. Verify list: V1 answered; V2 to V13 relabeled "build" or "blocking"; V8 narrowed to a fake that starts first and non-member accounts; V22 (pin after main moves), V23 (same-path workflow in the target repo) and V24 (run list) added. T3: decisions on review J2 of the T2 revision, 2026-10-25. Where a T3- line changes a T2- line, the T3- line wins. @@ -988,11 +988,11 @@ P2: phase 2 rulings, 2026-10-27 (UTC). Where a P2- line changes an earlier line, - P3-6. Account X results and Sonnet release (owner and coordinator, 2026-10-28). Account X is `irvingpop2`, a separate GitHub account outside the org that the owner made as a test identity; it became an outside collaborator when it got repo access. Steps ran from an isolated container. V8 b pass (PR 38): the central run started on a first-time-contributor fork PR with no approval click, and marking the draft ready started no new run. V8 a pass (PR 39): from a first-time fork, GitHub holds the fake `pull_request` job (`action_required`), so it cannot start before the central run; after a maintainer approved only that run it passed, and the box stayed BLOCKED and followed the central run; the fake-first case stays covered by PR 36. V23 pass (PR 40): a same-path `claude-review.yml` in the fork PR did not replace the central run, which failed on the bidi hard block; the fork run stayed `action_required` and never ran, so the running case is covered by PR 35. V6 f as a Write user: CONFIRMED residual (PR 41); deleting the newer run reopened the box on an older pass, and it stays in the residuals the owner must accept before org-wide enforcement. A Write user's plain merge of a failed PR was refused; this does not test the bypass, and the `--admin` attempt as Write was not run. V25 pass for Maintain and Admin, Write bypass NOT RUN: with the TR bypass list, a Maintain user (PR 42) and an Admin user (PR 43) merged a failed PR; GitHub records rule suite result `bypass` with the user ID as actor and no role ID, and the org rule-suites endpoint returns 403 on this plan, so use the repo endpoint; a direct push to main by Maintain is refused (GH013). The TR bypass list is a test-only setting and goes back to empty after phase 3 (owner's choice pending). Process: the owner ran the bypass merges and the direct push by hand, because the automation's safety classifier blocks bypass merges; the evidence comes from the GitHub API. Sonnet release: release evals 13 of 13 at M3 ([run 40592861233](https://github.com/OperationCode/.github/actions/runs/40592861233)); the live run [40593729924](https://github.com/OperationCode/claude-review-test/actions/runs/40593729924) passed with `claude-sonnet-5-5`. Console read-back: the owner read back R1 at 590a387 and R2 at 1cefe42 on 2026-10-27, before the M3 edit; R1 was then moved to M3; the service account and the workspace were not read back. V26 a is done for `job_workflow_ref`. NOT RUN, stays open: the Write `--admin` bypass attempt and V26 c deny at M2 before the R2 edit. Web UI "Re-run all jobs": pass (run 40628870689, PR 43, 2026-10-28): the click made attempt 2 of the same run ID, which re-read live state and updated the one summary comment; the merge box was not observed because the PR was already merged. - P3-7. Release M4, the comment redesign (owner and coordinator, 2026-10-28). M4 is `9e2795b97b22869b9e5c7a777e71a2afa8366eeb` on main (code from PR 12 `a410ddd`, docs PR 13). There is no separate candidate branch: the merge SHA was validated directly. Unit tests: `test.yml` at M4, [run 40710532524](https://github.com/OperationCode/.github/actions/runs/40710532524), success. Release evals: `eval.yml` at M4, [run 40713029193](https://github.com/OperationCode/.github/actions/runs/40713029193), 13 of 13 matched, success; the coordinator approved the run under the owner's delegation through phase 4. Console: the owner edited R1 `claude-review-prod-r1` (`fdrl_01N2ZktTiVKqUvvXwywBFfys`) so that `job_workflow_ref` is `OperationCode/.github/.github/workflows/claude-review.yml@9e2795b97b22869b9e5c7a777e71a2afa8366eeb`. The owner reported "rule updated" and shared a screenshot of the rules list; the rule detail value was not read back. TR (org ruleset 25835887) moved from M3 `2c63f49b461feac74c2c70c4f86ca90040df6a0a` to M4; read back: `repository_id` 1432342572, path `.github/workflows/claude-review.yml`, `sha` `9e2795b`, no `ref`, bypass list empty, enforcement active. Live checks in claude-review-test: PR 28 reopened, [run 40713506554](https://github.com/OperationCode/claude-review-test/actions/runs/40713506554) at M4, success, all four jobs `ok`, summary comment in the new layout (green PASS heading, TIP callout, checks strip). PR 40 reopened, [run 40713856249](https://github.com/OperationCode/claude-review-test/actions/runs/40713856249), failure, red FAIL heading, CAUTION callout, the hard block row (bidi) first, merged duplicate rows with "Found by" listing both passes. Both PRs were closed again. Console rules in the owner's screenshot: `claude-review-p3-r2`, `claude-review-prod-r1`, `eval-federation-rule`, `claude-reviewer-test`; the coordinator advised archiving `claude-review-p3-r2`. Deviation 1: no new federation rule for this release; R1 was edited in place, because the merged code names R1 (same as the phase 3 release, where R1 moved from S1 to M3). Cost: M3 can no longer get a token, so a rollback to M3 needs another Console edit. Deviation 2: the denied exchange at M4 before the Console edit (release step 5a) was NOT RUN, because the owner made the edit before the check. M4 is the current production SHA and the current pin. -P4: phase 4 pilot start, 2026-10-28 and 2026-10-29 (owner and coordinator). Where a P4- line changes an earlier line, it wins. +P4: phase 4 pilot start, 2026-10-28 and 2026-10-29 (owner and coordinator). The pilot ruleset was created and enabled on 2026-10-29. Where a P4- line changes an earlier line, it wins. - P4-1. Pilot scope (owner decision). The build plan named only `back-end`. The owner added three repos, so the pilot has four: `back-end` (182584277, `main`), `front-end` (135777271, `main`), `operationcode-pybot` (146210805, `master`) and `operationcode_infra` (107068150, `main`). - P4-2. Pilot ruleset. Org ruleset `claude-review-pilot` (id 26132750): active; targets the four repo ids and `~DEFAULT_BRANCH`; one workflows rule with `repository_id` 1432342572, path `.github/workflows/claude-review.yml`, `sha` `9e2795b97b22869b9e5c7a777e71a2afa8366eeb` (M4), no `ref`, `do_not_enforce_on_create` false; bypass `RepositoryRole` 2 (Maintain), `RepositoryRole` 5 (Admin) and `OrganizationAdmin`, all in `pull_request` mode. It was created disabled, read back, then enabled. Rollback: `PUT` with `enforcement` set to `disabled`. - P4-3. Approval protection and CODEOWNERS. Repo rulesets "Pilot approval protection" on `back-end` (26057791), `operationcode-pybot` (26057793) and `operationcode_infra` (26057794): 1 approval, code owner review, dismiss stale reviews, and the same bypass as the pilot ruleset. The reason for the bypass: `operationcode-pybot` and `operationcode_infra` have one active committer, so a rule without a bypass would block every merge. On `front-end` the existing ruleset "Protect main" (25634150) changed in one field only: `require_code_owner_review` false to true. CODEOWNERS was added on each default branch (`back-end`: @irvingpop @kylemh; `front-end`: @kylemh @irvingpop; `operationcode-pybot` and `operationcode_infra`: @irvingpop). - P4-4. Direct-push check (V25), 12 months: none on `back-end`, `operationcode-pybot` and `operationcode_infra`. `front-end` had a force push on 2026-10-28 that rewrote three merged PRs, a cleanup of a malicious dependency payload; it was not new unreviewed code. -- P4-5. Org 2FA. The owner turned on the org 2FA requirement in the web UI. All 7 members already had 2FA. The federation rule R1 `job_workflow_ref` was read back by the owner at M4 `9e2795b` (this closes the read-back gap in P3-7). +- P4-5. Org 2FA. The owner turned on the org 2FA requirement in the web UI. All 7 members already had 2FA. The federation rule R1 `job_workflow_ref` was read back by the owner at M4 `9e2795b` (this closes the read-back gap in P3-7; the P3-7 text below still says "not read back"). - P4-6. Spec gap: the immutable subject (design change). Repos created before GitHub's immutable OIDC subject default send `sub` as `repo:OperationCode/:...`, and the federation rule's prefix `repo:OperationCode@10479364/` rejects it (Console reason `match_subject_prefix`). The first pilot runs failed closed as designed: all three passes "authentication failed", gate FAIL, exit 1. Fix, per repo: `PUT repos/OperationCode//actions/oidc/customization/sub` with `use_default` true and `use_immutable_subject` true (no org template). This changes `sub` for every OIDC workflow in the repo. `back-end` and `operationcode-pybot` push images through a cloud role whose trust policy had to gain the immutable subjects; the owner did that and kept the old values. After the change all four test PRs passed, with all four jobs `success`. The spec now lists the subject as a prerequisite for every target repo ("Pilot and enforcement"), and phases 4, 5 and 6 of the build plan have the step.