[kikashy]

What changed

A third value of profiles.<name>.compare in the release evidence gate, release-os, for evidence jobs whose result does not depend on the hosted image patch version, given exact tools. The motivating case is judgment-pack-agent#332: during a staged image roll, a 28-shard table landed on three image versions, and neither release-runner nor pins could pass.

Under "compare": "release-os":

• Tools. Every tool of the profile must have a pin (config-invalid otherwise). Each job's recorded tools must equal the pins exactly, as under pins.
• OS release and platform. The release job resolves the profile on its own runner, as for release-runner: ImageOS, RUNNER_OS, RUNNER_ARCH and the digest of the variant's OS-release file. It does not resolve the tools, because the pins hold them. This happens before any evidence is read, so a release runner of another platform or image name, or one without the OS-release file, refuses there (runner-platform, identity-unexpected, identity-missing). Each job must report exactly those values (runtime-identity). The jobs of one variant and profile must also agree on everything except ImageVersion (identity-changed).
• ImageVersion. It must still have the reported form. It is recorded per job, and the group record gains image_versions and release_runner. It is compared with nothing: not with the release runner, and not among the profile's jobs.
• Configuration. The mode needs the identity policy (config-invalid: ... compare "release-os" needs config.identity). A variant with no OS-release file (windows-x64) is refused for this mode (identity-config), at capture and at verify.

A difference is a plain refusal, not a drift. An OS-release or platform difference refuses runtime-identity at that job. It is never revalidated, even with revalidation.allowed: true, and it wins over a release-runner drift. Revalidation reruns the dev.json check list on the release runner. That stands in for release-runner jobs because their work is that list. The work this mode is for (table shards, legs with empty checks) is not that list, so a revalidate here would publish with that work covered by nothing. This matches how release-runner already treats what revalidation cannot cover (a mismatched pin, a record of another shape or platform). When a release-runner drift alone decides revalidate, a release-os job's evidence is taken as a pins leg's is, and it is not in revalidation.jobs.

Code: runtime() is split into runner_runtime() (image and platform) plus the tools, with the same refusals in the same order. The group agreement rule moves into identity_group(). check_runtime() gains the release-os comparison, and identity_context() gains the OS-release-file rule. scripts/dev/release_evidence.py is copied again. Its version line (v0.14.0) and JPDEV_VERSION are unchanged; the release prep is a separate pull request.

Public surface: this adds a configuration value and two decision-record members (image_versions and release_runner on release-os groups only). The docs change in the same pull request: the compare and pins rows, the toolchain bullet, the runner identity comparison and image-roll cases, a new section "The release-os mode", the decision record and the revalidation bound in docs/ci-evidence.md; step 3 of the runner identity adoption in docs/adopting.md; and a comment in templates/ci-evidence.json.

What the mode gives up

It is weaker than release-runner, and it is an explicit choice per profile:

• the jobs may run on different image patch versions of one OS release, with different preinstalled software outside the pinned tools;
• only the tools the profile names are bound, and only to their pins; everything else on the image may differ between the jobs and from the release runner;
• the release runner does not run the profile's tools, so nothing shows that it could reproduce the jobs.

Pull-request-to-branch reuse is unchanged: the pull request job's runtime, image version included, must still equal the branch job's.

Tests

tests/release_evidence_test.py adds 11 tests. All 167 existing tests pass unchanged; no existing line was edited.

• ReleaseOS (decide() with three table shards on three image versions, beside the Linux release-runner job and the macOS pins leg). The release runner's resolution of the table runs for real, from a patched environment and an OS-release file the test writes. The tests cover:
  • acceptance across mixed ImageVersions with one OS release and pinned tools, including the group record and a check that only the release-runner profile's tools were resolved;
  • refusal on an OS-release digest difference, with and without revalidation, including the release runner's own OS release moving, which wins over a release-runner drift;
  • a tool not equal to its pin, and another tool set;
  • a platform difference, both on the release runner (refused before any run is read) and in a shard;
  • a direct check_runtime test covering each runner field other than ImageVersion;
  • a direct identity_group test;
  • pins (identity-changed) and release-runner (runtime-identity, or revalidate) still refusing the same roll, with the exact detail strings they had;
  • a revalidation that takes the shards' evidence.
• ReleaseOSConfiguration: no identity policy, a missing pin (another tool, empty pins, no pins member), the pins wording unchanged, an unknown mode, and the windows-x64 variant refused at identity_context over a fixture git repository.
• ReleaseOSCLI: capture on one image version and verify on another through the public command (fixture repository and a stand-in gh). The evidence is reused, refused for another OS release, and refused before any evidence is read for another image name.

Differential check (scratch, not committed): every decision record produced by the 114 pre-existing tests that drive decide() was compared under the old and the new gate. That is 340 records, and all were byte-identical. The same comparison against a mutant that changes the pins group reported 5 tests with different records, so the comparison can see a change.

Mutants. Each was applied to a scratch copy and run through the whole file via RELEASE_EVIDENCE_HELPER. Each was caught at the intended assertion; no setup errors are counted.

Mutant	Caught by
release-os without identity accepted	ReleaseOSConfiguration.test_release_os_needs_the_identity_policy_and_every_tool_pinned
release-os not in the every-tool-pinned rule	the same (3 cases)
OS-release-file rule removed from identity_context	ReleaseOSConfiguration.test_a_variant_without_an_os_release_file_refuses
release runner resolved lazily per job, not before any evidence	ReleaseOS.test_a_platform_that_differs_refuses (the run listing was read), ReleaseOSCLI (workflows recorded)
no comparison with the release runner	ReleaseOS.test_another_os_release_refuses_and_is_never_revalidated, test_the_comparison_holds_every_runner_field_but_the_image_version, ReleaseOSCLI
ImageVersion compared with the release runner	ReleaseOS acceptance, revalidation and comparison tests, ReleaseOSCLI
a difference returned as a drift	ReleaseOS.test_another_os_release_refuses_and_is_never_revalidated (revalidated), comparison test
only os_release compared (platform fields dropped)	ReleaseOS.test_the_comparison_holds_every_runner_field_but_the_image_version
the group compares ImageVersion	ReleaseOS acceptance, revalidation and group tests
the group ignores the other fields	ReleaseOS.test_the_group_holds_every_field_but_the_image_version
the release runner resolves the tools too (runtime())	ReleaseOS acceptance (tools resolved) and platform tests
pins group treated like release-os	Identity.test_legs_of_one_variant_and_profile_that_report_different_identities_refuse, ReleaseOS.test_pins_and_release_runner_still_refuse_the_same_roll, group test
release-os jobs counted in revalidation.jobs	ReleaseOS.test_a_revalidation_takes_the_shards_evidence
release-os removed from COMPARE	the configuration tests and ReleaseOSCLI
the group's identity-changed not raised	Identity.test_legs_of_one_variant_and_profile_that_report_different_identities_refuse, ReleaseOS.test_pins_and_release_runner_still_refuse_the_same_roll

Through decide(), the release-os group agreement is redundant with the per-job comparison: each job is already held to the release runner on every field other than ImageVersion and to the pins on its tools. Its rule is therefore caught by the direct identity_group test, not by a decide() vector.

Checks

Candidate: head 2707c9f, no uncommitted diff. The checks ran on the working tree whose content is this commit's tree. Python 3.12.11.

scripts/dev/check.sh, the whole list, once:

PASS     diff-check               0s
PASS     local-paths              0s
PASS     agent-docs               0s
PASS     bash-n                   0s
PASS     duplication              0s
PASS     json                     0s
PASS     shellcheck               5s
FAIL     tests                    109s, exit 1
== 7 passed, 1 failed, 0 skipped, 0 not run ==

The two failing cases in tests came from the run environment, not the change: a TMPDIR spelled with a .. segment (the generated-docs fixtures require a resolved directory), and a hard ulimit -v that kept the kit's Go wrapper test from changing its own cap. With a resolved TMPDIR and a soft cap, scripts/dev/check.sh --only tests:

PASS     tests                    111s
    449 passed, 0 failed
== 1 passed, 0 failed, 0 skipped, 0 not run ==

tests/release_evidence_test.py alone: 178 tests OK. They are also OK against the vendored scripts/dev/release_evidence.py. CI owns the exact-head run.

Closes #79

🤖 Generated with Claude Code

[kikashy]

Clean-room review (same vendor, not a cross-vendor claim) of #80 at 2707c9f

Scope

• Pull request Release evidence: compare "release-os", the OS release and exact tools but not the image version #80 (issue Release evidence: a compare mode that holds the OS release and exact tools but not the image patch version #79), one commit 2707c9f on main at 9f55549. Files: lib/release_evidence.py,
  scripts/dev/release_evidence.py, docs/ci-evidence.md, docs/adopting.md,
  templates/ci-evidence.json, tests/release_evidence_test.py.
• Read: the whole diff; the gate code it touches and the code around it (check_config,
  identity_context, check_identity, runner_runtime/runtime, check_runtime,
  verify_workflow, identity_group, verify, decide, the reuse plan/readmit paths that
  compare runtimes); the manifest templates/runner-identity.json; the docs sections the PR
  changes, in context.
• Not re-run: the whole suite (CI owns it; lint, tests and DCO are green at this head). Run
  here, in a scratch room with Python 3.12.11, a 4 GB address-space cap and a timeout: the new
  test classes once, the test file once as a baseline (178 OK), probes, a differential and the
  mutants below.
• Vendored copy: scripts/dev/release_evidence.py is byte-identical to lib/release_evidence.py
  apart from line 2 (# from judgment-pack-devkit v0.14.0 (lib/release_evidence.py)), which is
  exactly what tests/run.sh allows (line 2 = the kit version, the rest cmp-equal to lib).

What was established from the code

1. Acceptance holes: none found. Under release-os, each job passes check_identity first
   (identity block equal to the commit's; runner fields exactly the variant's set, each a
   non-empty string; ImageOS the profile's; OS and architecture the variant's; ImageVersion of
   the reported form; os_release a digest; tools non-empty strings), then check_runtime
   (runtime record of exactly runner/tools; platform; tool set equal to the profile's; every
   tool equal to its pin, and check_config makes the pins cover every tool; then every runner
   field of either side but ImageVersion equal to the release runner's resolution). The
   IMAGE_VERSION form check is never skipped: release-os requires the identity policy, so
   check_identity runs for every job, and the release runner's own version is form-checked in
   runner_runtime (variant always passed). A release-os difference is a plain Refusal raised in
   the job loop, so it ends verify before the Drift at the end can be raised: it cannot be
   revalidated and it wins over a release-runner drift. Groups are keyed by (variant, profile)
   and a profile has one mode, so no group mixes modes. here is keyed by profile name; the
   eager loop fills only release-os entries (runner fields, no tools), and only release-runner
   code reads here[...]['tools'].
2. Configuration fails closed. release-os without identity is config-invalid; a missing
   pin, empty pins, no pins member and a pin of no tool are config-invalid; a variant with no
   OS-release file (windows-x64) is identity-config in identity_context, which both capture
   and verify call; a release runner that cannot resolve the profile (missing field, other
   platform or image name, version of another form, no OS-release file) refuses before any run is
   read.
3. Drift vs refusal. An OS-release or tool difference under release-os is a refusal, as a pin
   mismatch already is under pins and for a release-runner profile's own pin; revalidation
   (raised only by Drift, and only for release-runner jobs, runner_bound_jobs) cannot accept
   it. A release-os job is never in revalidation.jobs.
4. Decision record. ImageVersion stays per job in workflows.<key>.jobs.<job>.runtime, and
   the release-os group adds image_versions (per job) and release_runner (the release
   runner's resolution, its version and OS-release digest included), also on a revalidate
   decision (probed).
5. Existing modes. runtime() = runner_runtime() + tools with the same refusals in the same
   order; identity_group() gives pins/release-runner the old record and the old
   identity-changed detail; the pins config message is unchanged. Differential run here: the
   base branch's own test file (167 tests, 114 of which drive decide()) was run against the base
   gate and against this head's gate with decide wrapped to log every record: 340 records each,
   identical once the fixture zip digests (time-stamped at fixture build) are normalized; 0
   failures under either gate.

Findings

ID	Severity	File:line	Finding	Evidence / probe
F1	LOW	docs/ci-evidence.md:310-314, 366-369, 382-383 (also the PR body and commit message)	The docs say a job whose platform is not what the release runner resolves refuses runtime-identity. A platform difference in a job (image name, OS, architecture) is refused earlier by check_identity (lib/release_evidence.py:606-610) with identity-unexpected; only an OS-release difference reaches the release-os comparison (lib/release_evidence.py:1045-1053) and refuses runtime-identity. Both refuse and neither is revalidated, so no acceptance claim is wrong; the code named is.	Probe: a shard with ImageOS ubuntu26, RUNNER_ARCH ARM64 or RUNNER_OS macOS refuses identity-unexpected; a shard with another os_release refuses runtime-identity. The PR's own test_a_platform_that_differs_refuses asserts identity-unexpected for the shard cases.
F2	LOW	docs/ci-evidence.md:313-314, 369-370	The docs name identity-changed as the refusal for release-os jobs that disagree among themselves. Through verify() it cannot be produced for this mode: each job is already held to the one release-runner resolution on every field but ImageVersion and to the pins on its tools, and that per-job refusal is raised inside the job loop, before the groups are compared (lib/release_evidence.py:1418 vs 1513-1518). The PR body says the group rule is redundant through decide(); the docs do not. Harmless defence in depth; the docs could say the per-job comparison refuses first.	Probe: two shards differing only in os_release refuse runtime-identity at the shard. Mutant M16 (the release-os identity-changed never raised in verify) survives the whole file: an equivalent mutant through decide(); the rule itself is held by the direct identity_group test (M08). It is a real backstop: with the per-job pin or tool-set check removed (M13, M14) it is what refuses.
F3	LOW	docs/ci-evidence.md:430-431	release_runner is described as "the image and platform the release runner resolved for the profile, its image version included"; it also holds the release runner's os_release digest. Under-statement, not over-statement.	Probe: the group record on revalidate holds ImageOS, ImageVersion, RUNNER_OS, RUNNER_ARCH and os_release.

No HIGH or MEDIUM finding.

Observed and not filed (documented behaviour, fails closed): every release-os profile is
resolved on the release runner before any evidence is read, so a release-os profile that no job
names and whose platform is not the release runner's refuses every release (runner-platform);
docs/ci-evidence.md:370-374 states that every release-os profile is resolved first.

Probes (over the repository's own ReleaseOS fixture, through decide(); all refused or accepted as the mode requires)

• A shard with ImageVersion absent, empty or a number: identity-missing; with a leading
  space, a trailing newline, full-width digits, no dot, latest or a wildcard:
  identity-unexpected (revalidation allowed in each).
• Identity block absent: identity-missing; another variant or another action set:
  identity-unexpected.
• Record shape: an extra runner field (identity-unexpected), no os_release
  (identity-missing), an extra runtime member (runtime-identity), runner not an object
  (identity-missing), a tool value not a string or a missing tool (identity-missing), an
  upper-case digest (identity-unexpected).
• Every shard on one other OS release, with a release-runner drift and revalidation allowed:
  runtime-identity (not revalidate). Every shard on one tool version other than its pin, same
  setting: runtime-identity. Every shard with an extra tool: runtime-identity (tool set).
• A validation-only run: mixed image versions validated; another OS release runtime-identity.
• A release runner missing ImageVersion, ImageOS, RUNNER_OS or RUNNER_ARCH:
  runner-identity, with no run listing read.
• Configuration: identity: null, an empty pin, no ImageOS, no tools, compare as a list, a pin
  of no tool: each config-invalid.
• revalidate on a release-runner drift keeps every shard's version in the job runtimes and in
  image_versions, and the release runner's in release_runner.

Mutants

Each mutant is one exact replacement (two for M15b and M15c) in a scratch copy of
lib/release_evidence.py, run through the whole test file via RELEASE_EVIDENCE_HELPER.

Baseline at the head: 178 tests OK. A mutant is killed when the file fails; the ERRORs under M09
and M10 are the tests' own reads of the missing record member, not setup errors.

Mutant	Result	Caught by
M01 the release-os comparison with the release runner removed	killed	ReleaseOS.test_another_os_release_refuses_and_is_never_revalidated, ReleaseOS.test_the_comparison_holds_every_runner_field_but_the_image_version, ReleaseOSCLI
M02 the OS-release digest left out of that comparison (platform fields only)	killed	the same three
M03 a release-os difference returned as a drift	killed	test_another_os_release_refuses_and_is_never_revalidated (revalidated instead of refused), the comparison test
M04 the OS-release-file rule removed from identity_context	killed	ReleaseOSConfiguration.test_a_variant_without_an_os_release_file_refuses
M05 release-os accepted without identity	killed	ReleaseOSConfiguration.test_release_os_needs_the_identity_policy_and_every_tool_pinned
M06 the every-tool-pinned rule for pins only	killed	the same (3 cases)
M07 the eager resolution without the variant (no form check of the release runner's version, the default OS-release file)	killed	five ReleaseOS tests
M08 the release-os group ignores the tools	killed	ReleaseOS.test_the_group_holds_every_field_but_the_image_version
M09 the group record without image_versions	killed	ReleaseOS acceptance, the group test, ReleaseOSCLI
M10 the group record without release_runner	killed	ReleaseOS acceptance and revalidation, the group test, ReleaseOSCLI
M11 the job's ImageVersion form check removed (check_identity; under release-os the only hold on it)	killed	ReleaseOS.test_a_platform_that_differs_refuses (shard latest), Identity.test_an_unexpected_identity_refuses_and_is_never_revalidated
M12 the release runner's ImageVersion form check removed (runner_runtime)	killed	ReleaseOS.test_a_platform_that_differs_refuses, IdentityAtCommit, IdentityCLI
M13 the recorded tools not held to the pins under release-os	killed	ReleaseOS.test_a_tool_that_is_not_its_pin_or_another_tool_set_refuses (the group rule refuses identity-changed instead of runtime-identity)
M14 the tool set not held under release-os	killed	the same, same way
M15 a lazy full resolution added beside the eager loop	survived: equivalent as constructed (the eager loop fills here first)	replaced by M15b and M15c
M15b the eager loop removed; the release runner resolves release-os lazily, tools included	killed	ReleaseOS.test_a_platform_that_differs_refuses (run listing read), ReleaseOS acceptance (tools resolved), ReleaseOSCLI
M15c the eager loop removed; lazy per-job resolution without tools	killed	ReleaseOS.test_a_platform_that_differs_refuses, ReleaseOSCLI (workflows recorded)
M16 the release-os group's identity-changed not raised in verify	survived: equivalent through decide() (F2); the rule is held by the direct group test	none needed
M17 the comparison over the release runner's keys only	killed	ReleaseOS.test_the_comparison_holds_every_runner_field_but_the_image_version
M18 a pins group never refuses (regression of the existing mode)	killed	Identity.test_legs_of_one_variant_and_profile_that_report_different_identities_refuse, ReleaseOS.test_pins_and_release_runner_still_refuse_the_same_roll, the group test

Regression check beside the mutants: the differential above (340 base-branch decision records,
identical under the old and the new gate).

Verdict

merge: yes