diff --git a/docs/adopting.md b/docs/adopting.md index 9425cdc..353adc3 100644 --- a/docs/adopting.md +++ b/docs/adopting.md @@ -292,7 +292,16 @@ the repository's own reviewed change: `windows-x64`; its `RUNNER_OS` and `RUNNER_ARCH` must be the variant's), its `platform.ImageOS` (the image name its runners report: `ubuntu24`, `macos15`), and at least one tool. A profile of the release job's platform keeps `compare: release-runner`; a leg of another OS or architecture - keeps `compare: pins`, and its legs must agree among themselves. + keeps `compare: pins`, and its legs must agree among themselves. A profile of the release job's + platform whose jobs cannot all land on one image, such as a sharded table during a staged image + roll, may instead take `compare: release-os`, as an explicit choice per profile: every tool + pinned, the image name, platform and OS-release digest held to what the release runner resolves + for the variant, and the image version recorded but never compared. It gives up the rest of the + image: the jobs may run on different image patch versions, with different preinstalled software + outside the pinned tools, so it is weaker than `release-runner`. A difference in the OS release + or platform refuses and is never revalidated + ([the release-os mode](ci-evidence.md#the-release-os-mode)). Use it only for jobs whose result, + given exact tools, does not depend on the image patch version. 4. Pin every `uses:` of the CI workflows, of the local actions and reusable workflows they call, and of the release workflow to a full commit id (a container image to a `sha256` digest). Write each `uses:` as a block-style key on its own line. diff --git a/docs/ci-evidence.md b/docs/ci-evidence.md index ffb9e55..9b9d27a 100644 --- a/docs/ci-evidence.md +++ b/docs/ci-evidence.md @@ -129,8 +129,8 @@ naming it); `"$comment"` is allowed and ignored in any object whose members are | `profiles..platform` | `RUNNER_OS`, `RUNNER_ARCH`, optionally `ImageOS`: what the job's runner must report; under `identity`, `ImageOS` is required | | `profiles..variant` | under `identity` only, and required there: the manifest variant (`linux-x64`, `macos-arm64`...) whose OS and architecture are the profile's platform | | `profiles..tools` | tool name to the command that prints its identity (`["go", "version"]`); each output is recorded and must not be empty | -| `profiles..compare` | `release-runner`: the release job resolves the same profile on its own runner, and the evidence must match it exactly (image, version, OS release, every tool); for a leg on the release job's platform. `pins`: the recorded tools must equal `pins` exactly; for a leg the release job cannot resolve itself (another OS or architecture) | -| `profiles..pins` | tool name to its exact expected output: required for every tool under `pins` (which needs at least one tool), optional under `release-runner`, where it pins a toolchain version on top | +| `profiles..compare` | `release-runner`: the release job resolves the same profile on its own runner, and the evidence must match it exactly (image, version, OS release, every tool); for a leg on the release job's platform. `pins`: the recorded tools must equal `pins` exactly; for a leg the release job cannot resolve itself (another OS or architecture). `release-os`, under `identity` only: the recorded tools must equal `pins` exactly, and the image name, OS, architecture and OS-release digest must be what the release job resolves for the profile's variant on its own runner; the image version is recorded and never compared. For a leg on the release job's platform whose result does not depend on the image patch version; weaker than `release-runner`, and an explicit choice ([the release-os mode](#the-release-os-mode)) | +| `profiles..pins` | tool name to its exact expected output: required for every tool under `pins` and `release-os` (each needs at least one tool), optional under `release-runner`, where it pins a toolchain version on top | | `workflows..path`, `.name` | the workflow file under `.github/workflows/` and its `name:` | | `workflows..events` | `push`, and `workflow_dispatch` if the fresh route dispatches it | | `workflows..skipped_jobs` | optional: display names of jobs that must be present and skipped on the trusted branch | @@ -217,7 +217,9 @@ that read another checkout refuses (`input-identity`). The rest of what a run de `pins`, the recorded tool output must equal the pin, the platform must match, and `ImageOS` when pinned; the image version is recorded but not compared with the release runner (under the [runner identity](#runner-identity) policy, the legs of one variant and profile must agree on - it). + it). Under `release-os`, the recorded tools must equal the pins, and the image name, platform and + OS-release digest must be what the release runner resolves for the variant; the image version is + recorded and compared with nothing ([the release-os mode](#the-release-os-mode)). - **Dependency and integration pins.** A pin file in `inputs` (a lock file, a companion version file) is bound by the commit. An input fetched at run time without a pin ("latest") is not reproducible and cannot be reused: pin it, or keep that check fresh at release. @@ -304,7 +306,13 @@ Neither is a drift: each refuses even where revalidation is allowed. Then: - a `pins` profile's jobs (a leg the release runner cannot resolve, such as macOS) must have reported one identity between them, image version and OS-release digest included: `identity-changed` otherwise, which is never revalidated (the release runner cannot run that - leg). + leg); +- a `release-os` profile's jobs must each report, in every runner field but `ImageVersion` (image + name, OS, architecture, OS-release digest), what the release runner resolves for the variant, and + the pinned tools; between them they must agree on all of it but `ImageVersion`. + `ImageVersion` is recorded and compared with nothing. A difference refuses (`runtime-identity`, + or `identity-changed` between the jobs), and is never revalidated + ([the release-os mode](#the-release-os-mode)). The release workflow's own `uses:` must be pinned the same way (`identity-unexpected`; read from `GITHUB_WORKFLOW_REF` at the commit). The decision record's `identity` holds the manifest record, @@ -318,10 +326,15 @@ actions, and per variant and profile the jobs and the identity they reported. evidence. The identity changed, so the evidence is not reused: the release refuses (`runtime-identity`), or decides `revalidate` where the repository allows it and runs its check list strictly on the release runner. `pins` legs are not compared with the release runner. + `release-os` jobs are not compared on the image version: they stand where the roll left their + OS release and pinned tools as they were, and refuse (`runtime-identity`, not revalidated) + where it changed the OS-release digest. 2. *During the CI run*: two jobs of one variant and profile ran on different images. Under `release-runner` at least one differs from the release runner: the same as 1. Under `pins`: - `identity-changed`. The fresh route is to re-run all jobs of the CI run at the commit; no older - run is searched for, and no version is normalized to another. + `identity-changed`. Under `release-os`: the evidence stands where the jobs report the release + runner's OS release and platform and the pinned tools; the versions are recorded as they are. + The fresh route is to re-run all jobs of the CI run at the commit; no older run is searched for, + and no version is normalized to another. 3. *The label moves to another OS release* (`ubuntu-latest` from `ubuntu24` to `ubuntu26`): capture refuses on the next main run (`runner-platform`), and a release whose runner reports the new name refuses (`runner-platform`), with no revalidation. Moving the profile's `ImageOS` @@ -336,6 +349,67 @@ held image. The route that pins a complete VM image is a GitHub larger runner wi version, which the [assessment](ci-release-environment-assessment.md) keeps as the direction if the organization's plan changes; this policy does not claim that guarantee. +## The release-os mode + +GitHub rolls a hosted image out in stages, and while it does, one label is served by several image +versions at once. A profile with many parallel jobs, such as a sharded mutation table, then almost +never lands on one image. `release-runner` needs every job to report the release runner's image, +and `pins` under the identity policy needs the jobs to report one image between them, so neither +can pass, and a re-run at the same commit draws the images again. `"compare": "release-os"` is for +evidence jobs whose result does not depend on the image patch version, given exact tools. It needs +the [runner identity](#runner-identity) policy (`config-invalid` without `identity`). + +What it holds: + +- **Tools.** Every tool of the profile has a pin (`config-invalid` otherwise), and each job's + recorded tool output equals its pin exactly (`runtime-identity` otherwise), as under `pins`. +- **OS release and platform.** The release job resolves the profile itself on its own runner, as + under `release-runner`, but only its image name (`ImageOS`), `RUNNER_OS`, `RUNNER_ARCH` and the + digest of the variant's OS-release file, not its tools, which the pins hold. Each job must report + exactly those values (`runtime-identity`), and the jobs of one variant and profile must agree on + them and on their tools (`identity-changed`). The release runner resolves every `release-os` + profile before any evidence is read: a release runner that reports no image name, version, OS or + architecture, one of another platform or image name, one whose image version is not of the + reported form, or one without the variant's OS-release file refuses there (`runner-identity`, + `runner-platform`, `identity-unexpected`, `identity-missing`). A variant with no OS-release file + (`windows-x64`) cannot be held to an OS release, and a `release-os` profile of it is refused + (`identity-config`). +- **ImageVersion.** Recorded in each job's evidence, held to the reported form as under the policy, + and kept in the decision record (each job's runtime, and its group's `image_versions` and + `release_runner`), but compared with nothing: not with the release runner, and not among the + profile's jobs. + +**A difference is a refusal, not a drift.** A job whose OS-release digest or platform is not what +the release runner resolves refuses `runtime-identity` at that job. It is never revalidated, even +where `revalidation.allowed` is `true`, and it wins over a `release-runner` profile's drift, which +is raised only at the end. Revalidation runs the check list `dev.json` declares on the release +runner, and it stands in for the `release-runner` jobs because their work is that list. The work +this mode is meant for (the shards of a table, matrix legs whose `checks` are empty) is not that +list, and no release runner runs it again. A `revalidate` decision on such a difference would +publish with that work covered by nothing, so the route is the [fresh route](#the-fresh-route), or +a new release once the label serves one OS release again. This is how `release-runner` already +treats what revalidation cannot cover: a pin that does not match, or a runtime record of another +shape or platform, refuses and is not revalidated. When a `release-runner` profile's drift alone +decides `revalidate`, a `release-os` job's evidence is taken as a `pins` leg's is: it has passed +every check, its OS release and pins included, and it is not one of `revalidation.jobs`. + +**What it gives up.** It is weaker than `release-runner`, and an adopter's explicit choice, per +profile: + +- the jobs may run on different image patch versions of one OS release, with different + preinstalled software outside the pinned tools (a system package, a library, a browser, a + runtime the profile does not name); +- only the tools the profile names are bound, and only to their pins; everything else on the image + may differ between the jobs, and from the release runner; +- the release runner does not run the profile's tools, so nothing shows that it could reproduce + the jobs. + +Choose it only for jobs whose result, given exact tools, does not depend on what else the image +holds, and pin every tool the result does depend on. Keep `release-runner` for jobs whose work the +release would revalidate, and `pins` for legs the release runner cannot resolve. +[Pull-request-to-branch reuse](#pull-request-to-branch-reuse) is unchanged under every mode: a pull +request job's runtime record, image version included, must equal the branch job's. + ## The decision record `verify` writes one JSON record, whatever the decision, to standard output (the release log) and @@ -351,8 +425,10 @@ It holds: runner); `null` otherwise; - `identity`: under the [runner identity](#runner-identity) policy, the manifest record (path, digest, revision, kind), the statement that it is a reported identity and not a pinned VM image, - the release workflow's actions, and per variant and profile the jobs and the identity they - reported; `null` otherwise; + the release workflow's actions, and per variant and profile the `compare` mode, the `jobs` and + the identity they reported (`runtime`, `null` where they differ); for a `release-os` profile also + `image_versions` (each job's) and `release_runner` (the image and platform the release runner + resolved for the profile, its image version included); `null` otherwise; - `mode` (`publication`, `validation-only`, or `null` when refused before it was known), `event` (the run's `GITHUB_EVENT_NAME`), `tag_prerelease`, `dispatch` and `revalidation_allowed` (the configured rules); @@ -451,21 +527,24 @@ artifacts, inputs, coverage, the attempt and selection read again, every workflo selection and the tag read at the end), and whose evidence differs only in a `release-runner` profile's recorded runner image or tool identity from what the release runner resolves; the runtime identity is each job's last check, and that difference is raised after the final reads, -so any other refusal wins. It never admits a pin that does not match (under `pins`, or a -`release-runner` profile's own pin, in the evidence or on the release runner), a runtime record of -another shape or platform, another refusal code, or a validation-only dispatch run (each refuses -as before), and it is off unless the configuration says `true`. The decision artifact records the -drift: `revalidation.reason` names each job and the keys that differ, `revalidation.differences` -holds each of those keys' recorded value and the release runner's, `revalidation.jobs` lists the -jobs whose evidence is bound to the release runner, and `refusal` is `null`. The evidence of those -jobs is not taken. The fresh run is the check list `dev.json` declares, its setup steps and every -check, run strictly (`scripts/dev/check.sh --setup --strict`, then `--all --strict`: a step -skipped for an empty `requires_env` fails, `optional: true` or not), on a release runner, and a -publishing job proceeds only on that run's success. Work a CI job does outside the check list (a -step of its own, a matrix leg whose `checks` are empty) is not covered by the fresh run: a -repository whose evidence jobs do such work either runs it in the release workflow and names it in -`fresh_at_release`, or does not opt in. A `pins` leg (another OS or architecture) is not run -again; its evidence has passed every check, its pins included. +so any other refusal wins. It never admits a pin that does not match (under `pins` or `release-os`, +or a `release-runner` profile's own pin, in the evidence or on the release runner), a runtime +record of another shape or platform, a `release-os` job whose OS release or platform is not the +release runner's ([the release-os mode](#the-release-os-mode)), another refusal code, or a +validation-only dispatch run (each refuses as before), and it is off unless the configuration says +`true`. The decision artifact records the drift: `revalidation.reason` names each job and the keys +that differ, `revalidation.differences` holds each of those keys' recorded value and the release +runner's, `revalidation.jobs` lists the jobs whose evidence is bound to the release runner, and +`refusal` is `null`. The evidence of those jobs is not taken. The fresh run is the check list +`dev.json` declares, its setup steps and every check, run strictly +(`scripts/dev/check.sh --setup --strict`, then `--all --strict`: a step skipped for an empty +`requires_env` fails, `optional: true` or not), on a release runner, and a publishing job proceeds +only on that run's success. Work a CI job does outside the check list (a step of its own, a matrix +leg whose `checks` are empty) is not covered by the fresh run: a repository whose evidence jobs do +such work either runs it in the release workflow and names it in `fresh_at_release`, or does not +opt in. A `pins` leg (another OS or architecture) is not run again; its evidence has passed every +check, its pins included. Neither is a `release-os` job: its evidence has passed every check, its +OS release and pins included, and is taken with the work it did, the check list's or not. ## Pull-request-to-branch reuse diff --git a/lib/release_evidence.py b/lib/release_evidence.py index f6b2514..7a94111 100644 --- a/lib/release_evidence.py +++ b/lib/release_evidence.py @@ -49,7 +49,12 @@ # Runs of these events are neither evidence nor part of the newer-attempt scan. OUTSIDE_EVENT_PREFIX = 'pull_request' OUTSIDE_EVENTS = ('merge_group',) -COMPARE = ('release-runner', 'pins') +# How a profile's evidence is compared (docs/ci-evidence.md): with what the release runner resolves +# (release-runner); with the pins alone (pins); with the pins and the OS release and platform the +# release runner resolves, the image version recorded and never compared (release-os). +COMPARE = ('release-runner', 'pins', 'release-os') +# The compare modes under which every tool of the profile is pinned. +PINNED = ('pins', 'release-os') # runner_group_id and runner_group_name of a job on a GitHub-hosted runner. HOSTED = (0, 'GitHub Actions') RUNNER_KEYS = ('ImageOS', 'ImageVersion', 'RUNNER_OS', 'RUNNER_ARCH') @@ -311,6 +316,10 @@ def path(value, where, pattern=PATH): text(value, '%s.platform.%s' % (where, key)) if profile['compare'] not in COMPARE: fail(where + '.compare', 'one of %s is required' % (COMPARE,)) + # "release-os" compares the OS release of the profile's variant, which only the identity + # policy names and records. + if profile['compare'] == 'release-os' and identity is None: + fail(where + '.compare', 'compare "release-os" needs config.identity') named(profile['tools'], where + '.tools', empty=True) for tool, argv in profile['tools'].items(): texts(argv, '%s.tools.%s' % (where, tool), empty=False) @@ -320,8 +329,8 @@ def path(value, where, pattern=PATH): text(pin, '%s.pins.%s' % (where, tool)) if not set(pins) <= set(profile['tools']): fail(where + '.pins', 'every pin names a tool of the profile') - if profile['compare'] == 'pins' and (not pins or set(pins) != set(profile['tools'])): - fail(where + '.pins', 'compare "pins" needs a pin for every tool, and a tool') + if profile['compare'] in PINNED and (not pins or set(pins) != set(profile['tools'])): + fail(where + '.pins', 'compare "%s" needs a pin for every tool, and a tool' % profile['compare']) named(config['workflows'], 'config.workflows') for key, workflow in config['workflows'].items(): @@ -532,8 +541,9 @@ def actions_of(sha, path): def identity_context(config, sha): """The identity policy at the commit: the manifest, held to the digest the configuration pins, - its record, each profile's variant (the profile's platform is the variant's), and the actions - each configured workflow names, every one pinned.""" + its record, each profile's variant (the profile's platform is the variant's, and under + "release-os" one with an OS-release file), and the actions each configured workflow names, + every one pinned.""" setting = config['identity'] raw = git('show', sha + ':' + setting['manifest'], code='identity-config') need(digest(raw) == setting['sha256'], 'identity-config', @@ -546,6 +556,10 @@ def identity_context(config, sha): need(variant is not None, 'identity-config', '%s names no variant of the manifest' % name) need(all(profile['platform'][key] == variant[key] for key in ('RUNNER_OS', 'RUNNER_ARCH')), 'identity-config', '%s: its platform is not the variant %s' % (name, profile['variant'])) + # The release runner cannot resolve an OS release the variant has no file for. + need(profile['compare'] != 'release-os' or variant['os_release'] is not None, 'identity-config', + '%s: compare "release-os" needs an OS-release file, and the variant %s has none' % ( + name, profile['variant'])) return {'record': {'manifest': setting['manifest'], 'sha256': setting['sha256'], 'revision': manifest['revision'], 'kind': manifest['identity']}, 'variants': manifest['variants'], @@ -602,10 +616,10 @@ def check_identity(context, profile, payload, workflow_key, job_key): # --- the evidence a CI job records -------------------------------------------------------------- -def runtime(profile, variant=None): - """This runner's image, platform and tool identity, for a profile. Under the identity policy - (a variant), the image version must have the form a hosted runner reports, and the OS-release - digest is that of the variant's file, on every OS that has one.""" +def runner_runtime(profile, variant=None): + """This runner's image and platform identity, for a profile, without its tools. Under the + identity policy (a variant), the image version must have the form a hosted runner reports, and + the OS-release digest is that of the variant's file, on every OS that has one.""" runner = {key: os.environ.get(key, '') for key in RUNNER_KEYS} need(all(runner.values()), 'runner-identity') for key, value in profile['platform'].items(): @@ -621,6 +635,12 @@ def runtime(profile, variant=None): raise Refusal('identity-missing', 'no OS-release file ' + variant['os_release']) from error elif runner['RUNNER_OS'] == 'Linux': runner['os_release'] = digest(Path('/etc/os-release').read_bytes()) + return runner + + +def runtime(profile, variant=None): + """This runner's image, platform and tool identity, for a profile.""" + runner = runner_runtime(profile, variant) tools = {name: command(argv, 'tool-identity', 65536).decode().strip() for name, argv in profile['tools'].items()} need(all(tools.values()), 'tool-identity') @@ -992,7 +1012,10 @@ def check_runtime(profile, recorded, here, job_key): an image roll). For "release-runner", a record that differs from what this runner resolves is returned as the drift (its detail, and each differing key with its recorded value and this runner's), None when it is the same: verify refuses it only after every other check has - passed.""" + passed. For "release-os", here holds only the runner fields this runner resolves: every one + but ImageVersion, which is recorded and never compared, must be the record's, or the job + refuses here, as a refusal and never a drift (the jobs' work is not the release runner's to + run again).""" need(isinstance(recorded, dict) and set(recorded) == {'runner', 'tools'} and isinstance(recorded['runner'], dict) and isinstance(recorded['tools'], dict), 'runtime-identity', job_key + ': no runtime record') @@ -1019,6 +1042,15 @@ def check_runtime(profile, recorded, here, job_key): for part in ('runner', 'tools')} changed = list(values['runner']) + list(values['tools']) return '%s: differs from this runner in %s' % (job_key, changed), values + if profile['compare'] == 'release-os': + # Every runner field of either side but the image version: a missing or extra field + # differs too. + changed = [key for key in sorted(set(runner) | set(here['runner'])) + if key != 'ImageVersion' and runner.get(key) != here['runner'].get(key)] + need(not changed, 'runtime-identity', + "%s: the OS release or platform is not this runner's (compare release-os, never revalidated): %s" % ( + job_key, '; '.join('%s is %r, this runner %r' % (key, runner.get(key), here['runner'].get(key)) + for key in changed))) return None @@ -1379,7 +1411,8 @@ def verify_workflow(api, config, source, key, now, here, trail, drift, identity= # admitted again now (a refusal, never a drift). reused_trail = readmit(api, config, source, key, job_key, now, run, payload) if reused else None # The runtime identity is the job's last check, so that any other refusal of the job wins - # over it; a drift from this runner is only collected, for the end of verify. + # over it; a drift from this runner is only collected, for the end of verify. A + # "release-os" profile was resolved on this runner before any evidence was read. if profile['compare'] == 'release-runner' and spec['profile'] not in here: here[spec['profile']] = runtime(profile, variant) differs = check_runtime(profile, payload.get('runtime'), here.get(spec['profile']), job_key) @@ -1407,6 +1440,34 @@ def verify_workflow(api, config, source, key, now, here, trail, drift, identity= return resolved['id'], run +def identity_group(compare, variant, name, members, release_runner=None): + """The decision record of one variant and profile's jobs (members: each job and its recorded + runtime), and the identity-changed detail where they disagree, None otherwise. The jobs of a + "pins" profile (a leg the release runner cannot resolve) must have reported one identity + between them. Those of a "release-os" profile must have reported one identity but for + ImageVersion, which is recorded, per job and as this runner resolved it (release_runner), and + never compared. A "release-runner" profile's jobs are each compared with this runner instead, + at the end of verify.""" + same = all(runtime_identity == members[0][1] for _, runtime_identity in members) + group = {'compare': compare, 'jobs': [job for job, _ in members], 'runtime': members[0][1] if same else None} + if compare == 'release-os': + group.update(image_versions={job: runtime_identity['runner'].get('ImageVersion') + for job, runtime_identity in members}, + release_runner=release_runner) + held = [{'runner': {key: value for key, value in runtime_identity['runner'].items() if key != 'ImageVersion'}, + 'tools': runtime_identity['tools']} for _, runtime_identity in members] + differ = sorted({'%s.%s' % (part, key) for other in held for part in ('runner', 'tools') + for key in set(other[part]) | set(held[0][part]) + if other[part].get(key) != held[0][part].get(key)}) + return group, ('the %s jobs of variant %s report different identities apart from ImageVersion: %s' % ( + name, variant, differ) if differ else None) + if same or compare != 'pins': + return group, None + return group, 'the %s jobs of variant %s report different identities: %s' % ( + name, variant, '; '.join('%s %s' % (job, runtime_identity['runner'].get('ImageVersion')) + for job, runtime_identity in members)) + + def runner_bound_jobs(config): """The job keys, in configuration order, whose profile compares with the release runner.""" jobs = [] @@ -1437,23 +1498,24 @@ def verify(api, config, source, release_tag, now, record): statement='a reported hosted-runner image identity, not a pinned VM image', release_workflow=release_workflow_actions(source), groups={}) here, chosen, drift = {}, {}, [] + # A "release-os" profile is resolved on this runner, its image and platform and OS release but + # not its tools, before any evidence is read: a profile this runner cannot resolve (another + # platform or image name, no OS-release file) refuses here, as a configuration fact. + for name, profile in config['profiles'].items(): + if profile['compare'] == 'release-os': + here[name] = {'runner': runner_runtime(profile, identity['context']['variants'][profile['variant']])} for key in config['workflows']: record['workflows'][key] = {} chosen[key] = verify_workflow(api, config, source, key, now, here, record['workflows'][key], drift, identity) if identity: - # Identity is compared per variant and profile, never across them. Jobs of a "pins" profile - # (a leg the release runner cannot resolve) must have reported one identity between them; - # a "release-runner" profile's jobs are each compared with this runner at the end. + # Identity is compared per variant and profile, never across them. for (variant, name), members in sorted(identity['groups'].items()): - same = all(runtime_identity == members[0][1] for _, runtime_identity in members) - record['identity']['groups']['%s/%s' % (variant, name)] = { - 'compare': config['profiles'][name]['compare'], 'jobs': [job for job, _ in members], - 'runtime': members[0][1] if same else None} - need(same or config['profiles'][name]['compare'] != 'pins', 'identity-changed', - 'the %s jobs of variant %s report different identities: %s' % ( - name, variant, '; '.join('%s %s' % (job, runtime_identity['runner'].get('ImageVersion')) - for job, runtime_identity in members))) + compare = config['profiles'][name]['compare'] + group, changed = identity_group(compare, variant, name, members, + here[name]['runner'] if compare == 'release-os' else None) + record['identity']['groups']['%s/%s' % (variant, name)] = group + need(changed is None, 'identity-changed', changed) # Collection takes time: every workflow's selection, then the tag, once more at the end. for key, (workflow_id, run) in chosen.items(): final = select(api, source, config['workflows'][key], workflow_id, branch) diff --git a/scripts/dev/release_evidence.py b/scripts/dev/release_evidence.py index 566947b..c139ae3 100644 --- a/scripts/dev/release_evidence.py +++ b/scripts/dev/release_evidence.py @@ -50,7 +50,12 @@ # Runs of these events are neither evidence nor part of the newer-attempt scan. OUTSIDE_EVENT_PREFIX = 'pull_request' OUTSIDE_EVENTS = ('merge_group',) -COMPARE = ('release-runner', 'pins') +# How a profile's evidence is compared (docs/ci-evidence.md): with what the release runner resolves +# (release-runner); with the pins alone (pins); with the pins and the OS release and platform the +# release runner resolves, the image version recorded and never compared (release-os). +COMPARE = ('release-runner', 'pins', 'release-os') +# The compare modes under which every tool of the profile is pinned. +PINNED = ('pins', 'release-os') # runner_group_id and runner_group_name of a job on a GitHub-hosted runner. HOSTED = (0, 'GitHub Actions') RUNNER_KEYS = ('ImageOS', 'ImageVersion', 'RUNNER_OS', 'RUNNER_ARCH') @@ -312,6 +317,10 @@ def path(value, where, pattern=PATH): text(value, '%s.platform.%s' % (where, key)) if profile['compare'] not in COMPARE: fail(where + '.compare', 'one of %s is required' % (COMPARE,)) + # "release-os" compares the OS release of the profile's variant, which only the identity + # policy names and records. + if profile['compare'] == 'release-os' and identity is None: + fail(where + '.compare', 'compare "release-os" needs config.identity') named(profile['tools'], where + '.tools', empty=True) for tool, argv in profile['tools'].items(): texts(argv, '%s.tools.%s' % (where, tool), empty=False) @@ -321,8 +330,8 @@ def path(value, where, pattern=PATH): text(pin, '%s.pins.%s' % (where, tool)) if not set(pins) <= set(profile['tools']): fail(where + '.pins', 'every pin names a tool of the profile') - if profile['compare'] == 'pins' and (not pins or set(pins) != set(profile['tools'])): - fail(where + '.pins', 'compare "pins" needs a pin for every tool, and a tool') + if profile['compare'] in PINNED and (not pins or set(pins) != set(profile['tools'])): + fail(where + '.pins', 'compare "%s" needs a pin for every tool, and a tool' % profile['compare']) named(config['workflows'], 'config.workflows') for key, workflow in config['workflows'].items(): @@ -533,8 +542,9 @@ def actions_of(sha, path): def identity_context(config, sha): """The identity policy at the commit: the manifest, held to the digest the configuration pins, - its record, each profile's variant (the profile's platform is the variant's), and the actions - each configured workflow names, every one pinned.""" + its record, each profile's variant (the profile's platform is the variant's, and under + "release-os" one with an OS-release file), and the actions each configured workflow names, + every one pinned.""" setting = config['identity'] raw = git('show', sha + ':' + setting['manifest'], code='identity-config') need(digest(raw) == setting['sha256'], 'identity-config', @@ -547,6 +557,10 @@ def identity_context(config, sha): need(variant is not None, 'identity-config', '%s names no variant of the manifest' % name) need(all(profile['platform'][key] == variant[key] for key in ('RUNNER_OS', 'RUNNER_ARCH')), 'identity-config', '%s: its platform is not the variant %s' % (name, profile['variant'])) + # The release runner cannot resolve an OS release the variant has no file for. + need(profile['compare'] != 'release-os' or variant['os_release'] is not None, 'identity-config', + '%s: compare "release-os" needs an OS-release file, and the variant %s has none' % ( + name, profile['variant'])) return {'record': {'manifest': setting['manifest'], 'sha256': setting['sha256'], 'revision': manifest['revision'], 'kind': manifest['identity']}, 'variants': manifest['variants'], @@ -603,10 +617,10 @@ def check_identity(context, profile, payload, workflow_key, job_key): # --- the evidence a CI job records -------------------------------------------------------------- -def runtime(profile, variant=None): - """This runner's image, platform and tool identity, for a profile. Under the identity policy - (a variant), the image version must have the form a hosted runner reports, and the OS-release - digest is that of the variant's file, on every OS that has one.""" +def runner_runtime(profile, variant=None): + """This runner's image and platform identity, for a profile, without its tools. Under the + identity policy (a variant), the image version must have the form a hosted runner reports, and + the OS-release digest is that of the variant's file, on every OS that has one.""" runner = {key: os.environ.get(key, '') for key in RUNNER_KEYS} need(all(runner.values()), 'runner-identity') for key, value in profile['platform'].items(): @@ -622,6 +636,12 @@ def runtime(profile, variant=None): raise Refusal('identity-missing', 'no OS-release file ' + variant['os_release']) from error elif runner['RUNNER_OS'] == 'Linux': runner['os_release'] = digest(Path('/etc/os-release').read_bytes()) + return runner + + +def runtime(profile, variant=None): + """This runner's image, platform and tool identity, for a profile.""" + runner = runner_runtime(profile, variant) tools = {name: command(argv, 'tool-identity', 65536).decode().strip() for name, argv in profile['tools'].items()} need(all(tools.values()), 'tool-identity') @@ -993,7 +1013,10 @@ def check_runtime(profile, recorded, here, job_key): an image roll). For "release-runner", a record that differs from what this runner resolves is returned as the drift (its detail, and each differing key with its recorded value and this runner's), None when it is the same: verify refuses it only after every other check has - passed.""" + passed. For "release-os", here holds only the runner fields this runner resolves: every one + but ImageVersion, which is recorded and never compared, must be the record's, or the job + refuses here, as a refusal and never a drift (the jobs' work is not the release runner's to + run again).""" need(isinstance(recorded, dict) and set(recorded) == {'runner', 'tools'} and isinstance(recorded['runner'], dict) and isinstance(recorded['tools'], dict), 'runtime-identity', job_key + ': no runtime record') @@ -1020,6 +1043,15 @@ def check_runtime(profile, recorded, here, job_key): for part in ('runner', 'tools')} changed = list(values['runner']) + list(values['tools']) return '%s: differs from this runner in %s' % (job_key, changed), values + if profile['compare'] == 'release-os': + # Every runner field of either side but the image version: a missing or extra field + # differs too. + changed = [key for key in sorted(set(runner) | set(here['runner'])) + if key != 'ImageVersion' and runner.get(key) != here['runner'].get(key)] + need(not changed, 'runtime-identity', + "%s: the OS release or platform is not this runner's (compare release-os, never revalidated): %s" % ( + job_key, '; '.join('%s is %r, this runner %r' % (key, runner.get(key), here['runner'].get(key)) + for key in changed))) return None @@ -1380,7 +1412,8 @@ def verify_workflow(api, config, source, key, now, here, trail, drift, identity= # admitted again now (a refusal, never a drift). reused_trail = readmit(api, config, source, key, job_key, now, run, payload) if reused else None # The runtime identity is the job's last check, so that any other refusal of the job wins - # over it; a drift from this runner is only collected, for the end of verify. + # over it; a drift from this runner is only collected, for the end of verify. A + # "release-os" profile was resolved on this runner before any evidence was read. if profile['compare'] == 'release-runner' and spec['profile'] not in here: here[spec['profile']] = runtime(profile, variant) differs = check_runtime(profile, payload.get('runtime'), here.get(spec['profile']), job_key) @@ -1408,6 +1441,34 @@ def verify_workflow(api, config, source, key, now, here, trail, drift, identity= return resolved['id'], run +def identity_group(compare, variant, name, members, release_runner=None): + """The decision record of one variant and profile's jobs (members: each job and its recorded + runtime), and the identity-changed detail where they disagree, None otherwise. The jobs of a + "pins" profile (a leg the release runner cannot resolve) must have reported one identity + between them. Those of a "release-os" profile must have reported one identity but for + ImageVersion, which is recorded, per job and as this runner resolved it (release_runner), and + never compared. A "release-runner" profile's jobs are each compared with this runner instead, + at the end of verify.""" + same = all(runtime_identity == members[0][1] for _, runtime_identity in members) + group = {'compare': compare, 'jobs': [job for job, _ in members], 'runtime': members[0][1] if same else None} + if compare == 'release-os': + group.update(image_versions={job: runtime_identity['runner'].get('ImageVersion') + for job, runtime_identity in members}, + release_runner=release_runner) + held = [{'runner': {key: value for key, value in runtime_identity['runner'].items() if key != 'ImageVersion'}, + 'tools': runtime_identity['tools']} for _, runtime_identity in members] + differ = sorted({'%s.%s' % (part, key) for other in held for part in ('runner', 'tools') + for key in set(other[part]) | set(held[0][part]) + if other[part].get(key) != held[0][part].get(key)}) + return group, ('the %s jobs of variant %s report different identities apart from ImageVersion: %s' % ( + name, variant, differ) if differ else None) + if same or compare != 'pins': + return group, None + return group, 'the %s jobs of variant %s report different identities: %s' % ( + name, variant, '; '.join('%s %s' % (job, runtime_identity['runner'].get('ImageVersion')) + for job, runtime_identity in members)) + + def runner_bound_jobs(config): """The job keys, in configuration order, whose profile compares with the release runner.""" jobs = [] @@ -1438,23 +1499,24 @@ def verify(api, config, source, release_tag, now, record): statement='a reported hosted-runner image identity, not a pinned VM image', release_workflow=release_workflow_actions(source), groups={}) here, chosen, drift = {}, {}, [] + # A "release-os" profile is resolved on this runner, its image and platform and OS release but + # not its tools, before any evidence is read: a profile this runner cannot resolve (another + # platform or image name, no OS-release file) refuses here, as a configuration fact. + for name, profile in config['profiles'].items(): + if profile['compare'] == 'release-os': + here[name] = {'runner': runner_runtime(profile, identity['context']['variants'][profile['variant']])} for key in config['workflows']: record['workflows'][key] = {} chosen[key] = verify_workflow(api, config, source, key, now, here, record['workflows'][key], drift, identity) if identity: - # Identity is compared per variant and profile, never across them. Jobs of a "pins" profile - # (a leg the release runner cannot resolve) must have reported one identity between them; - # a "release-runner" profile's jobs are each compared with this runner at the end. + # Identity is compared per variant and profile, never across them. for (variant, name), members in sorted(identity['groups'].items()): - same = all(runtime_identity == members[0][1] for _, runtime_identity in members) - record['identity']['groups']['%s/%s' % (variant, name)] = { - 'compare': config['profiles'][name]['compare'], 'jobs': [job for job, _ in members], - 'runtime': members[0][1] if same else None} - need(same or config['profiles'][name]['compare'] != 'pins', 'identity-changed', - 'the %s jobs of variant %s report different identities: %s' % ( - name, variant, '; '.join('%s %s' % (job, runtime_identity['runner'].get('ImageVersion')) - for job, runtime_identity in members))) + compare = config['profiles'][name]['compare'] + group, changed = identity_group(compare, variant, name, members, + here[name]['runner'] if compare == 'release-os' else None) + record['identity']['groups']['%s/%s' % (variant, name)] = group + need(changed is None, 'identity-changed', changed) # Collection takes time: every workflow's selection, then the tag, once more at the end. for key, (workflow_id, run) in chosen.items(): final = select(api, source, config['workflows'][key], workflow_id, branch) diff --git a/templates/ci-evidence.json b/templates/ci-evidence.json index b86ce76..7260bc0 100644 --- a/templates/ci-evidence.json +++ b/templates/ci-evidence.json @@ -51,7 +51,10 @@ "linux": { "$comment": [ "compare release-runner: the release job, on the same platform, resolves the image and the tools itself;", - "a CI image or tool that differs from what it resolves refuses. The release job must run on this platform." + "a CI image or tool that differs from what it resolves refuses. The release job must run on this platform.", + "compare release-os (under identity only) is a weaker, explicit choice for jobs that cannot all land on one", + "image: every tool pinned, the image name, platform and OS release held to what the release job resolves,", + "the image version recorded and never compared (docs/ci-evidence.md#the-release-os-mode)." ], "platform": { "RUNNER_OS": "Linux", diff --git a/tests/release_evidence_test.py b/tests/release_evidence_test.py index dd8896e..e08c627 100644 --- a/tests/release_evidence_test.py +++ b/tests/release_evidence_test.py @@ -2182,6 +2182,339 @@ def test_capture_refuses_an_unexpected_identity(self): self.assertIn('capture refused: ' + code, result.stderr) +# --- compare "release-os" (devkit #79): the OS release and exact tools, not the image version ----- + +OS_RELEASE = b'PRETTY_NAME="Fixture Linux 24.04.3 LTS"\nVERSION_ID="24.04"\n' +OS_RELEASE_SHA = hashlib.sha256(OS_RELEASE).hexdigest() +# Three image versions one hosted label served at once during a staged roll (judgment-pack-agent#332). +ROLL = ('20261025.179.1', '20261101.186.1', '20261018.50.1') +TABLE = {'platform': {'RUNNER_OS': 'Linux', 'RUNNER_ARCH': 'X64', 'ImageOS': 'ubuntu24'}, 'variant': 'linux-x64', + 'compare': 'release-os', 'tools': {'go': ['go', 'version']}, + 'pins': {'go': 'go version go1.26.5 linux/amd64'}} +RELEASE_RUNNER = {'ImageOS': 'ubuntu24', 'ImageVersion': ROLL[0], 'RUNNER_OS': 'Linux', 'RUNNER_ARCH': 'X64'} + + +class ReleaseOS(Gate): + """decide() under the identity policy with a sharded table (table-1..3, one per image version of + ROLL) under compare "release-os", beside the Linux release-runner job and the macOS pins leg. + The release runner resolves the table's profile for real, from its environment and an + OS-release file of the test's own; the Linux job's profile is resolved by the fixture (self.here), + and both report one release runner.""" + + def setUp(self): + super().setUp() + self.config = identity_config(self.config) + self.config['profiles']['table'] = copy.deepcopy(TABLE) + scratch = tempfile.TemporaryDirectory(prefix='release-evidence-release-os-') + self.addCleanup(scratch.cleanup) + self.os_release = Path(scratch.name) / 'os-release' + self.os_release.write_bytes(OS_RELEASE) + variants = e.parse_identity_manifest(MANIFEST_BYTES)['variants'] + variants['linux-x64']['os_release'] = str(self.os_release) + for name, target in (('identity_context', lambda config, sha: { + 'record': dict(IDENTITY_RECORD), 'variants': copy.deepcopy(variants), + 'actions': {'ci': [CHECKOUT, UPLOAD]}}), + ('release_workflow_actions', lambda source: { + 'path': '.github/workflows/release.yml', 'actions': [CHECKOUT]})): + patcher = patch.object(e, name, side_effect=target) + patcher.start() + self.addCleanup(patcher.stop) + environment = patch.dict(os.environ, RELEASE_RUNNER) + environment.start() + self.addCleanup(environment.stop) + self.here['runner'].update(ImageVersion=ROLL[0], os_release=OS_RELEASE_SHA) + self.evidence(0, lambda payload: (payload.update( + identity=dict(IDENTITY_RECORD, variant='linux-x64', actions=[CHECKOUT, UPLOAD])), + payload['runtime']['runner'].update(ImageVersion=ROLL[0], os_release=OS_RELEASE_SHA))) + self.evidence(1, lambda payload: (payload.update( + identity=dict(IDENTITY_RECORD, variant='macos-arm64', actions=[CHECKOUT, UPLOAD])), + payload['runtime']['runner'].update(os_release='e' * 64))) + for index, version in enumerate(ROLL, 1): + key = 'table-%d' % index + self.config['workflows']['ci']['jobs'][key] = { + 'name': 'table (%d)' % index, 'job_id': 'table', 'labels': ['ubuntu-latest'], 'profile': 'table', + 'checks': ['unit']} + self.set_jobs(self.jobs + [dict(JOB_LINUX, id=210 + index, name='table (%d)' % index)]) + self.set_artifacts(self.artifacts + [dict(ARTIFACT_LINUX, id=310 + index, name='ci-evidence-ci-%s-1' % key)]) + payload = copy.deepcopy(EVIDENCE_LINUX) + payload.update(job=key, coverage={'unit': 'PASS'}, + identity=dict(IDENTITY_RECORD, variant='linux-x64', actions=[CHECKOUT, UPLOAD])) + payload['runtime']['runner'].update(ImageVersion=version, os_release=OS_RELEASE_SHA) + pack(self.data, self.artifacts[-1], payload) + + def shard(self, index, change): + """Change table-'s evidence (artifacts 2, 3 and 4 are table-1, -2 and -3).""" + self.evidence(index + 1, change) + + def release_runner(self, **changes): + """The release runner reports changes, to the table's resolution and the Linux job's.""" + os.environ.update(changes) + self.here['runner'].update(changes) + + def test_shards_on_several_image_versions_are_reused_with_every_version_recorded(self): + self.assertEqual(self.decide(), 'reuse') + self.assertEqual(self.record['identity']['groups']['linux-x64/table'], { + 'compare': 'release-os', 'jobs': ['ci/table-1', 'ci/table-2', 'ci/table-3'], 'runtime': None, + 'image_versions': {'ci/table-1': ROLL[0], 'ci/table-2': ROLL[1], 'ci/table-3': ROLL[2]}, + 'release_runner': dict(RELEASE_RUNNER, os_release=OS_RELEASE_SHA)}) + jobs = self.record['workflows']['ci']['jobs'] + self.assertEqual([jobs['table-%d' % index]['runtime']['runner']['ImageVersion'] for index in (1, 2, 3)], + list(ROLL)) + # The other modes' groups keep their record. + for group in ('linux-x64/linux', 'macos-arm64/macos'): + self.assertEqual(set(self.record['identity']['groups'][group]), {'compare', 'jobs', 'runtime'}) + # The release runner resolved the table's image, platform and OS release, not its tools: only + # the release-runner profile was resolved with its tools. + self.assertEqual([call.args[0]['compare'] for call in e.runtime.call_args_list], ['release-runner']) + # A release runner on yet another image version, all shards on one: the same; and no shard is a + # drift, so an allowed revalidation changes nothing. + self.setUp() + for index in (1, 2, 3): + self.shard(index, lambda payload: payload['runtime']['runner'].update(ImageVersion=ROLL[2])) + os.environ['ImageVersion'] = '20261102.190.1' + self.config['revalidation'] = {'allowed': True} + self.assertEqual(self.decide(), 'reuse') + group = self.record['identity']['groups']['linux-x64/table'] + self.assertEqual((group['runtime']['runner']['ImageVersion'], group['release_runner']['ImageVersion']), + (ROLL[2], '20261102.190.1')) + + def test_another_os_release_refuses_and_is_never_revalidated(self): + other = 'd' * 64 + for allowed in (False, True): + with self.subTest(revalidation=allowed): + self.setUp() + self.config['revalidation'] = {'allowed': allowed} + self.shard(2, lambda payload: payload['runtime']['runner'].update(os_release=other)) + self.assertEqual(self.decide(), 'runtime-identity') + self.assertEqual(self.record['refusal']['detail'], + "table-2: the OS release or platform is not this runner's (compare release-os, " + "never revalidated): os_release is %r, this runner %r" % (other, OS_RELEASE_SHA)) + # The release runner's OS release moved: the first shard refuses at once, and the Linux job's + # drift, which an allowed revalidation would otherwise take, does not turn the refusal. + self.setUp() + self.config['revalidation'] = {'allowed': True} + moved = OS_RELEASE + b'VERSION="24.04.4 LTS (Noble Numbat)"\n' + self.os_release.write_bytes(moved) + self.here['runner']['os_release'] = hashlib.sha256(moved).hexdigest() + self.assertEqual(self.decide(), 'runtime-identity') + self.assertTrue(self.record['refusal']['detail'].startswith( + "table-1: the OS release or platform is not this runner's"), self.record['refusal']['detail']) + + def test_a_tool_that_is_not_its_pin_or_another_tool_set_refuses(self): + self.config['revalidation'] = {'allowed': True} + self.shard(3, lambda payload: payload['runtime']['tools'].update(go='go version go1.26.6 linux/amd64')) + self.assertEqual(self.decide(), 'runtime-identity') + self.assertEqual(self.record['refusal']['detail'], + "table-3: go is 'go version go1.26.6 linux/amd64', pinned 'go version go1.26.5 linux/amd64'") + self.setUp() + self.shard(1, lambda payload: payload['runtime']['tools'].update(node='v24.11.0')) + self.assertEqual(self.decide(), 'runtime-identity') + self.assertEqual(self.record['refusal']['detail'], 'table-1: tool set') + + def test_a_platform_that_differs_refuses(self): + # The release runner is not the profile's platform or image: refused before any run is read. + for change, refusal in ( + ({'RUNNER_ARCH': 'ARM64'}, 'runner-platform: RUNNER_ARCH is ARM64, not X64'), + ({'RUNNER_OS': 'macOS'}, 'runner-platform: RUNNER_OS is macOS, not Linux'), + ({'ImageOS': 'ubuntu26'}, 'runner-platform: ImageOS is ubuntu26, not ubuntu24'), + ({'ImageVersion': 'latest'}, "identity-unexpected: ImageVersion 'latest' is not a reported image version")): + with self.subTest(change=change): + self.setUp() + self.release_runner(**change) + code = self.decide() + self.assertEqual('%s: %s' % (code, self.record['refusal']['detail']), refusal) + self.assertNotIn(RUNS, self.api.calls) + self.setUp() + self.os_release.unlink() + self.assertEqual(self.decide(), 'identity-missing') + self.assertNotIn(RUNS, self.api.calls) + # A shard that reports another platform or image name refuses under the identity policy. + for change in ({'ImageOS': 'ubuntu26'}, {'RUNNER_ARCH': 'ARM64'}, {'ImageVersion': 'latest'}): + with self.subTest(shard=change): + self.setUp() + self.shard(2, lambda payload: payload['runtime']['runner'].update(change)) + self.assertEqual(self.decide(), 'identity-unexpected') + + def test_the_comparison_holds_every_runner_field_but_the_image_version(self): + profile, recorded = copy.deepcopy(TABLE), dict(copy.deepcopy(LINUX)) + here = {'runner': dict(LINUX['runner'], ImageVersion=ROLL[1])} + self.assertEqual(outcome_of(lambda: e.check_runtime(profile, recorded, here, 'table-1')), 'accepted') + for key, value in (('ImageOS', 'ubuntu26'), ('RUNNER_OS', 'macOS'), ('RUNNER_ARCH', 'ARM64'), + ('os_release', 'd' * 64), ('ImageName', 'ubuntu-24.04'), ('os_release', None)): + with self.subTest(key=key, value=value): + other = {'runner': dict(here['runner'], **{key: value})} + if value is None: + other['runner'].pop(key) + refusal = outcome_of(lambda: e.check_runtime(profile, recorded, other, 'table-1')) + self.assertEqual(refusal, "runtime-identity: table-1: the OS release or platform is not this runner's " + "(compare release-os, never revalidated): %s is %r, this runner %r" % ( + key, recorded['runner'].get(key), value)) + + def test_the_group_holds_every_field_but_the_image_version(self): + def member(index, **runner): + runtime_identity = copy.deepcopy(LINUX) + runtime_identity['runner'].update(runner) + return 'ci/table-%d' % index, runtime_identity + mixed = [member(1, ImageVersion=ROLL[0]), member(2, ImageVersion=ROLL[1])] + group, changed = e.identity_group('release-os', 'linux-x64', 'table', mixed, {'ImageVersion': ROLL[2]}) + self.assertIsNone(changed) + self.assertEqual(group, {'compare': 'release-os', 'jobs': ['ci/table-1', 'ci/table-2'], 'runtime': None, + 'image_versions': {'ci/table-1': ROLL[0], 'ci/table-2': ROLL[1]}, + 'release_runner': {'ImageVersion': ROLL[2]}}) + for change, field in (({'os_release': 'd' * 64}, 'runner.os_release'), ({'ImageOS': 'ubuntu26'}, 'runner.ImageOS'), + ({'RUNNER_ARCH': 'ARM64'}, 'runner.RUNNER_ARCH')): + with self.subTest(field=field): + members = [member(1, ImageVersion=ROLL[0]), member(2, ImageVersion=ROLL[1], **change)] + self.assertEqual(e.identity_group('release-os', 'linux-x64', 'table', members)[1], + 'the table jobs of variant linux-x64 report different identities apart from ' + "ImageVersion: ['%s']" % field) + members = [member(1), member(2)] + members[1][1]['tools']['go'] = 'go version go1.26.6 linux/amd64' + self.assertEqual(e.identity_group('release-os', 'linux-x64', 'table', members)[1], + "the table jobs of variant linux-x64 report different identities apart from ImageVersion: " + "['tools.go']") + # "pins" holds the image version too, in the words it had; "release-runner" is compared with + # the release runner instead, never here. + self.assertEqual(e.identity_group('pins', 'linux-x64', 'table', mixed), + ({'compare': 'pins', 'jobs': ['ci/table-1', 'ci/table-2'], 'runtime': None}, + 'the table jobs of variant linux-x64 report different identities: ' + 'ci/table-1 20261025.179.1; ci/table-2 20261101.186.1')) + self.assertEqual(e.identity_group('release-runner', 'linux-x64', 'table', mixed), + ({'compare': 'release-runner', 'jobs': ['ci/table-1', 'ci/table-2'], 'runtime': None}, None)) + + def test_pins_and_release_runner_still_refuse_the_same_roll(self): + # "pins": the shards must report one identity between them, image version included. + self.config['revalidation'] = {'allowed': True} + self.config['profiles']['table']['compare'] = 'pins' + self.assertEqual(self.decide(), 'identity-changed') + self.assertEqual(self.record['refusal']['detail'], + 'the table jobs of variant linux-x64 report different identities: ci/table-1 20261025.179.1; ' + 'ci/table-2 20261101.186.1; ci/table-3 20261018.50.1') + self.assertEqual(self.record['identity']['groups']['linux-x64/table'], + {'compare': 'pins', 'jobs': ['ci/table-1', 'ci/table-2', 'ci/table-3'], 'runtime': None}) + # "release-runner": each shard must be what the release runner reports; the two on other images + # drift, which refuses, or is revalidated where that is allowed. + self.setUp() + self.config['profiles']['table']['compare'] = 'release-runner' + self.assertEqual(self.decide(), 'runtime-identity') + self.assertEqual(self.record['refusal']['detail'], "table-2: differs from this runner in ['ImageVersion']; " + "table-3: differs from this runner in ['ImageVersion']") + self.config['revalidation'] = {'allowed': True} + self.assertEqual(self.decide(), 'revalidate') + self.assertEqual(self.record['revalidation']['jobs'], ['test-linux', 'table-1', 'table-2', 'table-3']) + + def test_a_revalidation_takes_the_shards_evidence(self): + # The release runner drew a newer image than main CI: the release-runner job drifts and is run + # again; the shards hold the OS release and pins, so their evidence is taken, as a pins leg's. + self.config['revalidation'] = {'allowed': True} + self.release_runner(ImageVersion='20261102.190.1') + self.assertEqual(self.decide(), 'revalidate') + self.assertEqual(self.record['revalidation']['jobs'], ['test-linux']) + self.assertEqual(self.record['revalidation']['reason'], "test-linux: differs from this runner in ['ImageVersion']") + self.assertEqual(self.record['identity']['groups']['linux-x64/table']['release_runner']['ImageVersion'], + '20261102.190.1') + + +class ReleaseOSConfiguration(unittest.TestCase): + def config(self, **table): + config = identity_config(CONFIG) + config['profiles']['table'] = dict(copy.deepcopy(TABLE), **table) + return config + + def test_release_os_needs_the_identity_policy_and_every_tool_pinned(self): + self.assertEqual(outcome_of(lambda: e.check_config(self.config())), 'accepted') + # No identity policy, no variant: no OS release to compare. + plain = copy.deepcopy(CONFIG) + plain['profiles']['table'] = {key: value for key, value in TABLE.items() if key != 'variant'} + self.assertEqual(outcome_of(lambda: e.check_config(plain)), + 'config-invalid: config.profiles.table.compare: compare "release-os" needs config.identity') + for name, config in (('another tool', self.config(tools={'go': ['go', 'version'], 'node': ['node', '--version']})), + ('no pins', self.config(pins={}))): + with self.subTest(case=name): + self.assertEqual(outcome_of(lambda: e.check_config(config)), + 'config-invalid: config.profiles.table.pins: compare "release-os" needs a pin for ' + 'every tool, and a tool') + config = self.config() + config['profiles']['table'].pop('pins') + self.assertTrue(outcome_of(lambda: e.check_config(config)).startswith('config-invalid: config.profiles.table.pins: ')) + # The other modes keep their rules and words. + config = self.config() + config['profiles']['macos']['tools']['cc'] = ['cc', '--version'] + self.assertEqual(outcome_of(lambda: e.check_config(config)), + 'config-invalid: config.profiles.macos.pins: compare "pins" needs a pin for every tool, and a tool') + config = self.config() + config['profiles']['linux'].pop('pins') + self.assertEqual(outcome_of(lambda: e.check_config(config)), 'accepted') + self.assertEqual(outcome_of(lambda: e.check_config(self.config(compare='release-image'))), + "config-invalid: config.profiles.table.compare: one of ('release-runner', 'pins', 'release-os') " + "is required") + + def test_a_variant_without_an_os_release_file_refuses(self): + scratch = tempfile.TemporaryDirectory(prefix='release-evidence-release-os-') + self.addCleanup(scratch.cleanup) + sha = git_fixture(Path(scratch.name), { + '.github/runner-identity.json': MANIFEST_BYTES, + '.github/workflows/ci.yml': 'name: CI\non: push\njobs:\n test:\n steps:\n - uses: %s\n' % CHECKOUT}) + cwd = os.getcwd() + os.chdir(scratch.name) + self.addCleanup(os.chdir, cwd) + windows = {'platform': {'RUNNER_OS': 'Windows', 'RUNNER_ARCH': 'X64', 'ImageOS': 'win25'}, 'variant': 'windows-x64', + 'compare': 'pins', 'tools': {'go': ['go', 'version']}, 'pins': {'go': 'go version go1.26.5 windows/amd64'}} + config = self.config() + config['profiles']['windows'] = windows + self.assertEqual(outcome_of(lambda: e.check_config(config)), 'accepted') + self.assertEqual(outcome_of(lambda: e.identity_context(config, sha)), 'accepted') + windows['compare'] = 'release-os' + self.assertEqual(outcome_of(lambda: e.check_config(config)), 'accepted') + self.assertEqual(outcome_of(lambda: e.identity_context(config, sha)), + 'identity-config: windows: compare "release-os" needs an OS-release file, and the variant ' + 'windows-x64 has none') + + +class ReleaseOSCLI(unittest.TestCase): + """capture and verify under compare "release-os", as the workflows run them: IdentityCLI's + fixture repository, its profile changed in a second commit.""" + + capture, verify = IdentityCLI.capture, IdentityCLI.verify + + def setUp(self): + IdentityCLI.setUp(self) + path = self.root / '.github/ci-evidence.json' + config = json.loads(path.read_text()) + config['profiles']['kit'].update(compare='release-os', tools={'tool': [sys.executable, '-c', 'print("tool 1.0")']}, + pins={'tool': 'tool 1.0'}) + path.write_text(json.dumps(config, indent=2) + '\n') + subprocess.run(['git', '-c', 'user.name=Fixture', '-c', 'user.email=fixture@example.invalid', 'commit', '-qam', + 'release-os'], cwd=self.root, env=self.env, check=True, stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL) + self.sha = subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=self.root, text=True, + stdin=subprocess.DEVNULL).strip() + self.env.update(GITHUB_SHA=self.sha, GITHUB_WORKFLOW_SHA=self.sha) + + def test_evidence_of_one_image_version_is_taken_on_another(self): + result, output = self.capture(ImageVersion=ROLL[2]) + self.assertEqual(result.returncode, 0, result.stderr) + captured = json.loads(output.read_text()) + self.assertEqual(captured['runtime'], { + 'runner': {'ImageOS': 'ubuntu24', 'ImageVersion': ROLL[2], 'RUNNER_OS': 'Linux', 'RUNNER_ARCH': 'X64', + 'os_release': self.os_release}, + 'tools': {'tool': 'tool 1.0'}}) + result, record = self.verify(captured, ImageVersion=ROLL[1]) + self.assertEqual((result.returncode, record['decision']), (0, 'reuse'), result.stderr) + group = record['identity']['groups']['linux-x64/kit'] + self.assertEqual((group['compare'], group['image_versions'], group['release_runner']['ImageVersion']), + ('release-os', {'ci/test': ROLL[2]}, ROLL[1])) + # Evidence of another OS release refuses. + runner = dict(captured['runtime']['runner'], os_release='d' * 64) + result, record = self.verify(dict(captured, runtime=dict(captured['runtime'], runner=runner)), ImageVersion=ROLL[1]) + self.assertEqual((result.returncode, record['decision'], (record['refusal'] or {}).get('code')), + (1, 'refuse', 'runtime-identity')) + # A release runner of another image name refuses before any evidence is read. + result, record = self.verify(captured, ImageOS='ubuntu26') + self.assertEqual((result.returncode, (record['refusal'] or {}).get('code'), record['workflows']), + (1, 'runner-platform', {})) + + # --- pull-request-to-branch reuse (devkit #60): tree equality alone never qualifies a result ------ BASE = 'e' * 40 # the branch commit the pull request was tested on, and the commit's first parent