Skip to content

packaging(ns8): read manifest arches from the registry, not local storage - #148

Merged
shockalotti merged 3 commits into
mainfrom
fix/ns8-manifest-registry
Oct 16, 2026
Merged

shockalotti merged 3 commits into
mainfrom
fix/ns8-manifest-registry

Conversation

@shockalotti

Copy link
Copy Markdown
Contributor

Fixes the failure that stopped the v0.11.0 module release. :latest is currently an amd64-only manifest, which works on amd64 and hard-fails on arm64 nodes.

The failure

buildah manifest add was handed a plain local reference, so it read each architecture out of local containers-storage. On the release runner buildah had two graph roots merged into one store, and the local image record existed while the manifest blob it pointed at did not:

Error: initializing source containers-storage:[overlay@/home/runner/.local/share/containers/storage+/run/user/1001/containers]...:
reading manifest for instance "sha256:43c2abfb...": ... file does not exist

Why the existing guard missed it

It asked buildah images -q, which inspects the same local store. So it answered "present" for an image whose manifest could not be read, and the script failed later and far more cryptically. That guard is removed: a failed add is now the check, and it is authoritative. It would also have produced the opposite false answer, reporting a published image as absent, since it never looks at the registry at all.

The fix

Read each arch from the registry, where build-images.sh has already pushed it. This is the documented form, not a workaround. The example in buildah-manifest-add(1) is:

buildah manifest add mylist:v1.11 docker://fedora

Side benefit: the existing refusal to publish a partial list now inspects the content that actually ships, rather than whatever happens to be in local storage.

SOURCE_PREFIX defaults to docker:// and is overridden only by the hermetic self-test. It uses ${x-default} rather than ${x:-default} deliberately: an empty value is a real request to read locally, and ${x:-} would silently replace it.

CI now covers the path that broke

The old self-test only exercised the local-read path, which is precisely why this shipped. It now runs twice:

  1. Hermetic - unchanged, builds scratch images locally, SOURCE_PREFIX="", pushes to oci:.
  2. Registry-backed - assembles from the real published :build-<arch> tags and pushes to a local oci: path. No credentials, publishes nothing. Asserts the assembled index.json really holds two arches rather than trusting the exit code.

Verified locally

bash -n clean, YAML parses, both selftest steps present, ${SOURCE_PREFIX-docker://} confirmed to yield docker:// when unset and empty when explicitly empty, and the buildah subcommand set the CI validator extracts is unchanged. The two-arch registry assembly itself needs a real buildah, so CI is where that gets exercised for real.

Not in this PR

The post-publish arch assertion and the README :latest/arm64 correction were noted as already being handled elsewhere, so I left both alone to avoid colliding. This PR is only the assembly fix and the self-test changes it requires.

…rage

This is the failure that stopped the v0.11.0 module release.

buildah manifest add was given a plain local reference, so it read each
architecture out of local containers-storage. On the release runner buildah
had two graph roots merged into a single store, and the local image record
existed while the manifest blob it pointed at did not:

  Error: initializing source containers-storage:[overlay@/home/runner/.local/
  share/containers/storage+/run/user/1001/containers]...: reading manifest for
  instance "sha256:43c2abfb...": ... file does not exist

The old pre-check missed it. It asked `buildah images -q`, which inspects the
same local store, so it answered "present" for an image whose manifest could
not be read, and the script failed later and far more cryptically. That guard
is gone: a failed add is now the check, and it is authoritative.

The fix is to read each arch from the registry, where build-images.sh has
already pushed it. This is the documented form: the example in
buildah-manifest-add(1) is `buildah manifest add mylist:v1.11 docker://fedora`.
It also means the existing refusal to publish a partial list now inspects the
content that will actually ship.

SOURCE_PREFIX defaults to docker:// and is only overridden by the hermetic
self-test. It uses ${x-default} rather than ${x:-default} on purpose: an empty
value is a deliberate request to read locally, and ${x:-} would silently
replace it.

The CI self-test previously covered only the local-read path, which is exactly
why this shipped broken. It now runs twice: the existing hermetic test, and a
new registry-backed test that assembles from the real published :build-<arch>
tags and pushes to a local oci: path, so it needs no credentials, publishes
nothing, and asserts the assembled index.json really holds two arches.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

The registry fix itself checks out — I reproduced the new step locally against the real published :build-* tags and the assembly succeeds on buildah 1.33.7. But both new CI paths fail on every run as written: the validate job's subcommand scan now extracts buildah ran from a new comment and exits 1 before any selftest starts, and the registry selftest counts arches in a file that structurally cannot hold them. Two small fixes, both inline.

Reviewed changes

  • push-manifest.sh reads each per-arch image from the registry (docker://<repo>:build-<arch>) instead of local containers-storage, so the buildah manifest add failure that broke the v0.11.0 release cannot recur; the unreliable buildah images -q pre-check is gone and a failed add is now the check.
  • Adds SOURCE_PREFIX (default docker://, empty string = local), deliberately using ${x-default} so the hermetic self-test can ask for a local read.
  • .github/workflows/ns8-module.yml: the existing selftest is renamed "(hermetic)" and passes SOURCE_PREFIX="" explicitly; a second selftest assembles from the published ghcr.io/calnode/ns8-calnode:build-{amd64,arm64} tags and pushes to a local oci: path, so it needs no credentials and publishes nothing.

For the record, the premise holds: ghcr.io/calnode/ns8-calnode:latest currently serves a single application/vnd.oci.image.manifest.v1+json (amd64 only), while :build-amd64 / :build-arm64 both exist and are anonymously pullable, and the per-arch layers are ~2.2 MiB, so pulling them on every PR costs seconds, not minutes. The README.md local-build recipe already does PUSH=1 on both arches, so it stays correct under the registry read.

ℹ️ Reading :build-<arch> trades local isolation for a shared, mutable tag

The old local read was immune to other runs: each runner's containers-storage held only its own freshly built arches. Reading from the registry means the assembled list is now assembled from whatever :build-amd64 / :build-arm64 point at at that moment, and those tags are rewritten by every branch and release run — concurrency only groups by ref, so a main run and a v* run can overlap and interleave. Narrow, and cancel-in-progress: true closes most of the window, but if you want the guarantee closed, per-run unique build tags (build-<sha>-<arch>) with the public list still assembled from the digests this run pushed would do it. Flagging as a design call, not a blocker.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using space-bunny-free (free) | 𝕏

Comment thread packaging/ns8/push-manifest.sh Resolved
Comment thread .github/workflows/ns8-module.yml Resolved
The validate job greps the build scripts for 'buildah [a-z]+' and asserts
each extracted word is a real subcommand. My explanatory comment said
'buildah ran with two graph roots...', so the scanner read 'ran' as a
subcommand and failed the build. Reworded to 'buildah(1) there had...'.

The scanner also matches inside comments generally, so any future sentence
phrasing a buildah command in prose can trip it. Not changing that here: it
is pre-existing behaviour, and it is exactly the check that caught this.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

One of the two blockers is fixed; the other is still open and the validate job is red on it.

Reviewed changes

  • Reworded the push-manifest.sh comment that the validate job's subcommand scan was tripping on, from buildah ran with two graph roots… to buildah(1) there had two graph roots…. I re-ran the scan's own snippet against this head: no MISSING entries, so buildah "$sub" --help no longer fails before the selftests.
  • Nothing else changed — the registry read, the SOURCE_PREFIX ${x-default} handling, and the two selftests are byte-identical to 11e5320.

That single word was the whole of the delta, and it clears the first finding. The second one is untouched: ns8-module.yml:110-113 still counts arches in ${tmp}/out/index.json, which holds a single descriptor for the list and no platform fields, so count is 0 and the step exits 1. Confirmed both ways on this head — a local reproduction of the step against the published :build-* tags gives COUNT=0 against index.json and 2 against the blob it names, and run 38825865179 fails with assembled index holds 0 arches, expected 2.

Technical details
# The registry selftest is still red — same root cause, still open

## Affected sites
- `.github/workflows/ns8-module.yml:110-113` — `jq` counts `.manifests[]` entries carrying a
  `platform` in `${tmp}/out/index.json`, which can never hold per-arch entries for a list

## Evidence at head 691682f
- local repro of the step: script exits 0, then `count=0`; the blob named by
  `.manifests[0].digest` in `blobs/sha256/<hex>` holds 2
- CI run 38825865179 (`validate`): `##[error]assembled index holds 0 arches, expected 2`

## Why it is not buildah-version specific
`vendor/go.podman.io/image/v5/oci/layout/oci_dest.go` `PutManifest` writes the blob, then
returns early when `instanceDigest != nil`, appending to the index only for the primary
manifest. Instances pushed under `--all` are blobs only.

## Required outcome
The assertion must read the blob referenced by `index.json`, so a wrong or partial assembly
is actually caught — a plain exit-code check would not have caught it, which was the point
of the assertion.

## Suggested approach
```bash
list=$(jq -r '.manifests[0].digest' "${tmp}/out/index.json")
count=$(jq '[.manifests[]
           | select(.platform.architecture != null
                    and .platform.architecture != "unknown")]
         | length' "${tmp}/out/blobs/sha256/${list#sha256:}")
```

## Verified
- hermetic path -> 2
- registry path against `ghcr.io/calnode/ns8-calnode:build-{amd64,arm64}` -> 2

Pullfrog  | Fix it ➔ | View workflow run | Using space-bunny-free (free) | 𝕏

The registry-backed selftest failed on its own assertion, not on the
assembly: the run logged 'Published ... (amd64 arm64)' and wrote a correct
two-arch list. buildah stores that list as a single entry in the OCI layout's
index.json, and the per-arch manifests live inside the index that entry
references, so the top level holds no platform entries to count.

Follow the reference when the top level is empty, and fail loudly if there is
neither a top-level platform set nor a readable blob. Validated locally
against a real OCI layout built from the published :edge index.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Reworked the registry selftest's arch assertion in .github/workflows/ns8-module.yml:113-129 so it follows the digest when index.json itself carries no platform entries: it counts .manifests[] with a platform, and only on a count of 0 resolves .manifests[0].digest and recounts in blobs/sha256/<hex>, erroring with the raw index if that blob is missing.
  • Added [ .manifests[]? ] to both jq selectors, so a layout without a manifests array reports 0 and takes the fallback rather than failing on a null iteration.

That is the whole delta since 691682f; the registry read, the SOURCE_PREFIX ${x-default} handling and the two selftests are byte-identical to the previous head. This closes the last open finding from the prior review, so the PR has no remaining threads.

Verified at 0ccb850 on buildah 1.33.7: the registry selftest ran verbatim against the published :build-{amd64,arm64} tags (Published … selftest-registry (amd64 arm64)), index.json came out as the single application/vnd.oci.image.index.v1+json descriptor with top=0, the fallback resolved the blob and returned final count=2. The hermetic selftest still passes, the validate subcommand scan reports no MISSING entries, and the validate and build checks are green on this head.

One thing carried forward rather than raised again: reading :build-<arch> from the registry means the assembled list is built from whatever those tags point at when the step runs, and they are rewritten by every branch and release run while concurrency only groups by ref. Per-run unique build tags would close it. Not a blocker and already flagged on the initial review.

Pullfrog  | View workflow run | Using space-bunny-free (free) | 𝕏

@shockalotti
shockalotti merged commit 0f721a4 into main Oct 16, 2026
7 checks passed
@shockalotti
shockalotti deleted the fix/ns8-manifest-registry branch October 16, 2026 00:15
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant