Skip to content

ci(ns8): gate releases on what actually reached the registry - #149

Merged
shockalotti merged 1 commit into
mainfrom
ci/ns8-publish-assertions
Oct 16, 2026
Merged

shockalotti merged 1 commit into
mainfrom
ci/ns8-publish-assertions

Conversation

@shockalotti

Copy link
Copy Markdown
Contributor

Two silent-failure guards, each for a bug that has already shipped once. Both are the kind that cannot be caught by looking at the code, only by looking at what actually happened.

1. Post-publish multi-arch assertion

A release once published an amd64-only image as :latest. The publish step exited 0. Nothing looked at the registry. The result installs fine on amd64 and hard-fails on arm64.

packaging/ns8/assert-multiarc.py runs after push-manifest.sh and fails the release unless, for every required architecture:

  1. the tag resolves to an image index / manifest list, not a single manifest
  2. the index contains that architecture
  3. that architecture's config really declares it, so an entry cannot be mislabelled inside the list
  4. that config carries org.nethserver.images (NS8 reads it at install time; without it the module installs and does nothing)
  5. the app image that label names is itself multi-arch and contains the same architecture

It reads the registry, not local storage, deliberately: a broken publish must not be able to pass by looking healthy from the runner. No extra secrets are needed, since the module repository is public and the token is anonymous. It retries briefly so push-to-read propagation latency cannot fail a good release, and then fails rather than waiting longer.

It runs for branch pushes too. :edge and :dev are published tags and deserve the same guarantee as a version tag.

2. state-include.conf assertion

You asked whether this deserved the cheap guard too. It does, and the case is strong: that file is handed to restic as --files-from, so restic backs up only the paths listed in it. Dropping a line loses data while the backup job still reports success. That is exactly how #142 shipped a module with the database but not the encryption key.

Both paths are now asserted in the validate job, with the reason in the comment so the next person to trim that file learns why they cannot.

On whether the two multi-arch checks are redundant

They are not, they test different things:

  • the registry-backed selftest (from packaging(ns8): read manifest arches from the registry, not local storage #148) proves the script can assemble a two-arch list, using locally-published :build-<arch> tags and pushing to an oci: path. It publishes nothing.
  • the post-publish assertion proves the artifact in the registry is correct, including the per-arch labels and the app image they point at, which the selftest never inspects.

Together they cover the assembly and the result. The missing third piece was exactly this one, which is why an amd64-only :latest could ship with both of the others green.

Verified locally

The assertion script exits 1 against the current amd64-only :latest with an accurate diagnosis, and exits 0 against :edge:

FAIL  calnode/ns8-calnode:latest: tag is a single-arch manifest
      (application/vnd.oci.image.manifest.v1+json), not a multi-arch index;
      an arm64 node cannot pull it
PASS  all 1 published tag(s) are multi-arch and coherent

The state-include.conf check passes against the real file and correctly detects a simulated dropped line. YAML parses, and the build-job step order is login, build, assemble, assert.

Notes

  • The post-publish assertion cannot run on a PR, since there is nothing published to assert against. So validate py_compiles the script, which otherwise would ship entirely unexercised.
  • .gitignore now covers __pycache__, which a local py_compile run would otherwise commit. I hit exactly that while building this.
  • I hit git reset --hard losing my workflow edits and had to redo them, so the wiring is worth a second pair of eyes.

Two silent-failure guards, both for bugs that already shipped once.

Post-publish multi-arch assertion. A release once published an amd64-only
image as :latest. It installs on amd64 and hard-fails on arm64, the publish
step exited 0, and nothing looked at the registry. packaging/ns8/assert-
multiarc.py now runs after push-manifest.sh against the published tags and
fails the release unless every required architecture is present, that
architecture's config really declares it, it carries an org.nethserver.images
label, and the app image that label names is itself multi-arch and contains
the same architecture.

It checks the registry rather than local storage on purpose: a broken publish
must not be able to pass by looking healthy from the runner. It needs no extra
secrets, since the module repository is public and the token is anonymous, and
it retries briefly so push-to-read propagation latency cannot fail a good
release. It runs for branch pushes too, because :edge and :dev are published
tags and deserve the same guarantee as a version tag.

state-include.conf assertion. That file is handed to restic as --files-from,
so restic backs up ONLY the paths listed in it. Dropping a line loses data
while the backup job still reports success, which is exactly how #142 shipped
the module with the database but not the encryption key. Both paths are now
asserted in the validate job, with the reason in a comment, so the next person
who trims that file finds out from CI rather than from a failed restore.

Also py_compiles the new script in the validate job, since the post-publish
assertion cannot run on a PR (there is nothing published to assert against) and
would otherwise ship completely unexercised. .gitignore now covers
__pycache__, which a local py_compile run would otherwise commit.

Verified locally: the assertion script exits 1 on the current amd64-only
:latest with an accurate diagnosis, and exits 0 on :edge. The state-include
check passes against the real file and detects a simulated dropped line.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

Two things to fix before this can be merged: the assert step as wired fails on every non-PR run, and the app-image check puts the module release behind a workflow it has no dependency on. Both findings below were reproduced against the live GHCR API, not reasoned about.

Reviewed changes

  • packaging/ns8/assert-multiarc.py (new) — a post-publish gate that reads the GHCR registry (anonymous pull token) instead of local storage and, per tag and per required arch, asserts the tag is an image index, holds linux/<arch>, that the arch's config really declares it, that the config carries org.nethserver.images, and that the app image that label names is itself multi-arch with the same arch. Fetches retry 4 × 10s before failing.
  • .github/workflows/ns8-module.yml — runs the assert after push-manifest.sh (skipped on PRs, matching the other publish steps), py_compiles it in validate, and adds a state-include.conf guard asserting both volumes/calnode-data and state/calnode.env are still listed, with the restic --files-from rationale inline.
  • .gitignore — ignores __pycache__/, which a local py_compile would otherwise leave committable.

The design reads well and the rationale is right in both places: the registry is the only honest source of truth after a publish, and state-include.conf really is a silent-data-loss surface. The hermetic #148 selftest and this gate are complementary, as the description argues. The issues are in the wiring, not the idea.

ℹ️ A failing gate here reports, it cannot repair

Worth being explicit about the recovery story before this lands: the tags are pushed by push-manifest.sh before the assert step runs, so when the gate goes red the bad or unverified tag is already live and :latest / :edge may already have moved. Re-running only the assert step cannot fix it — recovery means re-pushing the module tags. Not asking for a change here, but the runbook for "NS8 publish went red" should say that.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using space-bunny-free (free) | 𝕏

if: github.event_name != 'pull_request'
working-directory: packaging/ns8
env:
MODULE_REPO: ghcr.io/${{ github.repository_owner }}/ns8-calnode

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

${{ github.repository_owner }} expands to Calnode, and GHCR's token endpoint rejects a mixed-case scope — so token() in assert-multiarc.py raises HTTPError before any manifest is fetched. As written, every non-PR run of this workflow (every main/dev push, every v*.*.* tag) ends red with cannot fetch …: HTTP 400 — after the tags have already been pushed, and after 30s of retries that can never succeed. Both existing consumers of REPOBASE already normalise for exactly this reason.

Suggested change
MODULE_REPO: ghcr.io/${{ github.repository_owner }}/ns8-calnode
MODULE_REPO: ghcr.io/${{ github.repository_owner }}/ns8-calnode
IMAGETAGS: ${{ steps.tags.outputs.module_tags }}
run: python3 assert-multiarc.py "${MODULE_REPO,,}" $IMAGETAGS

print(f" ok linux/{arch} org.nethserver.images = {app_ref}")

app_repo, app_tag = split_ref(app_ref)
abody, actype = fetch_with_retry(app_repo, app_tag, f"app image {app_ref}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assertion makes a module release depend on docker-publish.yml having finished — both workflows fire on the same push/v*.*.* trigger with no ordering between them, and this fetch gives up after 30s. Because the app image is refetched per arch, a release tag (3 module tags × 2 arches) can burn ~3 minutes and then fail red on a module image that is itself perfectly good, purely because the arm64 app build is still running. Worth keeping the check, but decoupled from the retry budget that protects against push-to-read latency on the module's own tag.

@shockalotti
shockalotti merged commit 212f779 into main Oct 16, 2026
7 checks passed
@shockalotti
shockalotti deleted the ci/ns8-publish-assertions branch October 16, 2026 01:45
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant