Skip to content

ci(ns8): serialise module publishes, which shared a staging tag across refs - #172

Merged
shockalotti merged 1 commit into
mainfrom
fix/ns8-module-serialize
Oct 27, 2026
Merged

shockalotti merged 1 commit into
mainfrom
fix/ns8-module-serialize

Conversation

@shockalotti

Copy link
Copy Markdown
Contributor

What went wrong

While cutting v0.11.2 the main build of NS8 module publish failed, and the assertion was right:

ok    linux/amd64 org.nethserver.images = ghcr.io/calnode/calnode:0.11.2
ok    linux/arm64 org.nethserver.images = ghcr.io/calnode/calnode:edge
FAIL  architectures disagree on the app image: [('calnode/calnode', '0.11.2'), ('calnode/calnode', 'edge')]
FAIL  ghcr.io/Calnode/ns8-calnode:edge: architectures disagree on the app image

:edge published a manifest whose amd64 and arm64 came from different releases.

Cause

build-images.sh pushes each architecture to a fixed staging tag:

buildtag="build-${targetarch}"      # :build-amd64 / :build-arm64

push-manifest.sh then assembles the multi-arch index from whatever is on those tags. That state is shared by every run.

The concurrency group was keyed on the ref:

group: ns8-module-${{ github.ref }}

A release commit goes to main and the tag is pushed immediately afterwards, so the main push and the v0.11.2 push are different groups and ran concurrently. Each overwrote the other's staging images between the build and the manifest assembly, so the assembled index mixed them.

The release tags themselves (:0.11.2, :0.11, :latest) were coherent — the assertion only failed for :edge — and :edge was rebuilt successfully once the tag run had finished.

Fix

One concurrency group for all module publishes, and no cancellation:

group: ns8-module-publish
cancel-in-progress: false

cancel-in-progress must stay false: a release build must never be cancelled because an unrelated push to main arrived. Module publishes take 1-2 minutes, so serialising is cheap.

The post-publish architecture assertion is what caught this and stays as the backstop. A CHANGELOG entry is included under Unreleased.

Not done

Unique per-run staging tags (:build-amd64-<run_id>) would also fix it and allow parallel publishes, but they leave garbage tags in the registry unless a cleanup step is added. Serialising is simpler and sufficient at this cadence.

…s refs

build-images.sh pushes each architecture to a fixed staging tag
(:build-amd64 / :build-arm64) and push-manifest.sh assembles the multi-arch
index from whatever is there. The concurrency group was
`ns8-module-${{ github.ref }}`, so a push to main and a release tag - which are
cut seconds apart, because the release commit lands on main immediately before
the tag - were different groups and ran concurrently, overwriting each other's
staging images.

That published a manifest whose architectures came from different releases. It
happened on v0.11.2: :edge carried amd64 from the tag (app 0.11.2) and arm64
from main (app edge), and the post-publish architecture assertion failed the
main build. The tag's own :0.11.2 / :0.11 / :latest were coherent, and :edge was
rebuilt after the tag run finished.

Use one concurrency group for every module publish and stop cancelling in
progress, so a release build is never cancelled by an unrelated push. Module
publishes take 1-2 minutes, so queueing is cheap. The assertion that caught it
stays as the backstop.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

cancel-in-progress: false stops a running release from being cancelled, but the group still has a one-slot pending queue by default, so a queued release run is still cancelled by the next run that enters the group — which is exactly the window this PR sets out to protect.

Reviewed changes

  • Single publish concurrency group — .github/workflows/ns8-module.yml drops the ref-keyed group in favour of ns8-module-publish with cancel-in-progress: false, serialising every run that shares the :build-<arch> staging tags.
  • CHANGELOG entry — documents the v0.11.2 mixed-arch :edge incident under Unreleased → Fixed, including the :edge rebuild.

The diagnosis matches the scripts: build-images.sh pushes to fixed :build-${targetarch} tags and push-manifest.sh assembles from them, so ref-keyed concurrency genuinely allowed the interleaving. The shared group is the right shape; the remaining gap is in how GitHub drains the queue.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using deepseek-v4.1-flash | 𝕏

Comment on lines +35 to +36
group: ns8-module-publish
cancel-in-progress: false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With GitHub's default queue: single, a newer run entering this group cancels any existing pending run; cancel-in-progress: false only protects runs that are already executing. Since the tag is pushed seconds after the main push, the release run is usually the pending one, so a later main/dev push or a pull_request run (PR runs share this group but never write :build-<arch>) can cancel the release publish before it starts, leaving :X.Y.Z/:X.Y/:latest a 404 while the release exists. Add queue: max so publish runs queue instead of being replaced, and consider keying PR runs to their own group so they cannot occupy or cancel publish slots.

Technical details
# Pending release runs are still cancelled by the next queued run

## Affected sites
- `.github/workflows/ns8-module.yml:35-36` — shared group with the default `queue: single`

## Required outcome
- A release-tag publish must survive unrelated runs entering the group while it is waiting, not just while it is running.

## Suggested approach
- Add `queue: max` to the workflow-level `concurrency` block (keeps `cancel-in-progress: false`).
- Optionally give PR runs a separate group, e.g. `group: ${{ github.event_name == 'pull_request' && format('ns8-module-pr-{0}', github.ref) || 'ns8-module-publish' }}` — PR runs only read `:build-<arch>` in the registry selftest, so they do not need the publish lock.

## Open questions for the human
- Is the per-run staging tag (`:build-<arch>-<run_id>`) option preferred over widening the queue? That removes the shared-state coupling entirely but needs a cleanup step.
Suggested change
group: ns8-module-publish
cancel-in-progress: false
group: ns8-module-publish
cancel-in-progress: false
queue: max

@shockalotti
shockalotti merged commit 9b0eaf1 into main Oct 27, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant