Repository navigation
ci(ns8): serialise module publishes, which shared a staging tag across refs - #172
Conversation
…s refs
build-images.sh pushes each architecture to a fixed staging tag
(:build-amd64 / :build-arm64) and push-manifest.sh assembles the multi-arch
index from whatever is there. The concurrency group was
`ns8-module-${{ github.ref }}`, so a push to main and a release tag - which are
cut seconds apart, because the release commit lands on main immediately before
the tag - were different groups and ran concurrently, overwriting each other's
staging images.
That published a manifest whose architectures came from different releases. It
happened on v0.11.2: :edge carried amd64 from the tag (app 0.11.2) and arm64
from main (app edge), and the post-publish architecture assertion failed the
main build. The tag's own :0.11.2 / :0.11 / :latest were coherent, and :edge was
rebuilt after the tag run finished.
Use one concurrency group for every module publish and stop cancelling in
progress, so a release build is never cancelled by an unrelated push. Module
publishes take 1-2 minutes, so queueing is cheap. The assertion that caught it
stays as the backstop.
There was a problem hiding this comment.
Important
cancel-in-progress: false stops a running release from being cancelled, but the group still has a one-slot pending queue by default, so a queued release run is still cancelled by the next run that enters the group — which is exactly the window this PR sets out to protect.
Reviewed changes
- Single publish concurrency group —
.github/workflows/ns8-module.ymldrops the ref-keyed group in favour ofns8-module-publishwithcancel-in-progress: false, serialising every run that shares the:build-<arch>staging tags. - CHANGELOG entry — documents the v0.11.2 mixed-arch
:edgeincident under Unreleased → Fixed, including the:edgerebuild.
The diagnosis matches the scripts: build-images.sh pushes to fixed :build-${targetarch} tags and push-manifest.sh assembles from them, so ref-keyed concurrency genuinely allowed the interleaving. The shared group is the right shape; the remaining gap is in how GitHub drains the queue.
deepseek-v4.1-flash | 𝕏
| group: ns8-module-publish | ||
| cancel-in-progress: false |
There was a problem hiding this comment.
With GitHub's default queue: single, a newer run entering this group cancels any existing pending run; cancel-in-progress: false only protects runs that are already executing. Since the tag is pushed seconds after the main push, the release run is usually the pending one, so a later main/dev push or a pull_request run (PR runs share this group but never write :build-<arch>) can cancel the release publish before it starts, leaving :X.Y.Z/:X.Y/:latest a 404 while the release exists. Add queue: max so publish runs queue instead of being replaced, and consider keying PR runs to their own group so they cannot occupy or cancel publish slots.
Technical details
# Pending release runs are still cancelled by the next queued run
## Affected sites
- `.github/workflows/ns8-module.yml:35-36` — shared group with the default `queue: single`
## Required outcome
- A release-tag publish must survive unrelated runs entering the group while it is waiting, not just while it is running.
## Suggested approach
- Add `queue: max` to the workflow-level `concurrency` block (keeps `cancel-in-progress: false`).
- Optionally give PR runs a separate group, e.g. `group: ${{ github.event_name == 'pull_request' && format('ns8-module-pr-{0}', github.ref) || 'ns8-module-publish' }}` — PR runs only read `:build-<arch>` in the registry selftest, so they do not need the publish lock.
## Open questions for the human
- Is the per-run staging tag (`:build-<arch>-<run_id>`) option preferred over widening the queue? That removes the shared-state coupling entirely but needs a cleanup step.| group: ns8-module-publish | |
| cancel-in-progress: false | |
| group: ns8-module-publish | |
| cancel-in-progress: false | |
| queue: max |

What went wrong
While cutting v0.11.2 the
mainbuild ofNS8 module publishfailed, and the assertion was right::edgepublished a manifest whose amd64 and arm64 came from different releases.Cause
build-images.shpushes each architecture to a fixed staging tag:push-manifest.shthen assembles the multi-arch index from whatever is on those tags. That state is shared by every run.The concurrency group was keyed on the ref:
A release commit goes to
mainand the tag is pushed immediately afterwards, so themainpush and thev0.11.2push are different groups and ran concurrently. Each overwrote the other's staging images between the build and the manifest assembly, so the assembled index mixed them.The release tags themselves (
:0.11.2,:0.11,:latest) were coherent — the assertion only failed for:edge— and:edgewas rebuilt successfully once the tag run had finished.Fix
One concurrency group for all module publishes, and no cancellation:
cancel-in-progressmust stay false: a release build must never be cancelled because an unrelated push tomainarrived. Module publishes take 1-2 minutes, so serialising is cheap.The post-publish architecture assertion is what caught this and stays as the backstop. A CHANGELOG entry is included under Unreleased.
Not done
Unique per-run staging tags (
:build-amd64-<run_id>) would also fix it and allow parallel publishes, but they leave garbage tags in the registry unless a cleanup step is added. Serialising is simpler and sufficient at this cadence.