Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions .github/workflows/ns8-module.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,23 @@ on:
pull_request:

concurrency:
group: ns8-module-${{ github.ref }}
cancel-in-progress: true
# Deliberately NOT keyed on github.ref. build-images.sh pushes each architecture
# to a fixed staging tag (:build-amd64 / :build-arm64) and push-manifest.sh
# assembles the multi-arch index from whatever is on those tags. Two runs on
# different refs therefore share that staging state: a push to main and a
# release tag start seconds apart (the tag is cut right after the release
# commit lands) and are different groups when the ref is in the key, so they
# ran concurrently and overwrote each other's staging images. The result was a
# published manifest whose architectures came from different releases —
# observed on v0.11.2, where :edge carried amd64 from the tag (app 0.11.2) and
# arm64 from main (app edge). One group for all module publishes serialises
# them and removes the shared-state race.
#
# cancel-in-progress stays false: a release build must never be cancelled
# because an unrelated push to main arrived. Module publishes take ~1-2
# minutes, so queueing is cheap.
group: ns8-module-publish
cancel-in-progress: false
Comment on lines +35 to +36

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With GitHub's default queue: single, a newer run entering this group cancels any existing pending run; cancel-in-progress: false only protects runs that are already executing. Since the tag is pushed seconds after the main push, the release run is usually the pending one, so a later main/dev push or a pull_request run (PR runs share this group but never write :build-<arch>) can cancel the release publish before it starts, leaving :X.Y.Z/:X.Y/:latest a 404 while the release exists. Add queue: max so publish runs queue instead of being replaced, and consider keying PR runs to their own group so they cannot occupy or cancel publish slots.

Technical details
# Pending release runs are still cancelled by the next queued run

## Affected sites
- `.github/workflows/ns8-module.yml:35-36` — shared group with the default `queue: single`

## Required outcome
- A release-tag publish must survive unrelated runs entering the group while it is waiting, not just while it is running.

## Suggested approach
- Add `queue: max` to the workflow-level `concurrency` block (keeps `cancel-in-progress: false`).
- Optionally give PR runs a separate group, e.g. `group: ${{ github.event_name == 'pull_request' && format('ns8-module-pr-{0}', github.ref) || 'ns8-module-publish' }}` — PR runs only read `:build-<arch>` in the registry selftest, so they do not need the publish lock.

## Open questions for the human
- Is the per-run staging tag (`:build-<arch>-<run_id>`) option preferred over widening the queue? That removes the shared-state coupling entirely but needs a cleanup step.
Suggested change
group: ns8-module-publish
cancel-in-progress: false
group: ns8-module-publish
cancel-in-progress: false
queue: max


jobs:
validate:
Expand Down
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,19 @@ exact tag (`ghcr.io/calnode/calnode:0.1.0`) if you need stability between upgrad

## [Unreleased]

### Fixed
- **An NS8 module release can no longer mix architectures.** The module build pushes each
architecture to a fixed staging tag (`:build-amd64`, `:build-arm64`) and then assembles the
multi-arch index from them. The workflow's concurrency group included the git ref, so a
push to `main` and a release tag — which are cut seconds apart — ran concurrently and
overwrote each other's staging images, publishing a manifest whose amd64 and arm64 came
from different releases. This is exactly what happened while cutting 0.11.2: `:edge`
carried amd64 from the tag (app `0.11.2`) and arm64 from `main` (app `edge`). The release
tags themselves were coherent; `:edge` was rebuilt. All module publishes now share one
concurrency group and serialise, with `cancel-in-progress` off so a release build cannot be
cancelled by an unrelated push. The post-publish architecture assertion is what caught it
and remains in place.

## [0.11.2] - 2026-10-28

### Added
Expand Down
Loading