Repository navigation
fix(ci): stop the release assertion racing the app publish - #151
Conversation
assert-multiarc.py resolved the app image named by org.nethserver.images, but
ns8-module.yml and docker-publish.yml are triggered by the same tag push and run
CONCURRENTLY. On a fresh release the app image frequently is not published yet,
so the check raced the app publish and would fail most first-time releases with
FAIL cannot fetch app image ghcr.io/calnode/calnode:0.11.1: HTTP 404
while failing to catch the real thing. On v0.11.1 the module build finished
first and the app took about nine minutes more.
The fix is an ordering dependency, not a longer retry. A retry budget has to be
guessed, and the guess is wrong in both directions: long enough to cover a slow
app build, it stops failing a genuinely absent app image, and short enough to
stay honest, it fails releases whose app image was merely slow. The gap grows
with app build time, so any fixed number rots.
Split the two concerns instead:
assert-multiarc.py is now module-only by default. It checks what the module
workflow just published, which is intrinsic and cannot race: the tag is a real
image index, each required architecture is present, each config really declares
it, each carries a well-formed org.nethserver.images label, and every
architecture agrees on the same app image. Nothing is resolved from the app
repository.
--pair additionally resolves and checks the app image, and --expect-app-tag
additionally asserts every label names exactly the app tag this release
publishes. That makes the module/app matched pair an explicit assertion rather
than a side effect of the app lookup, which was the gap flagged earlier: today
a module tag pointing at a different app version would still have passed.
release-verify.yml is a new workflow triggered on the COMPLETION of both publish
workflows. The job runs only when both concluded successfully for the same
commit, and the run that does the work is triggered by whichever finishes last.
There is no sleep, no retry budget and no polling loop deciding whether to wait:
the two completion events are the dependency. A slow app publish means the job
runs later, not that it fails. Because both have finished by then, an app tag
still absent at that point is genuinely wrong rather than late, so --pair fails
on it promptly instead of waiting.
Every branch of the gate reports an explicit outcome: verify,
waiting-for-sibling, blocked, not-verified.
Proved against the real registry, not just reasoned about:
A module-only, real 0.11.1/0.11/latest exit 0
B --pair --expect-app-tag 0.11.1, same tags exit 0
C genuinely missing app tag FAILED in 2.2s, HTTP 404
D --expect-app-tag 0.0.0 against a 0.11.1 module exit 1, mismatch reported
E gate logic, executed from the shipped YAML:
both succeeded -> verify
sibling in_progress -> waiting-for-sibling, exit 0
sibling queued -> waiting-for-sibling, exit 0
sibling concluded failure -> blocked, exit 0
triggering run failed -> not-verified, exit 0
E is the distinction asked for: a slow app image must not fail the release, and
a missing one must. C is a bounded failure rather than a hang, and no case can
be made permissive by waiting longer.
There was a problem hiding this comment.
Important
The design is right — moving the app-image lookup behind an ordering dependency rather than a retry budget is the correct call, and splitting the script's module-only invariants from the pair check keeps each check where it can actually be trusted. Four things should land before merge, though, and they share one theme: three separate paths let this gate finish green without having verified the release.
Reviewed changes
assert-multiarc.pysplit into module-only (default) and--pairmodes — the module checks (image index, every required arch present and really declared by its config, a well-formedorg.nethserver.imageslabel) stay in the publish path; app resolution moves behind--pair. Adds a cross-architecture "every arch agrees on one app image" check and--expect-app-tagto pin the module/app version pair.ns8-module.ymlruns module-only — the racing app lookup is removed from the publish job, with the concurrency rationale documented next to the step.release-verify.yml(new) —workflow_runon both publish workflows, a gate that proceeds only once both concludedsuccessfor the same commit, tag resolution by commit SHA, then the--pair --expect-app-tagassertion overX.Y.Z/X.Y/latest.
I checked the premise the gate rests on, since a mismatch there would make the whole workflow a no-op: this repo's tags are annotated, and the values do line up. /repos/Calnode/calnode/tags reports .commit.sha 39954841 for v0.11.1 (dereferenced to the commit, not the tag object 174a7f4f), and the Docker publish run for that tag push has head_sha=39954841 too. Annotated tags will not break tag resolution here.
💡 Two consequences of the three-workflow shape worth a deliberate decision
Neither is wrong, but both are load-bearing and currently implicit.
Every PR now spawns two release-verify runs. ns8-module.yml and docker-publish.yml both trigger on pull_request, so each PR completion fires this workflow twice. Both firings gate to verify, then the tag step finds no tag at the PR's merge-commit SHA and skips — so the cost is a checkout plus three API calls twice per PR, not a correctness problem. if: github.event.workflow_run.event == 'push' on the job removes it for free.
:edge and :dev lose the app-image check entirely, not just the ordering guarantee. release-verify.yml skips anything without a matching vX.Y.Z tag, so the module :edge / :dev tags pushed on main / dev are now covered by module-only checks alone. Before this PR they were checked against their app image (:edge / :dev) too — and that check was itself racing, which is the point of the change. The result is a narrower guarantee for branch tags than for release tags, which the ns8-module.yml comment still reading :edge and :dev ... deserve exactly the same guarantee as a version tag no longer describes accurately. If dropping it is intended, say so in that comment; if not, the same pair logic extends to branch tags once both workflows have completed.
Worth noting alongside the coupling you already flagged: workflow_run gives no signal when its workflows: filter stops matching, so renaming either upstream workflow makes this workflow go dark rather than red. A filter that never fires is indistinguishable from a filter that has not fired yet.
The shape that removes the third workflow, the name coupling, the double firing, the workflow_run default-branch limitation and both of the above is folding the publishes into one workflow with three jobs (module, app, verify: needs: [module, app]) — same trigger, same tags, needs instead of polling. Larger change than what you have and it couples two things that are currently independent, which may be the reason you chose not to; just confirming the alternative was weighed rather than assumed away.
Technical details
# `release-verify.yml` can complete green without verifying a release
## Affected sites
- `.github/workflows/release-verify.yml:67-70` — sibling matched on `head_sha` only; `first` is whichever run the API lists first
- `.github/workflows/release-verify.yml:73-85` — `waiting-for-sibling`, `blocked` and `not-verified` all `exit 0`, and `outcome` is never consumed
- `.github/workflows/release-verify.yml:98-100` — `--paginate` without `--slurp` filters page-by-page; `head -1` then discards the rest
- `packaging/ns8/assert-multiarc.py:225-226` — `--app-repo` parsed, never read
- `packaging/ns8/assert-multiarc.py:196-198` — `--expect-app-tag` unreachable in module-only mode
## Required outcome
Every path through this workflow that does **not** verify the pair must be visible as such — in the job's own summary or name, not only in a step log that scrolls past. And each flag on `assert-multiarc.py` must do what its help text says, or not exist.
## Verified facts
- Ambiguous sibling runs are not hypothetical. `ns8-module` runs `38838599412` (`head_branch=v0.11.1`) and `38838477075` (`head_branch=main`) share `head_sha=39954841`; `docker-publish` runs `38838599371` and `38838477107` likewise. A run created *after* the tag run for the same sha (a re-run, or a `main` push landing later) becomes `first` and substitutes its conclusion for the tag run's.
- `gh api --paginate` applies `--jq` per page; only `--slurp` concatenates first (`gh api --help`). `/repos/:owner/:repo/tags` paginates at 30; the repo is at 16 tags.
- `grep -n app_repo packaging/ns8/assert-multiarc.py` matches only the `add_argument` at line 225 — `check_app_image` receives its repo from `split_ref(app_ref)` at line 169, via `check_tag`'s loop at line 200.
- `jq '[...] | first | .status // "absent"'` on an empty `workflow_runs` array returns `absent` with exit 0, so the `absent` sentinel in the gate is sound — no jq null-indexing hazard there.
## Open questions for the human
- Is losing app-image coverage for `:edge` / `:dev` intended, or should the pair check also run for branch pushes once both workflows complete?
- Should `blocked` (a sibling concluded non-`success`) really be exit 0, or is a red run the honest signal for "this is not a clean release"?space-bunny-free (free) | 𝕏
| return 2 | ||
| def check_tag(repo, tag, arch_names, pair, expect_app_tag): | ||
| found = app_refs_for(repo, tag, arch_names) | ||
| if not pair: |
There was a problem hiding this comment.
--expect-app-tag is only consulted below this early return, so assert-multiarc.py --expect-app-tag 0.11.1 calnode/ns8-calnode 0.11.1 exits 0 having checked nothing about the app tag — and prints PASS all 1 tag(s) are multi-arch with coherent labels, which reads like it passed. That is exactly the module/app skew this PR exists to catch, reported as a pass.
Technical details
# `--expect-app-tag` silently no-ops without `--pair`
## Affected sites
- `packaging/ns8/assert-multiarc.py:196-198` — early return precedes every `expect_app_tag` read
- `packaging/ns8/assert-multiarc.py:201` — first `expect_app_tag` use, inside the `--pair` loop
## Required outcome
`--expect-app-tag` can never be passed in a mode that ignores it. Both of these are acceptable:
1. treat `--expect-app-tag` as implying `--pair` (`pair = args.pair or bool(args.expect_app_tag)`), or
2. reject the combination outright.
Silently passing is not.| if not pair: | |
| if not pair: | |
| if expect_app_tag: | |
| raise Fail( | |
| "--expect-app-tag was given without --pair, so no app tag is read; " | |
| "pass --pair or drop --expect-app-tag" | |
| ) | |
| print(" -- not checking the app image (module-only mode)") | |
| return |
| exit 0 | ||
| fi | ||
|
|
||
| status=$(gh api "repos/${{ github.repository }}/actions/workflows/$SIBLING/runs" \ |
There was a problem hiding this comment.
The sibling is matched on head_sha alone, but a tagged commit normally also has main/dev branch runs at that same sha, and first takes whichever the API lists first. This repo is a live example: ns8-module runs 38838599412 (head_branch=v0.11.1) and 38838477075 (head_branch=main) both sit on 39954841, and for that sha the two workflows concluded differently. Any run created after the tag run — a re-run, or a main push landing later — becomes first and has its conclusion substituted for the tag run's. Match head_branch too, so the gate reads the run for the tag rather than any run at that commit.
Technical details
# Sibling run lookup is ambiguous by sha
## Affected sites
- `.github/workflows/release-verify.yml:67-70` — both `--jq` filters select on `head_sha` only
## Required outcome
The gate must read the sibling's conclusion for the **tag** ref, not an arbitrary run at the same commit.
## Suggested approach
`workflow_run` payload carries `github.event.workflow_run.head_branch`, which for a tag push is the tag name (`v0.11.1`). Export it as `REF` alongside `SHA` and add it to both filters:
```bash
--jq "[.workflow_runs[] | select(.head_sha == \"$SHA\" and .head_branch == \"$REF\")] | first | .status // \"absent\""
```
If `REF` turns out empty for any upstream event that matters, `absent` degrades to `waiting-for-sibling` (exit 0), so a wrong guess here is still fail-open rather than fail-closed — but it is a strictly tighter filter than sha alone, and it is worth confirming against a real tag run.
## Evidence
- `ns8-module` `38838599412` `head_branch=v0.11.1` `head_sha=39954841` `conclusion=failure`
- `ns8-module` `38838477075` `head_branch=main` `head_sha=39954841` `conclusion=failure`
- `docker-publish` `38838599371` `head_branch=v0.11.1` `head_sha=39954841` `conclusion=success`
- `docker-publish` `38838477107` `head_branch=main` `head_sha=39954841` `conclusion=success`| # workflow_run does not carry the ref, so find the tag that points at | ||
| # this commit. A plain branch push has no such tag and skips, which is | ||
| # how this stays out of the way of ordinary commits. | ||
| tag=$(gh api "repos/${{ github.repository }}/tags" --paginate \ |
There was a problem hiding this comment.
--paginate applies the jq filter to each page separately — only --slurp concatenates them first — so first // empty never sees past page 1 and head -1 then keeps just that one line. /repos/:owner/:repo/tags paginates at 30 and the repo is at 16 tags, so once that crosses 30 the matching tag falls off the end and the step reports no release tag points at <sha>; not a tag release, skipping — a green run that verified nothing, with no error anywhere. --paginate --slurp and drop the head.
Technical details
# Tag resolution is page-local, and the skip is indistinguishable from success
## Affected sites
- `.github/workflows/release-verify.yml:98-100` — `--paginate` without `--slurp`, piped to `head -1`
## Required outcome
The tag lookup must see every tag, and "found no tag" must not read as a pass.
## Suggested approach
```bash
tag=$(gh api "repos/${{ github.repository }}/tags" --paginate --slurp \
--jq "add | map(select(.commit.sha == \"$SHA\")) | map(.name) | map(select(test(\"^v[0-9]+\\\\.[0-9]+\\\\.[0-9]+$\"))) | first // empty")
```
`add` flattens the array-of-pages that `--slurp` produces, so the filter finally runs over all tags. Removing `head -1 | tr -d '\r'` also clears the `SIGPIPE` hazard: under `set -o pipefail` (`release-verify.yml:94`), `gh` at the far end of the pipeline exits non-zero once output outgrows the 64 KiB pipe buffer, which fails the step for a reason unrelated to the release. `tr -d '\r'` is still worth keeping — `gh` output is not CRLF-terminated in practice, so it is harmless either way.
## Evidence
- `gh api --help`: "Each page is a separate JSON array or object. Pass `--slurp` to wrap all pages of JSON arrays or objects into an outer JSON array."
- `curl -s 'https://api.github.com/repos/Calnode/calnode/tags?per_page=100' | jq length` → `16` (16 tags today, 30 per page by default)| echo "outcome=blocked" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "ready=true" >> "$GITHUB_OUTPUT" | ||
| echo "outcome=verify" >> "$GITHUB_OUTPUT" |
There was a problem hiding this comment.
outcome is written in all four branches and never read — steps.gate.outputs.ready is the only gate output anything consumes. So verify, waiting-for-sibling, blocked and not-verified all produce an identically green job with the same name, and a release whose pair check never ran looks the same as one that passed. Since three of the four outcomes are no-ops, that ambiguity is the whole risk of this design.
Technical details
# Three no-op gate outcomes are indistinguishable from a pass
## Affected sites
- `.github/workflows/release-verify.yml:63,77,81,84` — `outcome=` written, never consumed
- `.github/workflows/release-verify.yml:111` — final step gates on `ready` only, ignores `outcome`
## Required outcome
A reader of the Actions UI (or a branch-protection rule) can tell a run that verified the pair from one that did not, without opening the step log.
## Suggested approach
Cheapest, self-contained: append the outcome to the run summary.
```bash
echo "### release-verify: $OUTCOME" >> "$GITHUB_STEP_SUMMARY"
```
Better, if anything gates on this check: surface it as the job name.
```yaml
jobs:
verify:
name: Release verify (${{ needs... }})
```
A `verify` job cannot read the gate step's outputs at name-evaluation time (they are step outputs, produced at run time), so the name route needs a small follow-up job that always sets `ready` first — which is more machinery than it is worth here. The step summary plus a `::notice::`/`::warning::` per outcome gets most of the value; the notice severity matters most: `blocked` in particular is a real release failure being reported as a green run.
## Open question for the human
Should `blocked` — a sibling concluded `success`/`failure`/`cancelled` — really be `exit 0`? It reads as "this is not a clean release", which is the one outcome that arguably should be red.Reported by the PR-triage agent reviewing #151, and confirmed before fixing: pointing the flag at a repository that does not exist still passed. assert-multiarc.py --pair --expect-app-tag 0.11.1 \ --app-repo calnode/definitely-not-a-real-app calnode/ns8-calnode 0.11.1 PASS all 1 tag(s) are multi-arch and coherent args.app_repo was parsed and never read. The app repository was always derived from the module's own org.nethserver.images label, so the flag did nothing at all. It was dead, and the APP_REPO env var in release-verify.yml that fed it was dead with it. Removed rather than implemented, because an override is the wrong idea here. The app image a module actually runs is whatever its label names, so the label is the only authoritative source and is also what a fork needs: a module built under another namespace names their app image, and deriving from the label handles that with no configuration. An override would be worse than useless, because it would let a pair check be asserted against an app repository the module does not reference, hiding a mislabelled module. The docstring now says so explicitly, so the next person does not add it back. Also correct on one point of the report: the evidence given that module-only mode never resolves the app image does not show that, because it was produced by passing the dead --app-repo flag. The property does hold, and it is now demonstrated directly by instrumenting the fetch layer and recording every repository contacted: module-only, real 0.11.1/0.11/latest repositories contacted: ['calnode/ns8-calnode'] contacted the app repo calnode/calnode? False --pair --expect-app-tag 0.11.1, same tags repositories contacted: ['calnode/calnode', 'calnode/ns8-calnode'] contacted the app repo calnode/calnode? True Re-ran the whole proof set after the fix: dead flag now rejected by argparse exit 2, cannot mislead module-only, real tags exit 0 --pair --expect-app-tag 0.11.1 exit 0 genuinely missing app image FAILED in 2.3s, HTTP 404 mismatched pair exit 1, mismatch reported gate logic from the shipped YAML 5/5 correct
|
Review caught a real defect, now fixed in The defect
Removed, not implementedThe app image a module actually runs is whatever its label names, so the label is the only authoritative source, and it is also what a fork needs: a module built under another namespace names their app image, and deriving from the label handles that with no configuration. An override would be worse than useless, because it would let a pair check be asserted against an app repository the module does not reference, hiding a mislabelled module. The docstring now says exactly that, so it does not get added back quietly. One correction to the reviewThe evidence offered that module-only mode never resolves the app image does not demonstrate that, because it was produced by passing the dead The property does hold, and it is now shown directly by instrumenting the fetch layer and recording every repository contacted: Full proof set re-run
Both workflows still parse. Worktree is still at |
There was a problem hiding this comment.
ℹ️ No new issues in this delta. One prior finding is addressed cleanly; four remain open from the previous review.
Reviewed changes
Scope this run: the single new commit 83251c7 on top of 6e2fc01 — the removal of the dead --app-repo override. The rest of the PR (release-verify.yml, the module-only/--pair split) was reviewed previously and is unchanged.
- Removed the inert
--app-repooption —assert-multiarc.pyno longer parses it,DEFAULT_APP_REPOand theAPP_REPOmodule-level constant are gone, and both workflows drop the--app-repo "$APP_REPO"argument and theAPP_REPOenv beside it. This closes the finding that the flag was documented and passed but never read. - Documented why there is no app-repository override — the module docstring now states that the
org.nethserver.imageslabel is the only authoritative source, that this is what a fork needs, and that an override would let a pair check be asserted against a repository the module does not actually reference.release-verify.ymlcarries a matching two-line comment at the call site.
I verified the change end to end rather than reading it only: assert-multiarc.py compiles, --help no longer advertises the flag, and both remaining call sites (ns8-module.yml:266, release-verify.yml:125) pass the module repository positionally with no app-repository argument left anywhere — grep for APP_REPO across the workflows returns nothing.
Dropping the override also leaves the fork case working without configuration: check_app_image receives its repository from split_ref(app_ref) on the label, so a module built under another namespace resolves its own app image. --expect-app-tag still pins the tag, so the module/app version pairing this PR exists to prove is unaffected.
Four threads from the previous review are still open and still current — the sha-only sibling lookup, the dead outcome output, --paginate without --slurp, and --expect-app-tag no-opping without --pair. None were touched by this commit, so I have not re-raised them here.
space-bunny-free (free) | 𝕏

Fixes defect 2. Defect 1 (the lowercase token scope) is already on main as
d5cc8eeand is untouched here.The race
assert-multiarc.pyresolved the app image named byorg.nethserver.images, butns8-module.ymlanddocker-publish.ymlare triggered by the same tag push and run concurrently. On a fresh release the app image frequently is not published yet, so:On v0.11.1 the module build finished first and the app took roughly nine minutes more. This is by construction, not chance.
Why not a longer retry
A retry budget has to be guessed, and the guess is wrong in both directions. Long enough to cover a slow app build, it stops failing a genuinely absent app image. Short enough to stay honest, it fails releases whose app image was merely slow. The gap grows with app build time, so any fixed number rots. That is precisely the shape of bug that lies to you later rather than now.
So: an ordering dependency.
What I chose
assert-multiarc.pyis module-only by default. It checks only what the module workflow just published, which is intrinsic and cannot race: the tag is a real image index, each required architecture is present, each config really declares it, each carries a well-formedorg.nethserver.imageslabel, and every architecture agrees on the same app image. Nothing is resolved from the app repository.--pairresolves and checks the app image. Plus--expect-app-tag, which asserts every label names exactly the app tag this release publishes.New
.github/workflows/release-verify.yml, triggered on the completion of both publish workflows. The job runs only when both concluded successfully for the same commit, and the run that does the work is triggered by whichever finishes last. No sleep, no retry budget, no polling loop deciding whether to wait: the two completion events are the dependency. A slow app publish means the job runs later, not that it fails.Because both have finished by then, an app tag still absent is genuinely wrong rather than late, so
--pairfails on it promptly.On making the pair explicit
Yes, and
--expect-app-tagdoes it. Previously the pair was only checked as a side effect of the app lookup, which is what you flagged: a module tag pointing at a different app version would still have passed, as long as that version existed and was multi-arch. Now the module tag, itsX.Yalias and:latestmust all name the app tag this release publishes, so a half-published or skew release fails loudly.Proved against the real registry
0.11.1/0.11/latest--pair --expect-app-tag 0.11.1, same tags--expect-app-tag 0.0.0against a0.11.1moduleE, which is the distinction you asked for:
A slow app image does not fail the release. A missing one does, and bounded, not as a hang. No case becomes permissive by waiting longer. E runs the actual
run:block from the workflow file, substituting the two GitHub expressions, so it tests the text that will ship rather than a paraphrase.Cost, stated plainly
workflow_rundoes not fire for workflows absent from it. Merge before the next tag and it is armed.workflow_runfires twice, once per upstream workflow. The first firing no-ops unless the sibling has already finished, so the second does the work. In the rare case both land together, both firings verify: harmless, idempotent, occasionally a duplicate run.actions: readandcontents: read. The module repo is public, so no new secret.workflow_rundoes not carry the ref, so the tag is resolved by finding which tag points athead_sha. A plain branch push has no such tag and skips, which is how this stays out of ordinary commits' way.'NS8 module publish','Docker publish'). Renaming either breaks this silently. It is in one obvious place and the gate reports it aswaiting-for-siblingrather than passing quietly, but it is a real coupling.Note
The v0.11.1 tag run stays red, correctly: a tag pins its own commit, so the assertion fix does not and cannot retroactively apply to it. Its red is still attributable to defect 1 only.