Skip to content

fix(ci): stop the arm64 restart assertion failing on SIGPIPE - #211

Merged
shockalotti merged 1 commit into
mainfrom
fix/arm64-smoke-log-assert
Nov 1, 2026
Merged

shockalotti merged 1 commit into
mainfrom
fix/arm64-smoke-log-assert

Conversation

@shockalotti

Copy link
Copy Markdown
Contributor

The failure

#208 (a Dependabot bump of codeql-action, unrelated to any runtime code) failed arm64-smoke with:

##[error]restart did not reuse the persisted key

while the "Show app logs" step in the same run printed exactly the line the assertion was looking for:

keyvault: fresh install — generating DEK and storing in keystore
keyvault: DEK unwrapped from keystore (primary)

Cause

The step runs under set -euo pipefail and the assertion is:

docker logs calnode 2>&1 | grep -q "DEK unwrapped from keystore"

grep -q exits the moment it matches. docker logs is still writing, gets SIGPIPE, and pipefail makes the whole pipeline return 141 even though the match succeeded. It depends on whether docker logs finished writing before grep exited, so it is intermittent rather than constant.

Reproduced directly:

$ bash -c 'set -o pipefail; yes x | head -50000 | grep -q x; echo $?'
141

Fix

Capture the logs with redirection rather than a pipe, then grep the file, which cannot SIGPIPE. The same loop retries for up to 20 seconds, which also covers Docker's log flush lagging the first successful HTTP response. On failure it prints the tail of the file it actually searched, so the next occurrence is diagnosable from the log alone.

Verified: the fixed pattern exits 0 on a log that contains the line, and the original pattern still reproduces 141.

Scope

Only one | grep -q under pipefail exists in the workflows (ns8-module.yml greps a file directly and is unaffected). No application code changes.

The step runs under `set -euo pipefail` and tests the restart with

    docker logs calnode 2>&1 | grep -q "DEK unwrapped from keystore"

`grep -q` exits as soon as it matches, so `docker logs` takes SIGPIPE and the
pipeline returns 141 under pipefail. The assertion therefore fails even though
the line was found. It is timing dependent, which is why it was intermittent:
the failure dump in the same run printed the very line the assertion missed.
Reproduced directly: `yes x | head -50000 | grep -q x` exits 141.

This blocked a Dependabot PR that changes nothing related (#208, codeql-action).

Capture the logs to a file with redirection, then grep the file, which cannot
SIGPIPE. The same loop also retries for up to 20 seconds, covering Docker's log
flush lagging the first successful HTTP response. The failure path now prints
the tail of the log it actually searched.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • arm64-smoke restart assertion — replaces docker logs | grep -q with capture-to-file + grep, plus a bounded 20×1s retry, so a successful match is no longer turned into exit 141 by SIGPIPE under set -o pipefail; the failure path now dumps the tail of the file it actually searched.

I reproduced the original 141 and confirmed the file-based grep exits 0, and the workflow YAML still parses. The assertion stays meaningful: first boot logs fresh install — generating DEK…, and only the post-restart path logs DEK unwrapped from keystore (primary) (internal/keyvault/keyvault.go:104, :120), so it still proves the persisted key was reused rather than matching stale output.

ℹ️ Same early-exit-under-pipefail pattern elsewhere

Nothing to change in this PR. The body's sweep ("only one | grep -q under pipefail") is accurate, but the same root cause — an early-exiting consumer closing the pipe under pipefail — also sits in .github/workflows/release-verify.yml:159: gh api --paginate … | head -1 | tr -d '\r' inside a set -euo pipefail script. It only trips when a later paginated page emits a second match, so it is much rarer than this one, but it would fail the same silent, intermittent way.

Technical details
# Sibling SIGPIPE site in release-verify.yml

## Affected sites
- `.github/workflows/release-verify.yml:159` — `tag=$(gh api "repos/.../tags" --paginate --jq "… | first // empty" | head -1 | tr -d '\r')` runs under `set -euo pipefail`; `head -1` exits after one line, and a later paginated write from `gh` raises SIGPIPE → pipeline 141 → `set -e` aborts the step.

## Required outcome
- Out of scope for #211; track separately if you want the class fully swept.

## Suggested approach (optional)
- Same shape as this PR: capture to a temp file, then `head -1` the file.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash | 𝕏

@shockalotti
shockalotti merged commit b76ac1a into main Nov 1, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant