Repository navigation
fix(keyvault): persist the platform secret, and prove arm64 natively in CI - #173
Conversation
The documented first install was broken in two different ways depending on the
BASE_URL scheme, and it was the scheme that decided which one you got.
BASE_URL http:// -> devMode -> ephemeral DEK, warning only
BASE_URL https:// -> fatal error, server will not start
So `docker run -p 3000:3000 -v ./data:/data ghcr.io/calnode/calnode:latest`,
the one-liner in the README, silently generated an ephemeral key. Everything
encrypted in the database - SMTP password, Stripe keys, Zoom and LiveKit
credentials, OAuth client secrets - became unreadable at the next restart, while
the volume made it look as though the data was persisted. And a real deployment
behind a TLS-terminating proxy (BASE_URL http, TLS at the edge) took that same
silent arm, because the scheme was the only signal available.
Found by actually starting the published image on a native arm64 runner and
reading its log, not by reading the code.
The secret is now resolved before the vault is opened:
1. CALNODE_ENCRYPTION_KEY, if set, wins and nothing is written (NS8 and any
externally managed deployment keep working exactly as before)
2. otherwise an existing key file beside the database is read
3. otherwise one is generated, written 0600, and reused from then on
Beside the database is the right place: the key exists to unlock that file, and
it is where backups already go. The file is created with O_EXCL so two processes
starting at once cannot overwrite each other; the loser re-reads the winner's
key rather than proceeding with its own.
An empty key file is an error, not a cue to generate a new one. Generating a
replacement there would quietly destroy access to the existing ciphertext, which
is the exact failure this is meant to prevent.
Nothing is written when the environment variable is set, verified explicitly, so
the NS8 module's configure-module path is untouched.
Litestream replicates /data/calnode.db and not the key, so the README now says to
back the key up with the database. Keeping the key outside the database is the
whole point of envelope encryption, so it necessarily needs backing up outside
too.
Verified end to end with the real binary:
1st start, no key -> generated and persisted, mode 0600, fresh DEK stored
2nd start -> key loaded from file, "DEK unwrapped from keystore"
explicit key -> no key file written at all
https, no key -> starts instead of refusing (previously fatal)
8 new tests in internal/keyvault/secret_test.go: precedence, persistence, mode,
stability across restarts, empty-file refusal, existing-file read, missing
database path, and creating a missing directory.
The README claims a multi-arch image. The build cross-builds both architectures
under QEMU, which proves files were produced but not that the arm64 binary runs
on an ARM CPU. Every arm64 check until now was emulation.
Adds an arm64-smoke job to docker-publish.yml. It needs the build job, runs on a
standard ubuntu-24.04-arm runner, and pulls the image BY THE DIGEST THAT JOB JUST
PUSHED. Three things follow from testing the digest rather than a tag:
- no race: it runs after the publish, so there is nothing to wait for
- no stale tag: it cannot accidentally test yesterday's :edge
- no coupling by workflow display name, which is what silently broke the
release gate earlier this week
Then it asserts the things that matter and would actually fail on a broken arm64
build: the runner is aarch64, the container's own uname says aarch64, the app
answers over HTTP, /admin/ returns 200, and the encryption key file was written
beside the database rather than being ephemeral.
That last one is a regression guard for the quick-start trap fixed in the
previous commit, so the two defects cannot come back together.
Free and unmetered: standard arm64 runners cost nothing on public repositories.
Skipped on pull requests, where nothing is pushed to pull.
Verified locally on a real arm64 runner before wiring this in, using a throwaway
branch workflow: uname aarch64, image arch arm64/linux, container arch aarch64,
HTTP 302 then /admin/ 200, 26 migrations applied, server listening.
…t merges As written, arm64-smoke only ran on pushes to main, dev and tags, because docker-publish.yml has no other triggers. That meant the job would first execute on the commit that merged it, which is precisely the failure this whole week has been about: a gate whose logic looked right and had never run. Pull requests now take a second path. Instead of pulling the digest the build job published (there is none on a PR, since nothing is pushed), the job builds the image natively on the arm64 runner and tests that. So every PR exercises native ARM execution, and the job itself is proven before it reaches main. Also adds the restart assertion, which is the real test of the key fix: restart the container, confirm it serves again, and confirm the log contains "DEK unwrapped from keystore" rather than a freshly generated ephemeral key.
There was a problem hiding this comment.
Important
A generated platform secret is adopted as the legacy DEK on a pre-keystore database, which orphans the existing ciphertext and blocks re-supplying the real key; the deployment docs also still describe the old fatal-on-missing-key behaviour.
Reviewed changes
- Platform-secret resolution — new
internal/keyvault/secret.goimplementsResolvePlatformSecret:CALNODE_ENCRYPTION_KEYwins, else an existingcalnode.key, else a 32-byte hex secret generated and persisted0600beside the database. Bothserveandmcpresolve before opening the vault. - DB path helper —
db.FilePathexposes the DSN→filesystem-path mapping so the key lands beside the database it unlocks. - README — the quick-start now says a key is generated on first start and must be backed up with the database.
- arm64 CI — new
arm64-smokejob runs the freshly published digest on a nativeubuntu-24.04-armrunner (builds natively on PRs) and asserts HTTP,/admin/, key-file persistence, and key reuse after a restart. - Tests — 8 cases in
secret_test.gocovering precedence, persistence, file mode, restart stability, empty-file refusal, existing-file reads, missing DB path, and directory creation.
⚠️ Deployment docs still describe the old missing-key behaviour
Only the README was updated for the new persistence contract. DEPLOY.md:58 still says CALNODE_ENCRYPTION_KEY is required when BASE_URL is https and that the app refuses to start without it, its troubleshooting table repeats that at DEPLOY.md:407, and CONTRIBUTING.md:50 / docs/ARCHITECTURE.md:160 still describe the ephemeral-key dev path. More consequentially, DEPLOY.md §6 calls Litestream restore automatic and never lists calnode.key among what must be backed up, so a volume rebuilt from the replica now generates a fresh key and then fails to unwrap the restored keystore.
Technical details
# Deployment docs contradict the new key behaviour
## Affected sites
- `DEPLOY.md:58` — "Required when BASE_URL is https — the app refuses to start without it"; no longer true.
- `DEPLOY.md:60` — "encryption-key enforcement" listed as a property of the https prefix; gone.
- `DEPLOY.md:407` — troubleshooting row "App won't start (prod) | Missing CALNODE_ENCRYPTION_KEY with an https BASE_URL".
- `DEPLOY.md:305-309` — "Recovery is automatic: if the volume comes up empty, entrypoint.sh restores the latest replica … before starting the app". `entrypoint.sh` restores only `calnode.db`; the key file is not in the replica.
- `CONTRIBUTING.md:50-52` — "the vault uses an ephemeral per-process key … won't survive a restart".
- `docs/ARCHITECTURE.md:160-162` — same dev caveat, plus "Production (https BASE_URL) hard-fails without the key".
## Required outcome
- The canonical deploy guide should state that a missing `CALNODE_ENCRYPTION_KEY` now generates and persists `calnode.key`, and that a volume restored without that file cannot read the ciphertext.
- The Litestream backup/restore section should list `calnode.key` as a required backup item, since restoring onto an empty volume now ends in a keystore-unwrap failure rather than a running server.
- Update the two dev-caveat notes so local-development docs match what `serve`/`mcp` actually do.deepseek-v4.1-flash | 𝕏
| // beside the database, so a plain `docker run` survives a restart and a | ||
| // deployment behind a TLS-terminating proxy does not silently take the | ||
| // "development" path. An explicit CALNODE_ENCRYPTION_KEY still wins. | ||
| platformSecret, secretSource, secretPath, err := keyvault.ResolvePlatformSecret( |
There was a problem hiding this comment.
A generated secret is handed straight to Open, which cannot distinguish it from an operator-supplied legacy key. On a pre-keystore database (crypto_keystore empty, *_enc data present) Open takes the migrateLegacy branch, adopts this random 64-hex value as the DEK, and writes the keystore row — so the existing ciphertext becomes unreadable and the original key can no longer be re-adopted through the app. Before this change an empty secret left the keystore untouched, so supplying the real key later still recovered the data.
Technical details
# A generated secret must never be adopted as the legacy DEK
## Affected sites
- `cmd/calnode/main.go:88` — passes the resolved secret (possibly generated) into `Open`.
- `cmd/calnode/mcp.go:48` — same call, same exposure.
- `internal/keyvault/keyvault.go:116` / `migrateLegacy` (`:193`) — hex-decodes any 64-char secret and adopts it as the DEK when the keystore is empty and `*_enc` data exists.
## Evidence
Reproduced against a migrated in-memory DB seeded with `smtp_pass_enc` and no `crypto_keystore` row: `Open(db, <generated 64-hex>, "", false)` succeeds via `migrateLegacy`; a subsequent `Open` with the true legacy key fails with `failed to unwrap DEK — is CALNODE_ENCRYPTION_KEY correct? cipher: message authentication failed`.
## Required outcome
- A secret whose source is `SecretFromGenerated` must not be accepted as the legacy DEK. The same reasoning already applied to an empty key file applies here: a generated replacement silently destroys access to existing ciphertext.
- Fail with an actionable error telling the operator to supply the original `CALNODE_ENCRYPTION_KEY`; because the keystore stays empty, the data remains recoverable then.
- `Open`/`migrateLegacy` needs to know the secret was generated. Both call sites already hold `secretSource`, so pass that intent through instead of re-deriving it from the string.| env: | ||
| IMAGE_REF: ${{ steps.img.outputs.ref }} | ||
| run: | | ||
| docker image inspect "$IMAGE_REF" --format 'image arch: {{.Architecture}} / {{.Os}}' |
There was a problem hiding this comment.
This step only prints Architecture/Os; it never asserts the value, so "Confirm the image is arm64" can succeed on an amd64 image. The later native docker run only catches that by accident — because the runner has no binfmt/QEMU — which is an implicit dependency rather than the check the step claims to be. Add a test against the inspected value.
The audit workflow runs gosec over everything and requires every finding to be annotated, which is the claim AUDIT.md makes. My new file produced 7. Fixed rather than silenced where there was real code to improve: G104 x3 (unchecked errors) - the write and close paths discarded the close and cleanup errors. They are now joined into the returned error, and a half-written key file is removed rather than left looking usable. Annotated with a reason where the finding is a false positive: G304 x3 - the path comes from the operator's DATABASE_URL, not from request data. Where the key file lives is deployment configuration. G101 x2 - KeyFileName and SecretFromEnv are a filename and an environment variable NAME. Neither is a credential; the secret itself is generated at runtime.
There was a problem hiding this comment.
ℹ️ No new issues in this delta. The gosec annotations are correctly scoped and the tightened cleanup is an improvement. Approval is withheld only because the two threads from the prior review are still open and untouched by this commit.
Reviewed changes
Reviewed the one commit added since the prior pullfrog review at acc91ea.
- Gosec suppressions —
secret.gonow carries scoped#nosec G101/#nosec G304rationales for the env-var-name constant and the operator-controlled key paths, so theclaims-guardgosec step passes. - Key-write error handling — on a failed write or close,
secret.gonow joins the close and remove errors and treatsfs.ErrNotExiston remove as benign, instead of discarding them.
Verification: gosec ./internal/keyvault reports 0 issues (5 #nosec), gosec ./... reports 0 issues, and go test ./internal/keyvault ./internal/db passes.
deepseek-v4.1-flash | 𝕏
|
Merged as 30d6408. Review notes, including two gaps I am following up rather than blocking on. Verified rather than taken on trust
One thing worth clearing up, since a naive Gaps, following up
Worth considering separately (not doing unilaterally) If the database already has a |
Calnode#173 made the platform secret persist beside the database when it is not supplied in the environment, which fixes a real silent data-loss bug. Two documentation gaps were left: 1. No CHANGELOG entry. This is a user-facing fix plus new behaviour, and the CHANGELOG is what generates release notes. Recorded under Unreleased, describing what actually happens rather than the ideal: a restore missing the key generates one and then refuses to start, so the data is never re-keyed, and the generated file has to be replaced. 2. DEPLOY.md still says recovery is automatic when the volume comes up empty. That is no longer the whole story for deployments without CALNODE_ENCRYPTION_KEY: Litestream replicates the database, not calnode.key, so restoring the replica alone will not start. Added the caveat, the recovery step for a failed restore, and the note that env-key deployments are unaffected.

No behaviour change to the HTTP surface. Two commits: one real bug fix, one CI job that would have caught it.
The bug, found by running the image rather than reading the code
I ran the published arm64 image on a native ARM runner to check the multi-arch claim. It started fine and then logged this:
That is the documented first install. The README one-liner is:
So a new user gets an ephemeral key, and at the next restart their SMTP password, Stripe keys, Zoom and LiveKit credentials and OAuth client secrets all become unreadable. The volume makes it look like data is persisted, which is the worst part: the database file survives and the secrets inside it stop decrypting.
It was worse than one code path
The only signal was the
BASE_URLscheme:So the same gross mismatch had two mouths. Over http it stayed silent. Over https on a fresh deploy the app refused to boot. And a production deployment behind a TLS-terminating proxy (BASE_URL http, TLS at the edge) got the silent arm, because the proxy is invisible from inside.
The fix
The secret is resolved before the vault opens:
CALNODE_ENCRYPTION_KEY, if set, wins and nothing is written. The NS8 module and any externally managed deployment are untouched.0600, and reused from then on.Beside the database is the right place. The key exists to unlock that file, and it is where backups already go.
O_EXCLmeans two processes starting at once cannot overwrite each other; the loser re-reads the winner's key.An empty key file is an error, not a cue to generate a new one. Generating a replacement there would quietly destroy access to the existing ciphertext, which is the exact failure this fixes.
Verified end to end with the real binary:
0600, DEK storedDEK unwrapped from keystore8 new tests in
internal/keyvault/secret_test.go: precedence, persistence, file mode, stability across restarts, empty-file refusal, reading an existing file, missing database path, creating a missing directory.One consequence worth stating
Litestream replicates
/data/calnode.dband not the key. Keeping the key outside the database is the entire point of envelope encryption, so it necessarily has to be backed up outside it too. The README now says so explicitly.The CI job
arm64-smokeindocker-publish.yml, on a standardubuntu-24.04-armrunner. Free and unmetered on public repos.On push it pulls the image by the digest the build job just pushed. That gives three properties at once: no race (it runs after the publish), no stale tag (it cannot test yesterday's
:edge), and no coupling by workflow display name, which is what silently broke the release gate earlier.On pull requests it builds the image natively instead, because there is no digest to pull. That matters as much as the assertions: a gate that cannot run until it merges has not been tested at all, and this session has already produced three of those.
Assertions: runner is
aarch64, image isarm64/linux, the container's ownunamesaysaarch64, HTTP answers,/admin/is 200, the key file exists beside the database, and after a restart the app still serves and the log showsDEK unwrapped from keystorerather than a new ephemeral key.That last one is the regression guard tying both commits together.
Honestly
The arm64 assertion only covers the app image. It does not prove NS8 on arm64, which needs an arm64 cluster node. The app is the risky half, because it is a Go binary on a different CPU; the module is config scripts and UI files.