Skip to content

fix(keyvault): persist the platform secret, and prove arm64 natively in CI - #173

Merged
shockalotti merged 4 commits into
mainfrom
ci/arm64-native-and-key-persistence
Oct 28, 2026
Merged

shockalotti merged 4 commits into
mainfrom
ci/arm64-native-and-key-persistence

Conversation

@shockalotti

Copy link
Copy Markdown
Contributor

No behaviour change to the HTTP surface. Two commits: one real bug fix, one CI job that would have caught it.

The bug, found by running the image rather than reading the code

I ran the published arm64 image on a native ARM runner to check the multi-arch claim. It started fine and then logged this:

keyvault: CALNODE_ENCRYPTION_KEY is not set — using an ephemeral key
that will change on restart; stored secrets will become unreadable

That is the documented first install. The README one-liner is:

docker run -p 3000:3000 -v ./data:/data ghcr.io/calnode/calnode:latest

So a new user gets an ephemeral key, and at the next restart their SMTP password, Stripe keys, Zoom and LiveKit credentials and OAuth client secrets all become unreadable. The volume makes it look like data is persisted, which is the worst part: the database file survives and the secrets inside it stop decrypting.

It was worse than one code path

The only signal was the BASE_URL scheme:

BASE_URL http://   -> devMode -> ephemeral DEK, warning only
BASE_URL https://  -> fatal error, will not start

So the same gross mismatch had two mouths. Over http it stayed silent. Over https on a fresh deploy the app refused to boot. And a production deployment behind a TLS-terminating proxy (BASE_URL http, TLS at the edge) got the silent arm, because the proxy is invisible from inside.

The fix

The secret is resolved before the vault opens:

  1. CALNODE_ENCRYPTION_KEY, if set, wins and nothing is written. The NS8 module and any externally managed deployment are untouched.
  2. Otherwise an existing key file beside the database is read.
  3. Otherwise one is generated, written 0600, and reused from then on.

Beside the database is the right place. The key exists to unlock that file, and it is where backups already go. O_EXCL means two processes starting at once cannot overwrite each other; the loser re-reads the winner's key.

An empty key file is an error, not a cue to generate a new one. Generating a replacement there would quietly destroy access to the existing ciphertext, which is the exact failure this fixes.

Verified end to end with the real binary:

case result
1st start, no key generated and persisted, mode 0600, DEK stored
2nd start key loaded from file, DEK unwrapped from keystore
explicit key no key file written at all
https, no key starts (previously fatal)

8 new tests in internal/keyvault/secret_test.go: precedence, persistence, file mode, stability across restarts, empty-file refusal, reading an existing file, missing database path, creating a missing directory.

One consequence worth stating

Litestream replicates /data/calnode.db and not the key. Keeping the key outside the database is the entire point of envelope encryption, so it necessarily has to be backed up outside it too. The README now says so explicitly.

The CI job

arm64-smoke in docker-publish.yml, on a standard ubuntu-24.04-arm runner. Free and unmetered on public repos.

On push it pulls the image by the digest the build job just pushed. That gives three properties at once: no race (it runs after the publish), no stale tag (it cannot test yesterday's :edge), and no coupling by workflow display name, which is what silently broke the release gate earlier.

On pull requests it builds the image natively instead, because there is no digest to pull. That matters as much as the assertions: a gate that cannot run until it merges has not been tested at all, and this session has already produced three of those.

Assertions: runner is aarch64, image is arm64/linux, the container's own uname says aarch64, HTTP answers, /admin/ is 200, the key file exists beside the database, and after a restart the app still serves and the log shows DEK unwrapped from keystore rather than a new ephemeral key.

That last one is the regression guard tying both commits together.

Honestly

The arm64 assertion only covers the app image. It does not prove NS8 on arm64, which needs an arm64 cluster node. The app is the risky half, because it is a Go binary on a different CPU; the module is config scripts and UI files.

The documented first install was broken in two different ways depending on the
BASE_URL scheme, and it was the scheme that decided which one you got.

  BASE_URL http://  -> devMode -> ephemeral DEK, warning only
  BASE_URL https:// -> fatal error, server will not start

So `docker run -p 3000:3000 -v ./data:/data ghcr.io/calnode/calnode:latest`,
the one-liner in the README, silently generated an ephemeral key. Everything
encrypted in the database - SMTP password, Stripe keys, Zoom and LiveKit
credentials, OAuth client secrets - became unreadable at the next restart, while
the volume made it look as though the data was persisted. And a real deployment
behind a TLS-terminating proxy (BASE_URL http, TLS at the edge) took that same
silent arm, because the scheme was the only signal available.

Found by actually starting the published image on a native arm64 runner and
reading its log, not by reading the code.

The secret is now resolved before the vault is opened:

  1. CALNODE_ENCRYPTION_KEY, if set, wins and nothing is written (NS8 and any
     externally managed deployment keep working exactly as before)
  2. otherwise an existing key file beside the database is read
  3. otherwise one is generated, written 0600, and reused from then on

Beside the database is the right place: the key exists to unlock that file, and
it is where backups already go. The file is created with O_EXCL so two processes
starting at once cannot overwrite each other; the loser re-reads the winner's
key rather than proceeding with its own.

An empty key file is an error, not a cue to generate a new one. Generating a
replacement there would quietly destroy access to the existing ciphertext, which
is the exact failure this is meant to prevent.

Nothing is written when the environment variable is set, verified explicitly, so
the NS8 module's configure-module path is untouched.

Litestream replicates /data/calnode.db and not the key, so the README now says to
back the key up with the database. Keeping the key outside the database is the
whole point of envelope encryption, so it necessarily needs backing up outside
too.

Verified end to end with the real binary:
  1st start, no key  -> generated and persisted, mode 0600, fresh DEK stored
  2nd start          -> key loaded from file, "DEK unwrapped from keystore"
  explicit key       -> no key file written at all
  https, no key      -> starts instead of refusing (previously fatal)

8 new tests in internal/keyvault/secret_test.go: precedence, persistence, mode,
stability across restarts, empty-file refusal, existing-file read, missing
database path, and creating a missing directory.
The README claims a multi-arch image. The build cross-builds both architectures
under QEMU, which proves files were produced but not that the arm64 binary runs
on an ARM CPU. Every arm64 check until now was emulation.

Adds an arm64-smoke job to docker-publish.yml. It needs the build job, runs on a
standard ubuntu-24.04-arm runner, and pulls the image BY THE DIGEST THAT JOB JUST
PUSHED. Three things follow from testing the digest rather than a tag:

  - no race: it runs after the publish, so there is nothing to wait for
  - no stale tag: it cannot accidentally test yesterday's :edge
  - no coupling by workflow display name, which is what silently broke the
    release gate earlier this week

Then it asserts the things that matter and would actually fail on a broken arm64
build: the runner is aarch64, the container's own uname says aarch64, the app
answers over HTTP, /admin/ returns 200, and the encryption key file was written
beside the database rather than being ephemeral.

That last one is a regression guard for the quick-start trap fixed in the
previous commit, so the two defects cannot come back together.

Free and unmetered: standard arm64 runners cost nothing on public repositories.
Skipped on pull requests, where nothing is pushed to pull.

Verified locally on a real arm64 runner before wiring this in, using a throwaway
branch workflow: uname aarch64, image arch arm64/linux, container arch aarch64,
HTTP 302 then /admin/ 200, 26 migrations applied, server listening.
…t merges

As written, arm64-smoke only ran on pushes to main, dev and tags, because
docker-publish.yml has no other triggers. That meant the job would first execute
on the commit that merged it, which is precisely the failure this whole week has
been about: a gate whose logic looked right and had never run.

Pull requests now take a second path. Instead of pulling the digest the build job
published (there is none on a PR, since nothing is pushed), the job builds the
image natively on the arm64 runner and tests that. So every PR exercises native
ARM execution, and the job itself is proven before it reaches main.

Also adds the restart assertion, which is the real test of the key fix: restart
the container, confirm it serves again, and confirm the log contains "DEK
unwrapped from keystore" rather than a freshly generated ephemeral key.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

A generated platform secret is adopted as the legacy DEK on a pre-keystore database, which orphans the existing ciphertext and blocks re-supplying the real key; the deployment docs also still describe the old fatal-on-missing-key behaviour.

Reviewed changes

  • Platform-secret resolution — new internal/keyvault/secret.go implements ResolvePlatformSecret: CALNODE_ENCRYPTION_KEY wins, else an existing calnode.key, else a 32-byte hex secret generated and persisted 0600 beside the database. Both serve and mcp resolve before opening the vault.
  • DB path helper — db.FilePath exposes the DSN→filesystem-path mapping so the key lands beside the database it unlocks.
  • README — the quick-start now says a key is generated on first start and must be backed up with the database.
  • arm64 CI — new arm64-smoke job runs the freshly published digest on a native ubuntu-24.04-arm runner (builds natively on PRs) and asserts HTTP, /admin/, key-file persistence, and key reuse after a restart.
  • Tests — 8 cases in secret_test.go covering precedence, persistence, file mode, restart stability, empty-file refusal, existing-file reads, missing DB path, and directory creation.

⚠️ Deployment docs still describe the old missing-key behaviour

Only the README was updated for the new persistence contract. DEPLOY.md:58 still says CALNODE_ENCRYPTION_KEY is required when BASE_URL is https and that the app refuses to start without it, its troubleshooting table repeats that at DEPLOY.md:407, and CONTRIBUTING.md:50 / docs/ARCHITECTURE.md:160 still describe the ephemeral-key dev path. More consequentially, DEPLOY.md §6 calls Litestream restore automatic and never lists calnode.key among what must be backed up, so a volume rebuilt from the replica now generates a fresh key and then fails to unwrap the restored keystore.

Technical details
# Deployment docs contradict the new key behaviour

## Affected sites
- `DEPLOY.md:58` — "Required when BASE_URL is https — the app refuses to start without it"; no longer true.
- `DEPLOY.md:60` — "encryption-key enforcement" listed as a property of the https prefix; gone.
- `DEPLOY.md:407` — troubleshooting row "App won't start (prod) | Missing CALNODE_ENCRYPTION_KEY with an https BASE_URL".
- `DEPLOY.md:305-309` — "Recovery is automatic: if the volume comes up empty, entrypoint.sh restores the latest replica … before starting the app". `entrypoint.sh` restores only `calnode.db`; the key file is not in the replica.
- `CONTRIBUTING.md:50-52` — "the vault uses an ephemeral per-process key … won't survive a restart".
- `docs/ARCHITECTURE.md:160-162` — same dev caveat, plus "Production (https BASE_URL) hard-fails without the key".

## Required outcome
- The canonical deploy guide should state that a missing `CALNODE_ENCRYPTION_KEY` now generates and persists `calnode.key`, and that a volume restored without that file cannot read the ciphertext.
- The Litestream backup/restore section should list `calnode.key` as a required backup item, since restoring onto an empty volume now ends in a keystore-unwrap failure rather than a running server.
- Update the two dev-caveat notes so local-development docs match what `serve`/`mcp` actually do.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using deepseek-v4.1-flash | 𝕏

Comment thread cmd/calnode/main.go
// beside the database, so a plain `docker run` survives a restart and a
// deployment behind a TLS-terminating proxy does not silently take the
// "development" path. An explicit CALNODE_ENCRYPTION_KEY still wins.
platformSecret, secretSource, secretPath, err := keyvault.ResolvePlatformSecret(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A generated secret is handed straight to Open, which cannot distinguish it from an operator-supplied legacy key. On a pre-keystore database (crypto_keystore empty, *_enc data present) Open takes the migrateLegacy branch, adopts this random 64-hex value as the DEK, and writes the keystore row — so the existing ciphertext becomes unreadable and the original key can no longer be re-adopted through the app. Before this change an empty secret left the keystore untouched, so supplying the real key later still recovered the data.

Technical details
# A generated secret must never be adopted as the legacy DEK

## Affected sites
- `cmd/calnode/main.go:88` — passes the resolved secret (possibly generated) into `Open`.
- `cmd/calnode/mcp.go:48` — same call, same exposure.
- `internal/keyvault/keyvault.go:116` / `migrateLegacy` (`:193`) — hex-decodes any 64-char secret and adopts it as the DEK when the keystore is empty and `*_enc` data exists.

## Evidence
Reproduced against a migrated in-memory DB seeded with `smtp_pass_enc` and no `crypto_keystore` row: `Open(db, <generated 64-hex>, "", false)` succeeds via `migrateLegacy`; a subsequent `Open` with the true legacy key fails with `failed to unwrap DEK — is CALNODE_ENCRYPTION_KEY correct? cipher: message authentication failed`.

## Required outcome
- A secret whose source is `SecretFromGenerated` must not be accepted as the legacy DEK. The same reasoning already applied to an empty key file applies here: a generated replacement silently destroys access to existing ciphertext.
- Fail with an actionable error telling the operator to supply the original `CALNODE_ENCRYPTION_KEY`; because the keystore stays empty, the data remains recoverable then.
- `Open`/`migrateLegacy` needs to know the secret was generated. Both call sites already hold `secretSource`, so pass that intent through instead of re-deriving it from the string.

env:
IMAGE_REF: ${{ steps.img.outputs.ref }}
run: |
docker image inspect "$IMAGE_REF" --format 'image arch: {{.Architecture}} / {{.Os}}'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step only prints Architecture/Os; it never asserts the value, so "Confirm the image is arm64" can succeed on an amd64 image. The later native docker run only catches that by accident — because the runner has no binfmt/QEMU — which is an implicit dependency rather than the check the step claims to be. Add a test against the inspected value.

The audit workflow runs gosec over everything and requires every finding to be
annotated, which is the claim AUDIT.md makes. My new file produced 7.

Fixed rather than silenced where there was real code to improve:

  G104 x3 (unchecked errors) - the write and close paths discarded the close and
  cleanup errors. They are now joined into the returned error, and a half-written
  key file is removed rather than left looking usable.

Annotated with a reason where the finding is a false positive:

  G304 x3 - the path comes from the operator's DATABASE_URL, not from request
  data. Where the key file lives is deployment configuration.
  G101 x2 - KeyFileName and SecretFromEnv are a filename and an environment
  variable NAME. Neither is a credential; the secret itself is generated at
  runtime.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No new issues in this delta. The gosec annotations are correctly scoped and the tightened cleanup is an improvement. Approval is withheld only because the two threads from the prior review are still open and untouched by this commit.

Reviewed changes

Reviewed the one commit added since the prior pullfrog review at acc91ea.

  • Gosec suppressions — secret.go now carries scoped #nosec G101 / #nosec G304 rationales for the env-var-name constant and the operator-controlled key paths, so the claims-guard gosec step passes.
  • Key-write error handling — on a failed write or close, secret.go now joins the close and remove errors and treats fs.ErrNotExist on remove as benign, instead of discarding them.

Verification: gosec ./internal/keyvault reports 0 issues (5 #nosec), gosec ./... reports 0 issues, and go test ./internal/keyvault ./internal/db passes.

Pullfrog  | Fix it ➔ | View workflow run | Using deepseek-v4.1-flash | 𝕏

@shockalotti
shockalotti merged commit 30d6408 into main Oct 28, 2026
11 checks passed
@shockalotti

Copy link
Copy Markdown
Contributor Author

Merged as 30d6408. Review notes, including two gaps I am following up rather than blocking on.

Verified rather than taken on trust

  • The bug is real and severe. keyvault.Open with an empty secret and devMode returns ephemeralVault() after a warning, so the documented docker run loses every stored credential on the next restart while the volume makes it look persisted.
  • Precedence is implemented as described: explicit secret returns immediately and writes nothing, so the NS8 module and any externally managed deployment are untouched.
  • An empty key file is an error, not a cue to regenerate.
  • O_EXCL plus re-reading the winner on EEXIST; 0600; MkdirAll 0700; write and close failures joined and the half-written file removed.
  • db.FilePath reuses parseDSN, which strips only the scheme, so sqlite:///data/calnode.db and sqlite://./data/calnode.db both put the key beside the DB. Confirmed the container default is sqlite://./data/calnode.db with WORKDIR /, so ./data/calnode.key is inside the volume.
  • A wrong secret fails loudly: openExisting returns an error and the process exits. It never falls through to freshInstall, so a restore with the wrong key cannot silently re-key the database.
  • calnode mcp resolves the secret the same way as calnode serve, which avoids the two processes disagreeing.
  • arm64-smoke is a genuine gate: ubuntu-24.04-arm, asserts uname -m, image arch, uname inside the container, HTTP, /admin/ 200, key file present, and after a restart the log shows "DEK unwrapped" rather than a new ephemeral key. needs.build.outputs.digest is wired to the build-push-action step, so the push path is digest-pinned.

One thing worth clearing up, since a naive git diff is misleading here: the branch forked at fdd718c, before #172 merged, so git diff 9b0eaf1..HEAD appears to revert the NS8 concurrency fix. I did a real three-way merge first and confirmed the result keeps group: ns8-module-publish and cancel-in-progress: false, and the #172 CHANGELOG entry. Nothing is reverted.

Gaps, following up

  1. No CHANGELOG entry. This is a user-facing fix plus new behaviour, and the CHANGELOG is what generates release notes.
  2. DEPLOY.md still says "Recovery is automatic: if the volume comes up empty, entrypoint.sh restores the latest replica". With the key outside the database, a Litestream restore alone will not start the app. That needs a sentence in the backup section.

Worth considering separately (not doing unilaterally)

If the database already has a crypto_keystore primary row but no key file exists, ResolvePlatformSecret generates and persists a new key, and Open then fails to unwrap. The process exits loudly, so nothing is destroyed, but it leaves a wrong calnode.key on the volume — a landmine for exactly the restore the docs tell people to perform. A guard that refuses to generate when a primary row already exists ("this database is encrypted but its key is missing") would be a better failure. It is a behaviour change in the resolution path, so it is your call.

raccoder-com pushed a commit to raccoder-com/calnode that referenced this pull request Oct 29, 2026
Calnode#173 made the platform secret persist beside the database when it is not
supplied in the environment, which fixes a real silent data-loss bug. Two
documentation gaps were left:

1. No CHANGELOG entry. This is a user-facing fix plus new behaviour, and the
   CHANGELOG is what generates release notes. Recorded under Unreleased,
   describing what actually happens rather than the ideal: a restore missing
   the key generates one and then refuses to start, so the data is never
   re-keyed, and the generated file has to be replaced.

2. DEPLOY.md still says recovery is automatic when the volume comes up empty.
   That is no longer the whole story for deployments without
   CALNODE_ENCRYPTION_KEY: Litestream replicates the database, not calnode.key,
   so restoring the replica alone will not start. Added the caveat, the
   recovery step for a failed restore, and the note that env-key deployments
   are unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant