Skip to content

pool: rotate the usage and audit journals at 1 GiB instead of 64 MiB - #538

Merged
CMGS merged 3 commits into
mainfrom
pool/journal-cap
Nov 9, 2026
Merged

CMGS merged 3 commits into
mainfrom
pool/journal-cap

Conversation

@CMGS

@CMGS CMGS commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

A node keeps usage lines that have not landed only in usage.jsonl and its .1 backup. The rotation size therefore bounds how long the shared usage table can be down before lines are lost. At 64 MiB, a node writing 1 MB/s started losing lines after about a minute.

Change

  • journalMaxBytes: 64 MiB → units.GiB. The usage and audit journals share this size.
  • docs/deploy.md:
    • The data_dir row now gives the rotation size.
    • The existing Outage bullet under Shared usage table now says 1 GiB per rotation and gives the per-connection write size, the ~17 min at 1 MB/s and the per-cell landing bound.
  • docs/sandboxd-api.md: the usage journal rotates at 1 GiB.

Evidence (hardware, base e38c762)

  • Each egress connection writes about 220 B to the journal with egress_usage_bytes on, and about 100 B with it off.

  • Database outages at 64 MiB:

    Outage Lines lost
    15 s 0 of 626,239
    60 s 344,953 of 1,461,988 (23.6%)
  • At 1 GiB the same write rate holds about 17 minutes.

Cost

  • Disk: up to 2 GiB per journal, counting the live file and .1.
  • Shipper: after a restart, it scans both files once to find its resume line. At 1 GiB this is up to 2 GiB of sequential read and SHA-256, in the shipper goroutine and off the claim path.
  • Claim path: no change.

Gates

  • make go-lint: 10× 0 issues. (5 modules × linux + darwin, with fmt --diff).
  • asl -forwarder=false ./...: clean on both GOOS.
  • go test -race ./pool/ ./usage/: ok.

CMGS added 3 commits November 9, 2026 21:52
A node holds unlanded usage lines only in usage.jsonl and its .1 backup, so
the rotation size bounds how long a shared usage table can be down. At
64 MiB a node writing 1 MB/s lost lines after about a minute: a 60 s outage
on test2 lost 23.6% of the events and a 15 s outage none. 1 GiB holds about
17 minutes at that rate.
@CMGS

CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Hardware verification: pass

Setup. Testbed host, PostgreSQL 17 shared usage table. Each run used a fresh data_dir and its own cell. 50 claims ran unpaced egress loops at about 2.2 MB/s of journal. PG was stopped about 40 MiB past a 64 MiB rotation boundary and kept down for 60 s. Order A, C, C, A.

  • A: 707ef2c.
  • C: 768511a. The later commits on this branch change only docs and a constant spelling.
Run Produced Landed Lost "last landed line is gone" warnings
A 1,387,579 1,124,892 262,687 (18.9%) 2
C 1,441,678 1,441,678 0 0
C 1,389,695 1,389,695 0 0
A 1,399,570 1,103,385 296,185 (21.2%) 2

Restart resume cost on C. Measured from sandboxd start to the first landed row, with a warm page cache:

Journal size First land
32 MB 0.39 s
1.04 GB 3.50 s
2.02 GB (live plus .1) 7.01 s

That is about 3.1 s per GB, run once at restart. It blocks only the shipper; claims are not on that path.

End to end. Across three restarts, all 20,534,507 produced lines landed, with 0 lost and 0 warnings.

@CMGS
CMGS merged commit 86e65f1 into main Nov 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant