Repository navigation
pool: rotate the usage and audit journals at 1 GiB instead of 64 MiB - #538
Merged
Merged
Conversation
A node holds unlanded usage lines only in usage.jsonl and its .1 backup, so the rotation size bounds how long a shared usage table can be down. At 64 MiB a node writing 1 MB/s lost lines after about a minute: a 60 s outage on test2 lost 23.6% of the events and a 15 s outage none. 1 GiB holds about 17 minutes at that rate.
…api reference says 1 GiB
Contributor
Author
Hardware verification: passSetup. Testbed host, PostgreSQL 17 shared usage table. Each run used a fresh data_dir and its own cell. 50 claims ran unpaced egress loops at about 2.2 MB/s of journal. PG was stopped about 40 MiB past a 64 MiB rotation boundary and kept down for 60 s. Order A, C, C, A.
Restart resume cost on C. Measured from sandboxd start to the first landed row, with a warm page cache:
That is about 3.1 s per GB, run once at restart. It blocks only the shipper; claims are not on that path. End to end. Across three restarts, all 20,534,507 produced lines landed, with 0 lost and 0 warnings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A node keeps usage lines that have not landed only in
usage.jsonland its.1backup. The rotation size therefore bounds how long the shared usage table can be down before lines are lost. At 64 MiB, a node writing 1 MB/s started losing lines after about a minute.Change
journalMaxBytes: 64 MiB →units.GiB. The usage and audit journals share this size.docs/deploy.md:data_dirrow now gives the rotation size.docs/sandboxd-api.md: the usage journal rotates at 1 GiB.Evidence (hardware, base e38c762)
Each egress connection writes about 220 B to the journal with
egress_usage_byteson, and about 100 B with it off.Database outages at 64 MiB:
At 1 GiB the same write rate holds about 17 minutes.
Cost
.1.Gates
make go-lint: 10×0 issues.(5 modules × linux + darwin, withfmt --diff).asl -forwarder=false ./...: clean on both GOOS.go test -race ./pool/ ./usage/: ok.