Repository navigation
fix(storage): stream growth measurements and preserve registry errors - #4794
ScriptedAlchemy wants to merge 32 commits into
Conversation
|
Refs #4099. Aggregate each table's dbstat payload inside a correlated subquery instead of materializing all pages for GROUP BY. Preserve the existing reader snapshot, cancellation, deadline and typed-error path. Add nonzero filesystem-backed regressions for exact payload attribution, growth, logical deletion versus physical compaction, VM-work reduction, and interruption followed by a successful retry. No retention policy or measurement budget is relaxed; full 24-hour profile acceptance remains open.
Refs #4099. Apply the exact rustfmt delta reported by the hosted Repository gates; no production logic or acceptance criteria changed.
Refs #4099. Use a value assertion so the hosted Clippy assert_is_empty policy can display an unexpected store on failure.
Refs #4099. LIKE treats the underscore in sqlite_% as a wildcard, silently excluding valid user tables such as sqlitemessages. Match the literal reserved sqlite_ prefix instead without changing the streaming page census or its cancellation and snapshot boundaries. Add a file-backed regression that requires the omitted table's nonzero payload and subsequent growth to match an independent dbstat sum while still excluding sqlite_sequence. Preserve the concurrent CI repair.
|
Published a3654ac directly to this PR through GitHub ( Production bug fixed: the census excluded Regression added: the file-backed Rust test calls the production reader, requires a nonzero Executed SQL verification: using SQLite 3.46.1 on temporary files and SQL extracted from the candidate source, the old filter omitted 1,048,581 payload bytes; the new filter reports them and subsequently reports 2,097,162 bytes after another insert. The original fixture still has exact reference parity and 6,577 versus 19,043 VM steps. Interruption returns SQLITE_INTERRUPT, and retry returns the complete exact result. These are actual SQL checks, not a Rust/Bazel test pass. The uploaded source blob Fresh exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442890231 (queued at verification). Local Bazel and Rust toolchain are unavailable. Require the real current-head Bazel/Clippy/format results before merging. This is a bounded correctness fix, not evidence that the original 153 GB incident or #4099's 24-hour master acceptance is resolved. |
Refs #4099. Align the new user-table growth assertion with the exact format expected by the Bazel rustfmt gate; no behavioral changes.
|
Published The Repository gates log showed only a formatting difference in the new On The master baseline is unchanged at |
Refs #4099. The offline registry snapshot followed symlinked WAL/SHM members and could copy arbitrarily large foreign files into scratch. Refuse nonregular members before copying, and keep physical family byte counts from charging symlink targets outside the inspected profile. Add real offline-report and family-copy regressions for symlinked source members, plus partial-coverage and size assertions for user sessions. No operator data is copied or deleted.
|
Published 1e3d62f (Refs #4099), direct to this issue-specific PR. One-file update in New root-cause finding: the offline Fix: fail closed on every nonregular/linked snapshot member via Behavioral regressions: a real on-disk SQLite registry with a symlinked WAL and SHM, each targeting a sparse 512 MiB foreign file, must reject copying before a destination is created; the end-to-end offline report must return a typed Database failure for a linked registry WAL; and an offline report with a regular 4096-byte Independent checks: manually reproduced the original This is partial correctness only; #4099 remains open until 24-hour fresh-profile, multi-store bounds and daemon-owned reclamation are proved on |
|
Published The hosted Repository gates on Fresh exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42462849057 (queued at check). The previous run's Repository gates failed only on this formatting hunk; the Linux/Clippy jobs were still queued there at review time and are not evidence of a pass. There is no local Bazel executable or supplied checkout, so I do not claim a local Rust test pass. Require passing real nonvacuous The |
Refs #4099. The SQLite snapshot enumerated sidecars before capturing its validation generation. A writer creating a WAL in between could commit new project registrations that the report then misclassified as unregistered. Capture the generation first, then inspect only regular members and validate after copying. Add a deterministic on-disk WAL race regression: stale generations must fail closed, while a fresh generation preserves the committed project. Also fix the pinned Clippy collapsible-if failure in family sizing.
Refs #4099. The registry snapshot preflight rejected symlinks, but fs::copy reopened each checked path and could follow a link swapped in after inspection, copying foreign bytes into report scratch. Open through a held parent capability with no-follow leaf policy, then copy from that file handle. A real filesystem regression swaps a checked WAL member to a foreign link and proves the copy refuses without creating a scratch member. Preserve existing CI repairs.
|
Published New independently identified correctness/safety issue: The offline SQLite family copier checked each source member with The Linux filesystem TOCTOU mechanism was independently reproduced using a regular-file probe followed by symlink replacement: ordinary copy followed the foreign target while |
Refs #4099. A no-follow open still accepts a regular-file replacement between preflight and copy, potentially copying a huge foreign WAL before post-copy generation validation rejects it. Compare the opened descriptor against checked identity and length before creating scratch; bound the copy to the inspected length plus one. Add a filesystem regression for same-sized and oversized regular-file swaps without reading the foreign bytes.
|
Published Independent bug found: The preceding no-follow fix rejects symlink replacement but still opens an ordinary file swapped into the WAL/SHM pathname after the first metadata check. Post-copy Production repair: Preserve checked metadata for each SQLite main/WAL/SHM member; compare the opened, held file descriptor's identity and inspected length before creating the scratch member (Unix device/inode; non-Unix conservative size/time checks with existing generation validation). Cap each copy at the previously measured length plus one byte and reject a concurrent size change, rather than chasing an unbounded WAL tail. No dependencies, CI policy, thresholds, retention ownership, or production data were changed. Nonvacuous Unix regression: Replaces a previously checked, 4 KiB regular WAL with a different regular inode of equal size and again with a 512 MiB sparse file. It calls the production copy helper, requires a typed Independent filesystem reproduction: On Linux, |
Refs #4099. A rollback-mode writer may spill uncommitted registration pages into global.db while the undo journal remains. Copying only the main/WAL/SHM family would turn those pages into false registry truth. Fail closed on the rollback journal before writing scratch. Add direct and end-to-end real-SQLite regressions for refusal during an open transaction and correct registration after rollback. Preserve all prior snapshot generation and no-follow protections.
|
Published focused Root cause: the offline registry reporter copies only Published repair: refuse a present rollback journal before writing snapshot scratch and propagate the existing typed report failure instead of classifying stores using an unsafe registry. Two real-SQLite Rust behavioral regressions test the lower-level snapshot refusal (including no output) and the end-to-end The exact-head non-force fast-forward preserved |
Apply eight exact Rust 2024 rustfmt differences from hosted Bazel Repository gates. No behavior, schema, or test assertion changes.
Refs #4099. Allow committed PERSIST/TRUNCATE journals while rejecting active/uncertain rollback state and verifying the journal witness across the registry snapshot copy. Cover committed, active and rolled back states with real SQLite report regressions.
Refs #4099. Apply the exact format-only correction reported by Bazel Repository gates to the committed-journal probe. No behavior or test assertion changes; preserve the existing concurrent issue-specific work.
|
Published Fresh exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42493906252 (queued at check). No local Bazel/Bazelisk executable or Rust workspace checkout is available, so Rust/Clippy/format acceptance on the new head is not claimed. Do not merge until current-head gates and review are verified. Issue #4099 still requires complete 24-hour fresh-profile, per-store bounded-growth and daemon-owned reclamation proof on |
Refs #4099. Avoid a correlated dbstat rescan per table while retaining exact payload totals, interruptible page mode and bounded table-only state. Add a many-table VM-work regression that fails under the prior query.
Refs #4099. Keep the SQLite page name ValueRef alive while its UTF-8 slice is used to update the bounded table totals.
|
Published an additional focused New root cause: the previous correlated Behavioral evidence: a 36-table real-SQLite fixture test requires the new reader's exact payload totals to match the old independent census and its measured VM work to be less than one-third of the grouped baseline. The existing overflow, WITHOUT ROWID, logical deletion/physical compaction, user-table-name and interruption/retry regressions remain. An independent SQLite 3.46.1 mechanism reproduction used 36 tables with 32-KiB payloads: 14,353 grouped, 10,525 correlated and 1,798 single-page-stream VM steps (the Rust implementation has not been run locally). No arbitrary deadline or budget was raised. Fresh exact-head CI is pending; Bazel/Bazelisk and the Rust checkout are unavailable locally, so no Rust pass is claimed. #4099 stays open pending master verification, a 24-hour fresh-profile bounded-growth run and evidence of daemon-owned reclamation. Held draft #4776 and Kev lanes, release #4763, production and other owner branches are untouched. |
Refs #4099 Apply the precise rustfmt diff from CI run 42506693304 without changing census behavior, bounds, or regression assertions.
|
Published focused CI repair for Refs #4099 in issue-specific PR #4794: Independently reread the registry generation ordering, no-follow metadata import and persisted-journal checks in |
Refs #4099 A committed PERSIST journal remains allocated but the report's registry and user-sessions family totals counted only main, WAL, and SHM bytes. Count the rollback journal without following foreign symlinks and test a real persisted journal through the offline report and footprint census.
|
Refs #4099 — exact-head CI blocker / independent gate review (2026-11-08). On current PR head The required Linux fleet gate is still red, for failures outside the three files changed by this PR:
The same three automation-runner assertions also failed on unmodified Blocker: PR #4794 cannot be represented as fully CI-passing and must not merge or claim closure while required Linux CI is red. #4099 additionally still lacks the 24-hour fresh-profile bounded-growth and daemon-owned reclamation proof on |
Repair three hosted automation_runner_test failures by supplying the configured executable identity required by the production prefilter. Use the same identity for the prefilter and retained runner, preserving manual-schedule, exact-ledger, live-lock and lock-race assertions. Add a production-prefilter regression for unconfigured refusal, deduplication and same-occurrence recovery after binding, with zero backend calls and no effectful terminal. Do not relax backend admission, change timeouts, or alter the graph publication changes. Refs #4532
|
Published focused Doctor footprint correctness fix for Refs #4099 on issue-specific PR #4794: Root cause: on The new filesystem-backed Rust regression makes a 4 KiB project WAL and 2 KiB LCM payload, imposes 1 KiB project/profile bounds, requires named degraded Doctor findings for both, and distinguishes registry-failure root state from a successfully empty registry and a confirmed deleted registered root. It uses only a disposable TempDir; no user profile or credentials touched. Rechecked against master Validation: exact-head CI is queued; local Bazel/Bazelisk are unavailable, so no Rust compilation, Clippy, formatting or test pass is claimed yet. This is a partial repair. Do not close #4099 before CI, master integration, daemon-owned reclamation and the required 24-hour fresh-profile growth evidence. #4776 and Kev/decision-model PRs remain draft and excluded from merge; release #4763 unchanged. |
Refs #4099. Validate all offline and daemon report project IDs at the common store-sampling boundary, before constructing or walking their paths. Keep direct project reports and persisted registry rows on one canonical path-segment policy. Regression proves path escape is rejected before any scan and a normal store is still measured.
Refs #4099. Keep the invalid-identity fixture list compliant with the pinned formatter.
|
Refs #4099 — published registry-path containment fix on issue-specific PR #4794: Root cause: The offline storage-report snapshot's Fix: Validate IDs using the existing Regression: The actual Published using two exact expected-old-head, non-force fast-forwards (from Verification gate: New-head hosted CI #42541449261 is queued; no local Bazel executable or Rust workspace checkout is available, so the new regression is not yet verified by Rust CI. Do not merge solely on this reproduction. The 24-hour fresh-profile bounded-growth, daemon-owned retention/reclamation and complete acceptance on |
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Audited the current table-size census at aa507c4, specifically reader/worker/table_sizes.rs. The current implementation is newer than the body summary: it initializes user tables from schema and accumulates a single SELECT name, payload FROM dbstat page walk, not one correlated SUM per table. It preserves empty/rootless tables, excludes index/internal payload from user-table totals, checks negative/overflow payloads, and propagates interruption rather than publishing partial success.
Independent executable check in this audit: SQLite 3.46.1, five synthetic schemas (0, 1, 4, 32 and 128 additional indexed tables), 45 assertions passed. Compared the literal production SQL plus equivalent accumulator against an independent grouped page reference; included WITHOUT ROWID/overflow data, rootless virtual and empty tables, insert/delete payload changes with unchanged physical page count, actual SQLite interruption, and successful retry. This is SQL/algorithm evidence only—not execution of Rust/Bazel, native macOS, or the rest of this PR's filesystem/registry paths, and not a wall-clock speedup claim.
No merge clearance is implied: current-head full applicable CI and the rest of the PR remain separate review gates. Keep #4099 open for its complete master-backed storage-growth acceptance. The stale correlated-query description should not be used as the reviewed implementation.
Refs #4099. Retain measured baselines and latest evidence across restarts, but keep deadline-based skip markers process-local so a new reader can retry after improvements. Add a real-store restart regression that reproduces the previous 24-hour suppression.
|
Published Root cause: Fix: Keep deadline-based skip state process-local ( Regression: Added a real database + maintenance-registry restart test that establishes a baseline, inserts nonzero payload, simulates a recent timeout marker, verifies the same process skips and persists its last unavailable state, then restarts and requires a new census with positive measured growth from the original baseline. Before this fix the restart would still skip and fail this test. Publication: Exact prior SHA |
Scope
Refs #4099. This remains an issue-specific partial repair, not a claim that the deleted 153 GB profile has been reconstructed or that the 24-hour master acceptance has passed.
Measurement-path root cause and repair
The actual reader-worker
TableSizescommand joined onedbstatrow per database page tosqlite_schema, then usedGROUP BYto recover one total per table. That creates a temporary grouping sorter proportional to the number of pages being measured and consumes avoidable SQL VM work inside the existing measurement deadline. Large-store measurement was doing expensive page sorting in addition to the necessary page walk.The current reader initializes one total per user table from
sqlite_schema, then walksSELECT name, payload FROM dbstatonce on the same snapshot. It includes empty and rootless user tables, excludes index and internal pages from their totals, and returns an interruption or arithmetic error before publishing results. The earlier correlated per-tableSUMat8b403d1was an intermediate implementation.The existing registry fixes are preserved: the offline report distinguishes true
global.dbabsence from metadata/read errors and refuses registry symlinks rather than importing foreign registration identities into a profile whose census cannot account for those linked bytes.Behavioral regressions
Three new filesystem-backed tests call the production census implementation:
The earlier dangling/valid-external-registry behavioral tests remain in the PR.
Validation and review
9edee1472e796e6bde5bc1564b49a452c56e75d0before editing. Published blobs match the locally inspected source. Comparison against prior head43cef92shows only the production dispatch change and the new census/test module; the concurrent formatting fix is preserved.43cef92bc121f1bf9b451de1a3b92c8e40fd624f.8b403d1found only pinned-rustfmt differences in the new test module; the exact reported formatting corrections were published asd9559c8using a non-force fast-forward with expected old head8b403d1. The new-head CI run #42437543266 is queued; the old run's Rust jobs were cancelled by supersession, not passed. Required validation on the exact head:bazel test //crates/tracedecay-rusqlite-runtime:unit_test //crates/tracedecay-maintenance:unit_test --test_output=errors bazel build //crates/tracedecay-rusqlite-runtime:rustfmt_check_2024 //crates/tracedecay-maintenance:rustfmt_check_2024Still required before closing #4099
This removes a demonstrated source of measurement work amplification; it does not make the physical page walk constant-time or prove that every large store fits its deadline. Per-family bounds, daemon-owned retention/reclamation and the fresh-profile 24-hour normal-use acceptance must still be demonstrated on
master. Keep #4099 open; keep Kev/decision-model work and mixed #4776 draft and excluded from merging.Current CI repair (679f636)
On the prior head
d9559c8, pinned Repository gates passed. Clippy failed onstorage_report.rs:2492withclippy::assert_is_empty(-D warnings): the added test usedassert!(absent.stores.is_empty()). Commit679f636replaces that with a typedassert_eq!(absent.stores, Vec::<StoreSizeReportEntry>::new()), retaining the failure check and adding value-level diagnostic output. Published via exact-headexpected_sha=d9559c8,force=false, with no concurrent commits overwritten.The earlier fleet Linux job failed with
UNAVAILABLE: io exceptionfrom the external Bazel remote executor (not a repository test assertion). The earlier Clippy job additionally observed that transient executor failure. Fresh hosted CI is pending on679f636. No Bazel executable or Rust toolchain is available locally, so the current Rust test and Clippy results remain unverified. Do not claim passing gates, merge, or #4099 closure until actual results and master acceptance exist.