Repository navigation
fix(retention): queue writer admission after bounded deferrals - #4813
ScriptedAlchemy wants to merge 14 commits into
Conversation
Retention fast probes never join the daemon writer queue. Under sustained store-scoped writers, even a fair RwLock cannot admit the maintenance pass. After two per-store nonqueued refusals, join the fair writer queue with cooperative shutdown cancellation; reset/prune the counters on admission. Exercise bounded deferrals, progress with competing store writers, and cancellation of the queued acquisition in native Rust tests. Refs #4616
|
Drive the existing registered-store production maintenance tick through two real writer refusals and then a queued admission, with six durable code-index generations and a concurrent unrelated store writer. Verify that the writer release enables the same tick to collect five superseded generations. Refs #4616
A shutdown cancellation during an already-admitted retention operation must not drop journaled work that can still own blocking store clients. Separate the cancellable writer wait from the critical section, preserving cooperative operation completion and the existing writer lease after admission. Add a native Rust test that cancels after admission and requires the critical section to complete and release the gate. Refs #4616
The old finite competitor burst could finish before maintenance acquired the daemon gate. Keep unrelated store writers active through the queued retention admission, asserting churn both before and afterward and only stopping writers after the admission proof. This remains a focused gate regression; sealed-artifact N>3 acceptance requires its own native test. Refs #4616
Refuse already-cancelled retention before either fast or queued admission, without interrupting operations that already own the writer gate. Start ongoing writer churn before queueing exclusive retention. Explicitly poll queue admission and a later store writer instead of waiting for later readers to make progress behind the exclusive waiter. Check that cancelling the waiter admits its follower while the original shared holder is alive. Add cancellation-before-admission coverage and restore the native runtime import required by the real maintenance-tick regression. Refs #4616 Bazel/native execution remains required; no Cargo or CI changes.
|
Published review repair 58a76567, directly on this PR via expected-old-head Three concrete corrections in
Cancellation coverage is also strengthened: cancelling a queued retention request must admit a following store writer while the original shared holder remains alive, with no operation construction. Cancelling after admission must retain exclusion until the running operation completes. Validation is source/diff review only, not a native test pass. The exact commit changes one file (155 additions, 58 deletions, largely behavioral regression repair); other PR branches and mixed draft #4776 are unchanged. This environment has no repository checkout/Bazel toolchain, and no Cargo commands were run. Exact-head CI 42463150166 is queued. Required: |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Use a fair store-local maintenance lane shared by owner/content writers, while retaining daemon-wide exclusivity for truly profile-wide work. Queue verified project retention only against that project's writers and preserve independent project opens during a long running sync. Retain nonqueued daemon-wide probes for session work that lacks proven per-store identity. Exercise same-store fairness, independent writer progress, cancellation and a real registered-project GC tick. Refs #4616
The store-local writer admission regression borrowed a temporary WriterScope across a pinned asynchronous acquisition. Native Bazel Rust compilation fails with E0716 at writer_gate.rs:401; retain the exact scope in a binding until that future completes. Existing behavioral regressions continue proving fairness and cross-project isolation. Refs #4616
|
Refs #4616 — published 5c37cde directly to #4813 via exact old head Root cause fixed: exact-head Linux-fleet and Clippy CI both failed native Rust compilation with E0716 at Previous CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42484799429 — both blocking Clippy and fleet failure were caused by this E0716 compiler diagnostic. New exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42494703075 (queued, not a passing native Bazel result). Bazel and native macOS are unavailable in this execution environment; no Cargo was run. No changes to #4776, model PRs, #4763, infra or concurrent owner branches. Issue remains open; this only restores the fairness regression's ability to compile, not the remaining acceptance criteria on |
Repair three hosted automation_runner_test failures by supplying the configured executable identity required by the production prefilter. Use the same identity for the prefilter and retained runner, preserving manual-schedule, exact-ledger, live-lock and lock-race assertions. Add a production-prefilter regression for unconfigured refusal, deduplication and same-occurrence recovery after binding, with zero backend calls and no effectful terminal. Do not relax backend admission, change timeouts, or alter the graph publication changes. Refs #4532
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Preserve the failed-mount fixture repair from 4984a54 byte-for-byte in its own reviewable change above the shared scheduler repair #4821. The rollback journey uses the existing session-retrieval opener, checks the authoritative session path, and verifies both reopened projects bind their own session stores. No deadline or assertion is removed, and the malformed graph must still refuse admission. This separates CI fixture work from the retention-specific #4798 review without reverting fixes, rewriting history, or editing #4813. It is not evidence that the affected-host retention CPU incident is resolved. Refs #4787.
Join the reviewed #4821 and #4827 prerequisite ancestry without changing the existing #4798 tree. Both prerequisite file blobs already match this head exactly. Preserve all retention source, behavioral tests, and concurrent history while allowing a focused six-file retention review above the prerequisites. Do not merge into the prerequisite branch. Land prerequisites in master, retarget, revalidate, and retain the issue acceptance hold. No changes from #4813, held #4776, or decision-model work are moved. Refs #4787
|
Ownership coordination from #4787/#4798: the earlier claim that filename overlap itself blocked integration was too broad. I inspected your current maintenance writer-admission patch against the two #4798 outcome-log hunks; they are at separate locations. No writer-gate, branch-admin, maintenance-admission or transport-fixture code from this PR was imported or altered. The identical scheduler jobs.rs repair is already owned by #4821 (blob |
Root cause
Refs #4616 — partial sustained-writer admission repair, not full sealed-generation retirement.
The maintenance coordinator calls
StoreAdministration::try_with_writerfor every store unit. That always attempts an instantaneous, process-wide exclusive lock without ever joining the queue. While store-scoped content/owner work repeatedly takes shared admission, retention can lose every fast probe even though Tokio's underlyingRwLockis fair to queued writers. Merely rotating the store window does not repair admission starvation.Focused change
data_rootas owner/content writers. Owner and content share the same new per-store RwLock in read mode; maintenance holds its write guard. The daemon gate remains shared during local retention so an unrelated project can still open and sync. The one store-identity-less session pass keeps only a nonqueued daemon-global probe; global maintenance operations are unchanged.Files:
crates/tracedecay/src/daemon/maintenance.rs,crates/tracedecay/src/daemon/branch_admin.rs, andcrates/tracedecay-store-runtime/src/writer_gate.rsonly. Original basemasterat91c72ab. Commitsbbc7ad9c43c947ef9615ed0a0c8f24750b209099,c13a88f0a72f9b2e1db3b3ab7ae102e716ce4ab8and61fd4684c48fb553485a7c29b4d1a20d005ed25b. Created as an exact-base issue-specific branch and advanced by expected-old-headforce=falsefast-forward. No changes moved or cherry-picked from the mixed held #4776; its only overlapping file is modified at distinct hunks.Validation and remaining boundaries
Source and Rust regression are published; no repository Bazel or native macOS test pass is claimed. Run
bazel test //crates/tracedecay:unit_test //crates/tracedecay-store-runtime:unit_test --test_output=errors(confirm the exact target and a nonzero count for both regressions) and applicable CI on this head. Independently review lock ordering, shared runtime admission/park semantics, and cancellation under contention.This closes one root cause of sustained-writer admission starvation, but does not independently prove the full #4616 end-to-end test with N>3 sealed generations, unsafe-artifact isolation, Doctor guidance, and ~70-generation/15-GiB OOM-safe storage reporting. Retention can still legitimately retry on other store-specific locks. Keep #4616 open until every acceptance criterion is proven on
master. Do not merge/auto-merge Kev/decision-model PRs or mixed draft #4776. Release #4763, production/infra, credentials and other issue owners remain untouched.Scoped writer isolation correction (Nov 8, 2026)
Commit
41362a6(Refs #4616) addresses the independent-review cross-project stall: a queued daemon-exclusive waiter could block unrelated project opens during a long sync. The new fair, per-store maintenance lane avoids that global queue; all owner/content and maintenance acquisitions share the existing canonical store identity. A gate regression polls the queued maintenance and same-store follower, asserts independent project admission while the original store writer is still held, then verifies actual same-store progress. The daemon's real registered-project retention fixture now contends on its own canonical data root, not an unrelated path. Keep the original two-fast-probe/cancellation policy, including no cancellation of already admitted work. No extra dependencies or CI changes.New exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42473743469 (pending at publication). No local Bazel or native macOS execution available and no result claimed. Prior #4813 Linux fleet failure also included an external
hermetic-launcherHTTP 504 and unrelated dashboard failure; do not weaken tests, infer success, or merge until exact-head verification. This is only an issue-specific writer-admission slice, not the completed sealed-generation collector.