Repository navigation
perf(graph): avoid duplicate pathname stat and time segment opens - #4796
ScriptedAlchemy wants to merge 18 commits into
Conversation
|
Recheck cancellation and deadlines after segment reads, including errors, clear late bytes, and preserve typed interruption precedence. Add real-file regressions for cancellation, deadline expiry, truncation, and fresh-pass recovery without changing handle bounds or publication budgets. Refs #4532
|
Published source/test follow-up ed860ca. Refs #4532. Defect repaired: the pass reader checked interruption only before segment I/O. Endpoint-scanned metadata/open/read work can outlive that probe; a completed read could then return bytes after expiry, while failed I/O could be treated as source corruption. The reader now rechecks the caller's existing probe after the operation, including its error path, clears late output, and retains the original Cancelled/DeadlineExceeded result at the pass boundary. This cannot preempt an OS call already blocked in the kernel and is not a claim that cold-publication stalls are solved. Falsifiable regressions added:
Both target the previous missing post-I/O check. They are written, not yet executed: no passing/nonzero Rust test result is claimed. Existing between-read cancellation fixtures were adjusted to still interrupt between reads with the added post-I/O probe; their byte, error and open-count assertions remain intact. The GitHub update was nonforce, expected-head checked against baab1cb. Only Validation: the exact commit diff was inspected. New-head hosted CI is https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42443095332 (queued when checked). Native coverage is still For the issue/sweep owners: retain this PR as the #4532 lane; coordinated distinct-page packed storage remains #4533. Keep #4532 open until hosted/nonzero/native validation, same-host publication/open-count measurements and all acceptance are proven on master. No merge, release or Kev/decision-model change. |
Reuse the short-read-safe segment range reader for whole-file evidence verification, and reject stale bytes after missing or mismatched paths. Add real-filesystem recovery and refusal regression coverage. Refs #4532
Record File::open attempts at the verified open boundary, excluding rejected path metadata from endpoint-scanned publication telemetry. Regress invalid file types and mismatched extents without changing durability checks. Refs #4532
|
Refs #4532 — published ffc7ef6 on the existing PR branch using an expected-head, nonforce update. The graph pass's Gate status: previous head |
|
2026-11-08 current-head CI triage for Hosted run 42451186352 has now passed Repository gates (Bazel formatting), Feature gates, Clippy, Dashboard and the focused The targeted store-runtime unit-test success is meaningful hosted Bazel evidence, but it is not independent code review, the same-host Defender/CorpLink before/after graph-publication measurement, nor proof of the cold O(distinct pages) first-open reduction. That format cutover remains coordinated under #4533/#4792. Leave #4796 and #4532 open; no merge or auto-merge. No speculative push while platform CI is pending. |
Repair three hosted automation_runner_test failures by supplying the configured executable identity required by the production prefilter. Use the same identity for the prefilter and retained runner, preserving manual-schedule, exact-ledger, live-lock and lock-race assertions. Add a production-prefilter regression for unconfigured refusal, deduplication and same-occurrence recovery after binding, with zero backend calls and no effectful terminal. Do not relax backend admission, change timeouts, or alter the graph publication changes. Refs #4532
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🔍 Devin Review: 1 flag
Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
|
Published Concrete CI repairThe three scheduler failures in the Linux fleet log are in
Their fixtures passed no backend executable, so the correct production A new integration regression exercises the actual prefilter with an unconfigured backend, proves its refusal and diagnostic deduplication, then binds an identity and proves that the same due occurrence becomes admissible without cooldown, backend execution or an effectful terminal record. Verification and remaining workThe published diff is limited to Correction to the earlier CI summary: the other failure was Coordination: the deterministic scheduler-fixture repair is now on this PR; no duplicate repair branch is needed. #4532 remains open and partial, including the coordinated #4533 storage work. No merge, release, infrastructure or protected Kev/decision-model action was taken. |
The sealed proof reader checked interruption only before component I/O. If an endpoint-scanned read failed after cancellation or deadline expiry, it could report corruption instead of the observed interruption. Clear stale output before each proof read, recheck after success or failure, and retain the typed interruption without publishing a partial proof. A real partitioned-seal regression corrupts the opened range during the read, requires the late cancellation/deadline to win with a positive open, then restores the disposable fixture and proves successful verification. Refs #4532
Published segment fixtures may be read-only. Make only the disposable fixture writable before intentionally truncating it during an opened range read, so the late-interruption regression exercises the read and not a permission error. Preserve the original expected outcomes. Refs #4532
The real sealed-evidence interruption test calls the bundle verifier, but its test module omitted that private helper import. Hosted Bazel Clippy E0425 identified two unresolved calls. Restore the exact import without touching graph logic, test assertions, or CI settings. Refs #4532
Match the hosted Bazel Rust 2024 rustfmt output in the existing proof regression module. No behavior or assertions change.
Refs #4532 — partial hot-path and correctness repair. Not full issue acceptance; do not close #4532 on merge of this slice alone.
Root cause and coordination
On the reported 6.6k-file managed macOS repository, cold graph publication timed out three times at 15 minutes with >90% of sampled stack frames in
open(). Existing merged #4547 and #4725/#4742 keep up to eight authenticated segment handles per pass, eliminating interleaved reopens. However the current index stores each graph page as a distinct content-addressed file, so cold first-opens remain O(distinct pages). #4533, with ongoing PR #4792, owns the coordinated packed-storage/offset-index cutover. Do not duplicate or bypass that storage work here.Graph changes
File::open, eliminating an unnecessary secondsymlink_metadata()per cold segment. Keep opened-handle identity, extent, Windows identity, digest and retention checks.segment_open_wall_secondsandgraph_pass_wall_secondsto the existingcode_graph_segment_read_passevent alongsidesegment_open_attemptsandsegment_reads. The open counter advances only at the actual primary File::open boundary, not for rejected path/extent probes.Late sealed-evidence interruption correctness —
bd9b1a6The full-seal verification callback (distinct from the graph-row reader) previously checked cancellation/deadline only before component I/O. An endpoint-scanned open/read that failed after expiry was consequently classified as sealed corruption. Its read callback now clears stale output first, rechecks after successful or failed I/O, preserves the observed typed interruption, and never commits an interrupted proof. New real partitioned-seal regression verifies the intact generation, truncates disposable segment files after a real range handle opens, requires a nonzero open, exact Cancelled/DeadlineExceeded precedence with one late probe, restores the fixture, and proves a fresh successful verification. Commit
108b1aeensures the fixture's published read-only segments are made writable only within the disposable test before the deliberately injected truncate, so the assertion measures the intended post-open failure instead of a permission refusal. The test is published but not yet executed. This does not reduce first-opens.CI repair in
2ecbdf1Three
//crates/tracedecay:automation_runner_testscheduler tests supplied no backend identity, so the productionbackend_unconfiguredguard correctly refused them before their intended schedule/lock checks. Bind the existing test backend to a real temporary-file identity, consistently across the prefilter and retained runner. Preserve manual-schedule, exact-ledger, lock precedence, reacquisition and zero-call assertions. Add a regression proving unconfigured refusal, diagnostic deduplication, and same-occurrence recovery after binding without cooldown or effectful terminal records. No production backend guard is relaxed.Behavioral regression evidence required
Real-filesystem graph tests cover exact bytes/digest recovery, valid route fallback, fail-closed directory/manifest-size/missing-file paths with zero attempted opens, interleaved reuse and bounded eviction, failed-read LRU neutrality, previous-output clearing, short/truncated file reads, pre/post I/O cancellation/deadline precedence, and a fresh successful pass. The existing cold graph-builder regression proves row/digest parity and observes one first-open per distinct page (not O(1)). Require fresh, nonzero execution at the current head rather than inferring success from source review or earlier heads.
Current verification and remaining acceptance
43db8db1c96b63346bc6699a33700376a7e08cae, a Rust 2024 rustfmt-only successor to48b411cthat importsverify_checked_seal_bundleinto the test module. The partial implementation and its regression assertions are otherwise unchanged. Two files differ frommaster.43db8db; prior CI 42481424377 exposed the missing import corrected by48b411c. Older Linux CI 42469054316 for2ecbdf1passed 183/183 targets (22 executed, rest cached), but it predates the new seal-proof regression and cannot validate it.//crates/tracedecay-store-runtime:unit_test,//crates/tracedecay:automation_runner_test, and//crates/tracedecay:dashboard_api_test, formatting/Clippy, relevant hosted platform checks, and independent review. Never use Cargo, bypass protections, or treat zero-test execution as evidence.open()counts,segment_open_wall_seconds, total cold graph publication time, and actual graph readiness before/after; demonstrate a completed under-budget publication on master. The remaining O(distinct-page) cold first-open cost requires coordinated code-index: store and read the index in few files so endpoint security open() cost does not dominate #4533 packed storage; this partial PR cannot claim its acceptance.No budget, storage-format or release change. No production or infrastructure actions. Kev/decision-model work (including draft #4776) and release #4763 are untouched.