[ScriptedAlchemy]

Refs #4533

Scope: reduce redundant opens, not issue completion

This PR addresses two production reopen paths. It does not complete the packed-storage cutover: cold indexing still creates and verifies one immutable segment-<sha256>.json per distinct segment. No large-repository endpoint-security improvement or full #4533 acceptance is claimed. Related graph-pass work remains in #4796.

Staged reads (Unix/macOS)

The preparation reader retains an eight-entry handle LRU instead of reopening the same segment for every Whole/Range request. Actual staged File::open attempts are counted immediately before the call. The original metadata identity is preserved across eviction and checked against the pathname and descriptor before/after reads. Detected replacement or mutation clears output, releases cached handles and permanently invalidates reads, writes and commit. Cold/pre-open and aborted-reader refusals also clear output. The separate sealed evidence-pack reader replaces its additional Unix pathname open with metadata rechecks; non-Unix handle-ID checking remains.

Existing regression coverage includes exact overlapping/interleaved ranges, counters 1/2/9/10, bounded eviction, pathname replacement before/after eviction, same-inode mutation, refusal after restoring bytes, poisoned commit rejection, cleanup with escaped readers, and a real generation committed and cold-reloaded with the same snapshot identity. The intended sixteen-reads/one-open assertion is not an executed measurement. Replacement is refused without another open, not silently reread.

Rejected-read cache admission — 4d0ad18d0d8cba52a24e29c31536dbe73ae51b5e

The follow-up review found that cold invalid ranges, wrong-size requests and missing files evicted a valid handle from a full cache before being rejected. Invalid cache-hit requests also promoted recency before validation, changing the next eviction victim and causing extra opens on later valid reads.

The reader now validates range arithmetic/addressability and the requested extent before changing LRU state. Cold path metadata is admitted before eviction and passed into the verified opener, avoiding the duplicate pre-open stat on previously observed paths. The descriptor bound, original-identity checks across opening, and post-read validation remain intact. A real OS open failure after admission can still require dropping an old descriptor to respect the bound; this repair specifically prevents pre-open request/path refusals from disturbing valid cached handles.

staged_segment_rejected_reads_preserve_full_cache_and_recency fills all eight slots with real staged files, then rejects overflowing/out-of-bounds ranges, wrong sizes and a missing cold path. Each refusal must clear prefilled output, preserve all eight cached handles and leave actual open attempts at eight. Subsequent valid reads require exact bytes and counts 9/9/10, proving the correct LRU victim rather than merely asserting that invalid reads return errors. All five pre-existing staged-reader regressions are unchanged. These are assertions awaiting execution, not measured passing results.

Linux durability batches — 32868523be33a85fb091feebc1d2b11c1b346143

The direct production caller is StagedGenerationSegmentsV1::publish in the code-index publication store. Previously DurableFileBatch::sync reopened every recorded pathname through sync_file_at, paying an additional open per segment and potentially flushing a substituted inode rather than the originally written file.

• Retain duplicated writer descriptors in the existing 32-member concurrency window. No per-segment pathname reopen is needed for durability.
• Before admitting another member to a full window, fsync that window concurrently and release its descriptors only after success. Keep metadata identities for earlier flushed windows; the final sync rejects missing, replaced or modified members rather than silently trusting the old pathname.
• Check pending member path/descriptor identity before and after the real fsync. Failed windows retain ownership and return the error; no partial window is marked successful.
• Non-Linux durability behavior, publication ordering, checksums, directory barriers, storage format and existing integration tests are unchanged.

Tradeoffs: Linux staging now performs bounded early flushes instead of deferring all barriers to the end. The batch owns at most 32 duplicated writer descriptors (in addition to the caller's handle), and retains O(segments) metadata. This is not O(1) generation storage. Wall-time and memory effects still require measurement; no budget or concurrency limit was increased.

New falsifiable durability regressions

crates/tracedecay-private-fs/src/framed_log_batch_tests.rs is wired into the existing unit-test source glob. Seven tests cover:

1. Linux kernel IN_OPEN events across 67 real written files and multiple flush windows. A positive probe must first observe one actual open. The assertion then permits exactly the 67 writer opens and zero extra opens on a repeated sync, plus exact byte readback and the descriptor bound. This is designed to distinguish the former create+reopen path from descriptor reuse; it has not yet been run.
2. Nonzero barrier counts for both early windows and the final window, through the existing fsync instrumentation (test-helpers), so merely skipping flushes cannot satisfy the open-count test.
3. Refusal of a pathname that does not belong to the supplied writer, followed by successful correct admission.
4. Same-byte inode replacement for pending and already-flushed members.
5. A missing member from an earlier flushed window.
6. A same-inode, same-size mutation after early flush, with explicit timestamp change.
7. Failed full-window flush ownership, no admission of the next member, and repeated refusal.

The existing member-only and concurrent slow-fsync integration tests remain intact. These tests are not crash/power-loss proof or a substitute for the full index/snapshot/retention journey.

Hosted Bazel repair — 688f3a64a6ed5cdf7885b1a8ccced6fcd69c99b6

At previous head 4d0ad18, Repository gates on run 42443415493 failed specifically at //crates/tracedecay-private-fs:rustfmt_check_2024: rustfmt demanded the one-line File::options().write(true).open(&path).expect("same inode") expression in the nonzero same-inode mutation regression. The exact source-formatting correction is now committed; no assertion or implementation behavior was changed in this follow-up.

CI for 688f3a6 is the required new-head evidence, not yet proved passing. Native macOS is skipped by the default workflow; independent review and nonzero focused Bazel Rust verification remain required. Do not merge or close #4533.

Immutable publication and directory-open follow-ups — 70188c5, 7184d12

• During staged publication, a canonical segment can appear between the missing-path check and rename. The old plain rename could replace that concurrent immutable name. Publication now uses the existing atomic rename_noreplace authority. An identical concurrently published file is deduplicated by exact byte comparison; a conflicting name is refused without replacing either file. The production-path regression checks same-length differing bytes, identical collision and absent destination, including both paths' remaining contents.
• Unix/macOS preparation and commit previously called Handle::from_path for each of three directories twice per generation (six endpoint-authorized pathname opens in this source path). They now use directory device/inode metadata checks without opening those directories, preserving substitution refusal and keeping the Windows handle path unchanged. A real staged-publication test substitutes and restores the segments directory and asserts a fail-closed compare-and-swap refusal followed by acceptance of the original inode.
• These are source-level open-count deltas, not measured endpoint-security wall-time improvements. Both commits were exact-head, nonforce appends to this existing PR, without touching perf(graph): avoid duplicate pathname stat and time segment opens #4796 or protected work. No compiled/pass claim is made for the new regressions.

Latest hosted CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42449904068 . New head 7184d12 awaits Bazel nonzero test execution, formatting and independent review; default native macOS remains skipped. Do not merge or close #4533 on summary-green alone.

Validation / no-merge hold

The cache-admission repair 4d0ad18 was appended with force=false and expected head 32868523be33a85fb091feebc1d2b11c1b346143, preserving both the concurrent durability work and earlier Clippy fixes. Its diff touches only the staged reader and appends one regression; durability source/tests are unchanged by this follow-up. No Cargo or direct Rust compiler/formatter invocation was used.

No passing Rust or formatting result is claimed for the current 7184d12 head. This environment has no Bazel checkout/tool and cannot resolve GitHub for a checkout. Hosted standard CI was triggered: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42449904068 . At the last inspection the run was queued and Test macOS was skipped, not passed. The connector exposes workflow inspection/reruns, but no dispatch action for native macOS/Windows.

Bazel validation to inspect for actual nonzero results:

bazel build //crates/tracedecay-private-fs:rustfmt_check_2024 //crates/tracedecay-code-index-runtime:rustfmt_check_2024
bazel test //crates/tracedecay-private-fs:unit_test --test_output=all --test_arg=framed_log::batch_tests
bazel test //crates/tracedecay-private-fs:framed_log --test_output=all
bazel test //crates/tracedecay-code-index-runtime:unit_test --test_output=all --test_arg=code_index_scheduler::publication_store::staged_segment_reads

Verify target names against the current generated BUILD before execution. Also require applicable broader Bazel tests, Clippy, native platform coverage and independent review. The original Linux run reported a separate remote-execution UNAVAILABLE: io exception; no infrastructure change or source claim is made for that failure.

Do not merge/auto-merge based solely on summary greens. This author review is not independent approval. Keep #4533 open until all acceptance is proven on master: a proper packed-storage/offset-index cutover or equivalent first-open reduction, the broader daemon open-count audit, snapshot/read/corruption/retirement proofs, and before/after open counts and wall time on the same endpoint-scanned large repository.

No release, tag, production, credentials, infrastructure or Kev/decision-model changes.

Generation manifest publication correctness and open count — df3986a, b3bc9ec

Reviewed the final immutable generation-manifest publication path. It still used NamedTempFile::persist after a missing-path probe, which can replace a generation filename installed by a concurrent writer. It also reopened the just-fsynced temporary by pathname to SHA-256 its bytes.

• Publish a new immutable generation via the existing atomic rename_noreplace filesystem authority, after closing the staging file handle. If another publisher wins, compare exact staged/canonical bytes: accept identical content and remove staging, or refuse conflicting content without overwriting the canonical file. The follow-up b3bc9ec replaces the initial tempfile::persist_noclobber attempt because that API can leave an extra hard link after interrupted publication on some platforms; the existing canonical rename avoids this.
• Hash the actual fsynced staged bytes through the existing read/write writer handle, with no second File::open; assert that the result equals the original serialized manifest digest before publication. This removes one pathname open per attempted generation manifest from this code path, a source-level count, not a measured endpoint-security speedup.
• Add generation_manifest_publication_preserves_concurrent_canonical_bytes: real named-temp files, equal-length conflicting bytes, identical-content deduplication, fresh immutable publication, canonical byte preservation and staging cleanup. The existing real staged-generation commit/cold-reload regression remains, unweakened.

Both commits were published to this same PR by exact-head, non-force updates from 7184d12 and df3986a. Bazel tests have not yet run successfully for b3bc9ec; latest hosted CI is run 42453075773, queued at inspection. Native macOS is not enabled in the default run; independent review is still absent. No Cargo, infrastructure, external delegation, releases or protected Kev changes. Do not merge/auto-merge or close #4533.

The core #4533 work is still incomplete: packed/offset-based generation storage or equivalent O(1) first-open behavior, full daemon open-count audit, correctness/retention/restore proofs and before/after measurements on the same endpoint-scanned large repository are outstanding. Refs #4533 remains partial; #4796 owns the separate graph pass.

Retained snapshot read interruption correctness — 1d021a5

The retained-generation reader used by lexical serving checked cancellation/deadline before acquiring the generation read lock, but not after a potentially slow endpoint-authorized segment open() and read. It also returned early before clearing a reused buffer. This could expose bytes from a previous read on an early refusal, or report a late success after the request deadline.

• Clear the supplied output buffer before any lock, pointer or path work.
• Recheck cancellation then deadline after the real segment I/O (including failed I/O). Refuse with the existing typed interruption and clear output instead of exposing late bytes. No additional pathname opens or altered storage format.
• Add retained_generation_read_refuses_pre_and_post_io_interruptions_without_leaking_bytes on a real committed generation: verify the SHA-256 of a retained segment, force cancellation/deadline at the pre-I/O and post-I/O checkpoints, assert typed errors and empty stale buffers, then show an uninterrupted read still returns identical bytes.

Commit appended from exact head bcb039e, with force=false; no other branch, CI configuration or existing assertions changed. The new regression has not been proven passing yet. Hosted Bazel CI for this head was queued at inspection; default native macOS remains skipped and an independent correctness review is still required. No Cargo execution. This is a correctness repair, not a claim that cold first-opens are O(1); #4533 remains open and this PR must not auto-merge.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 1d021a5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ScriptedAlchemy]

#4533 owner / sweep coordination (partial PR, Refs #4533): Please do not merge or enable auto-merge based solely on green workflow gates. This head needs a nonzero Bazel Rust regression run, relevant native hosted checks and an independent correctness review first. The full packed-generation storage / large-repository endpoint-security acceptance remains separate, so #4533 must stay open even if this reviewed partial lands. PR #4796 owns the related graph-pass path; avoid overlapping edits. No release or Kev/decision-model changes are authorized in this lane.

[ScriptedAlchemy]

#4533 owner / maintenance coordination — repair published at 262e340 on this existing PR, using an expected-head, non-force update from concurrent formatting head 21827ea. No duplicate PR, merge, or issue closure.

The repair retains original metadata for each pinned staged file and compares both path and descriptor against it before and after reads. It clears failed output even for cold/pre-open and aborted-reader failures, drops cached descriptors on poison, and evicts before opening to keep the eight-handle bound. The moved-PathBuf regression error is corrected in the rehomed test. The dedicated test module preserves prior open-count/range/replacement assertions and adds true interleaving, same-inode mutation, byte-restoration/commit refusal, cold refusals and escaped-reader cleanup using the real writer/reader.

Validation blocker remains: no passing Rust or formatting result is claimed. Current CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42435471813 . Change-scope execution has started; Repository gates, Clippy and Test Linux (fleet) are queued. Test macOS is skipped, not passed. Native coverage needs the existing workflow_dispatch; the available connector exposes reads/reruns but no dispatch action, and local Bazel is absent. The PR body records exact focused Bazel commands and native inputs. No Cargo/direct Rust invocation, infrastructure edit or external delegation was used.

The old PR-body statement that replacement returns new bytes after eleven opens was incorrect and is removed: replacement must refuse with the count still ten. These counts remain regression assertions, not executed measurements.

Do not merge/auto-merge on green summary gates alone. Obtain nonzero regression and broader correctness results, relevant native checks, and independent review. This author repair/review is not independent approval. The full packed-storage and large-repository endpoint-security acceptance remains unfinished; keep #4533 open. #4796 continues to own the separate graph-pass work. No release or Kev/decision-model changes.

[ScriptedAlchemy]

#4533 owner review update — published, still NO MERGE: Exact-head, non-force commits 262e340 (original pin identity, pre-open failure output hygiene, mutation/commit tests), b850dcb (persist first-open identity across LRU eviction + poisoned no-reopen regression), 18269c6 (real staged whole/range bytes through successful commit and cold snapshot reload regression) are now on fix/4533-staged-segment-read-opens. Tests are added but not yet executed; hosted Bazel CI for the current tip is pending, macOS native CI is not running by default, and independent review is still missing. Please do not merge/auto-merge this partial PR solely on default green CI. Fresh audit finding: stage_segment still creates one file per new segment; verify_prepared_segments invokes state_digest_file for each distinct segment (another per-file open); on Linux, DurableFileBatch::sync calls sync_file_at for every written member, which uses OpenOptions::open again per file. The full #4533 packed-storage cutover must remove those O(segments) first opens and prove durability/retention as well as repeated read behavior. PR #4796 independently owns graph-read instrumentation; do not merge overlapping change here. Refs #4533; keep issue open. No release, production, Kev/decision-model or restricted actions.

[ScriptedAlchemy]

#4533 owner repair published — do not merge yet. Reviewed the latest hosted Bazel run at previous head 103101b6 and fixed its actionable Clippy failures in commit c76238b, appended non-force with expected remote head to the existing issue-specific PR. The blocking Rust Clippy errors were doc_markdown (unquoted Handle::from_path in publication_store.rs:2162) and two assert_is_empty findings in the staged-reader regressions (staged_segment_reads.rs:240,334). Both equality assertions continue proving the output buffer is exactly empty; no test was removed, skipped or weakened. The earlier Linux fleet failure (UNAVAILABLE: io exception building tracedecay-daemon-service) is a separate remote execution failure, not evidence of a source fix. New hosted run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42441637324 — not yet green at this writing. Native macOS is skipped in default CI; independent review and nonzero focused Bazel execution remain required. The full O(1)-first-open packed-storage cutover, before/after macOS endpoint-security measurements and index/restore/snapshot acceptance remain open. No duplicates, merging, release, Kev work or infrastructure actions. Keep #4533 open and do not auto-merge #4792 on CI green alone; coordinate with #4796 graph reader work.

[ScriptedAlchemy]

#4533 owner — production fix published at 3286852 on this PR. Appended with force=false and expected remote head c76238bd05e9eb356ee4963010f1036f62f88ba4; concurrent Clippy repairs are preserved. No duplicate PR or merge.

Changed crates/tracedecay-private-fs/src/framed_log.rs: Linux durability batches retain at most 32 duplicated writer descriptors, flush full windows concurrently, and use those descriptors rather than reopening every pathname. Earlier windows retain original metadata so missing/replaced/modified members still refuse final sync. Pending path/descriptor identity is checked before and after fsync. A failed full window retains its handles and cannot be marked successful or admit another descriptor. Non-Linux durability, existing integration tests, checksums, format and publication ordering are unchanged.

Added seven real-filesystem regressions in src/framed_log_batch_tests.rs, wired into the existing Bazel unit target: positive-control kernel IN_OPEN counts across 67 files, bounded descriptors, exact byte readback, nonzero early/final barrier counts, mismatched writer/path refusal, pending/early-flushed replacement, missing/modified early files, and failed-window ownership/retry. Expected open count is 67 writer opens with no durability reopens, NOT an executed result. Existing member-only and concurrent slow-fsync integration tests remain intact.

Verified generated targets: //crates/tracedecay-private-fs:unit_test, :framed_log, and :rustfmt_check_2024; no BUILD/dependency or workflow changes. Root diff reviewed against the exact parent: changes are limited to the batch implementation/imports/test wiring and the new tests. #4796's graph-reader lane and existing staged-read code were not overwritten.

Validation remains a merge blocker. Hosted run https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442181017 is queued; its native macOS job is skipped, not passed. No passing Rust/rustfmt/Clippy result is claimed for this head. Local Bazel is absent and checkout network resolution fails; the connector has no workflow-dispatch action. No Cargo/direct Rust execution, external delegation, or infrastructure changes were used.

This removes a redundant Linux durability reopen path, not O(segments) file creation/verification. Early-flush wall time and added O(segments) metadata still require measurement. Keep #4533 open and the no-merge/auto-merge hold until nonzero Bazel results, broader correctness/native checks, independent review, and full issue acceptance are established. No release or Kev/decision-model work.

[ScriptedAlchemy]

Published the direct GitHub source repair at 4d0ad18, appended with force=false and expected remote head 32868523be33a85fb091feebc1d2b11c1b346143. The concurrent framed_log.rs durability commit and prior Clippy fixes are preserved. No duplicate PR or merge.

The full-cache bug is fixed in source: validate request ranges/addressability and named extents before evicting or promoting staged read handles. Carry the admitted pathname metadata into the verified opener instead of repeating the pre-open stat. Original identity, post-open/post-read checks, poisoning and the eight-handle bound remain. Actual OS open failures after admission may still require an eviction; malformed/pre-open-refused requests do not.

Added staged_segment_rejected_reads_preserve_full_cache_and_recency using the real preparation/writer/reader: nine staged files, eight positively exercised cached handles, overflow/out-of-bounds/wrong-size/missing-path refusals, empty output and unchanged cache on each refusal, then exact bytes and open counts 9/9/10 to prove eviction order. All five pre-existing staged-reader tests are unchanged; no source-shape acceptance or weakened assertions.

Validation is pending, not passed. Latest exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42443415493 . Last inspection: run queued, Test macOS skipped. There is no local Bazel checkout/tool, and this connector has workflow inspection/reruns but no native workflow dispatch. No Cargo or direct Rust compiler/formatter was invoked. The final diff was reviewed, but that is not execution or independent approval.

The PR description now records this fix alongside the preserved durability work. Existing sweeps must keep the no-merge/auto-merge hold until nonzero Bazel regressions, applicable broader/native checks and independent review are established. Keep #4533 open: packed generation storage, the full open-count audit and endpoint-scanned large-repository acceptance remain incomplete. #4796 graph ownership is unchanged. Refs #4533.

[ScriptedAlchemy]

#4533 owner — hosted gate repaired on the existing PR (no merge). At 4d0ad18, Repository gates failed its Bazel rustfmt_check_2024 on one line in framed_log_batch_tests.rs:214. I committed the exact required formatting correction as 688f3a6 and advanced fix/4533-staged-segment-read-opens by expected-head non-force update. No test assertion or production semantics changed in this repair. New-head hosted CI is pending; macOS is skipped in the standard workflow and no independent review/nonzero Rust test result is established. Maintenance sweeps: please preserve the no-merge/auto-merge hold. Refs #4533 remains partial: packed storage, broad open audit and endpoint-scanned before/after verification are unfinished. No changes to #4796, releases, infrastructure or Kev.

[ScriptedAlchemy]

Refs #4533 — direct engineering on existing PR #4792, no duplicate branch/PR. Two expected-head, nonforce commits published:

• 70188c58: atomic no-replace for staged immutable segment names; verified collision/dedupe/absence paths in a new real-filesystem regression. Prevents a concurrent publisher from being overwritten by a stale missing-path observation.
• 7184d12e: on Unix/macOS replace six per-generation directory Handle::from_path opens (three on prepare and three on commit validation) with dev/inode metadata identity comparisons; preserve Windows checks. Added a real staged-publication substitution/restore regression exercising compare-and-swap refusal.

Source-level delta: 6 directory opens -> 0 for that prepare/commit identity route; still no measured macOS endpoint-security wall time or full O(segments) first-open reduction. Both tests are published but not yet executed. Latest exact-head hosted CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42449904068 (pending when inspected); native macOS is skipped by the default run. Independent review and nonzero Bazel results remain missing. No Cargo, Kody/Copilot, infrastructure, protected release #4763, or Kev work.

Maintenance sweeps: do not merge/auto-merge #4792 solely because summary CI turns green. Issue #4533 remains open until the packed/offset storage cutover or equivalent verified first-open reduction, full audit, cold snapshot/restore/corruption/retention proof, native validation and same-repository endpoint-scanned measurements land on master. PR #4796 owns related graph-pass work; avoid overlapping it.

[ScriptedAlchemy]

Refs #4533 — two commits published to existing PR #4792: df3986a and current head b3bc9ec, both exact-head, non-force. Generation-manifest publication now uses the existing atomic no-replace rename and exact byte deduplication, preventing late same-name concurrent overwrite. The fsynced staged manifest is SHA-256 hashed through its original writable handle instead of reopening its pathname (one fewer source-level open() per generation publication). A production-helper test covers same-length conflicts, identical generation collision, fresh publication and staging cleanup. A follow-up replaced the initial tempfile no-clobber method with the canonical rename to prevent possible extra hard-link residue.

Not yet validated: Hosted CI https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42453075773 is queued for b3bc9ec; prior head run was canceled when the branch moved. Native macOS is skipped by the default workflow; independent review, nonzero Bazel Rust tests and endpoint-scanned before/after measurements remain unavailable. No merge/auto-merge or issue closure. This PR remains partial (Refs #4533): packed/offset-index storage, other per-item open paths and full correctness/retention/restore acceptance are still missing. Maintenance sweeps: preserve the hold; #4796 owns the graph pass. No Cargo, Kody/Copilot, release, infrastructure, production or Kev changes.

[ScriptedAlchemy]

#4533 owner update — published 1d021a5 (Refs #4533), still NO MERGE. Exact-head, non-force append from bcb039e on the existing issue PR. The retained-generation read path now clears pre-existing output before admission and rechecks cancellation/deadline after potentially slow endpoint-authorized segment I/O, refusing late bytes with typed interruptions. Added a real committed-generation regression for SHA-256 exact bytes, pre/post-I/O cancellation/deadline, empty output on refusal, and normal successful read. No graph-reader overlap with #4796, protected Kev/decision work, or release changes. Hosted CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42459848747 (queued at inspection); no nonzero Bazel Rust pass or native macOS pass claimed, and independent review remains outstanding. Sweep tasks: do not merge/auto-merge #4792 merely on summary green. Packed first-open storage, full path audit and endpoint-scanned large-repository acceptance remain incomplete; keep #4533 open.