Repository navigation
perf(sessions): skip settled Codex directory fingerprints - #4815
ScriptedAlchemy wants to merge 9 commits into
Conversation
|
|
Follow-up a1f2a63f — Refs #4756, partial Codex discovery CPU/correctness repair. Additional cause: During the coarse-clock window, Fix: First detect changed native identity in both settled and unsettled windows. Only when the identity is equal and the clock remains unsettled do we compare entry-name fingerprints; an older settled witness without an entry fingerprint remains conservatively invalidated when appropriate. No persistence contracts or worker limits changed. Regression: A new Unix filesystem test creates a real Codex rollout directory, verifies the coarse-clock path and identical entry fingerprints, then models a previously observed different native identity and requires invalidation. The assertion fails under the prior implementation. The already-added settled-directory and new-rollout checks remain. Verification: Reviewed the exact commit diff (two files, one parent, non-force exact-head fast-forward). Hosted Bazel CI #42482223482 is queued. There is no nonvacuous Rust/Bazel test result or incident-scale macOS CPU/write measurement yet; draft workflows may skip Rust test jobs. Do not merge or claim full issue acceptance. This PR stays draft and #4756 stays open; durable mid-scan restart continuation remains unresolved. #4793, protected #4776/Kev drafts, release #4763, and other owners' branches were untouched. |
|
Refs #4756 — exact-head Bazel gate repair, not incident acceptance. The preceding CI run #42482223482 failed New exact-head CI #42486957234 is queued. Draft workflow Rust tests are not a validated pass; no local Bazel checkout/toolchain or affected macOS performance environment is available. This PR remains draft, and #4756 remains open: durable mid-discovery restart continuation and the reported four-hour CPU/write-cost outcomes are still unproven. Protected #4776/Kev drafts, release #4763, and other owners' branches are untouched. |
Refs #4756. Keep the native directory change-time in the negative metadata witness rather than a fresh per-sample Unvouched nonce. An unsettled equal stat still requires a name fingerprint; the first settled probe forces revalidation of an unsettled witness. Add deterministic regression for repeated filesystem probes.
Refs #4756. Exercise the raw-vs-directory identity distinction on a real filesystem bucket and use a valid JSONL fixture.
|
Refs #4756 — published correction to the hot-directory CPU path (partial fix, not incident acceptance). I found a correctness/performance regression in the preceding #4815 optimization by tracing Published 6346d7ba and test refinement 12a0e824 with exact-head, non-force fast-forwards. Directory change detection only now hashes a stable raw native change time when the stamp is Unvouched, rather than its process-random token. The existing unsettled-entry fingerprint remains mandatory when the stat is unchanged; the transition to a settled stamp still invalidates the earlier unsettled witness. File-content identity and its conservative nonmatching Unvouched semantics are unchanged. The regression uses a real filesystem Codex bucket: two distinct Unvouched tokens must differ under file-content identity but agree under the directory's metadata-change identity; a settled transition must still invalidate it. Independent commit comparisons verified only the Codex source and its targeted regression changed; no other owner, CI policy, or release change. Exact-head Bazel CI is queued. Draft-PR CI may skip Rust execution, and the environment has no Bazel checkout, so nonvacuous Rust tests and incident-scale CPU/write-latency outcomes are not yet proven. Keep this PR draft and #4756 open; durable mid-scan restart continuation and the 125–160 ms write costs remain unresolved. |
Refs #4756. The rotating idle probe can revisit the same hot buckets already checked in the same pass. Advance the rotation without a second stat or unsettled directory fingerprint; preserve urgent hot change detection and the complete rotation cycle. Add a filesystem regression that detects a new rollout when rotation starts on the hot prefix.
|
Refs #4756 — focused idle-discovery CPU reduction (partial). Published The added real-directory regression covers a new rollout appearing in a hot bucket while the rotating cursor starts there; it requires the earlier hot check to invalidate the complete frontier and subsequent bounded passes to discover the rollout. Only the Codex discovery source and test file changed. The commit diff was separately inspected after publication. Exact-head CI #42501909517 is queued. This is not a claim of a nonvacuous Bazel Rust pass or incident-scale improvement. Draft-PR CI skips Rust jobs, and a local Bazel checkout is unavailable. Durable mid-scan restart continuation, the 4-hour CPU plateau, and 125–160 ms write latency are still unproven; keep this PR draft and issue #4756 open. Related #4793 and #4806, held #4776/Kev PRs, release #4763, and other owner branches remain untouched. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🔍 Devin Review: 2 flags
Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
|
Refs #4756 — new full-CI verification blocker (no source change). The first non-draft exact-head Bazel CI #42505092291 compiled this branch and passed The Codex unit-test pass establishes targeted behavior on this exact head, not durable mid-scan process-restart continuation or the reported multi-hour CPU/write improvement. Do not merge or close #4756 without further proof; no other owner/CI/release paths were changed. |
|
Refs #4756 — independently confirmed existing master CI blocker (not a source fix). I compared the completed exact- This PR additionally fails PR remains an unmerged partial |
|
Refs #4756 — newly verified full-CI timeout blocker; no source change. I read the current The earlier Codex unit/session integration/Clippy passes remain valid; this timeout means the full |
Repair three hosted automation_runner_test failures by supplying the configured executable identity required by the production prefilter. Use the same identity for the prefilter and retained runner, preserving manual-schedule, exact-ledger, live-lock and lock-race assertions. Add a production-prefilter regression for unconfigured refusal, deduplication and same-occurrence recovery after binding, with zero backend calls and no effectful terminal. Do not relax backend admission, change timeouts, or alter the graph publication changes. Refs #4532
Refs #4756 — partial historical Codex directory-scan CPU reduction
A Codex discovery pass currently calls
codex_directory_witnessbefore enumerating each directory. That helper always runs an additionalread_dir, collects every entry name into aVec, and sorts the names for a fingerprint. For settled historical directories, the idle change probe uses only the native change-time/stat digest and never consumes that fingerprint. Reset/restart catch-up thus repeatedly performs an unnecessary full directory enumeration and sort outside the bounded candidate-walk budget.Focused source change
A new Unix behavioral test creates a real dated Codex bucket, establishes a settled witness, verifies the entry-name fingerprint is omitted and unchanged state is accepted, then creates a new rollout and requires change detection. Existing Codex epoch, recent-first, directory-change and restart tests remain applicable.
Evidence and limitations
Source-level review establishes the eliminated duplicate enumeration for settled buckets. No TraceDecay end-to-end CPU or transaction benchmark, and no nonvacuous Bazel Rust pass is claimed: this execution environment has neither a checkout nor Bazel, and hosted draft-PR CI may skip Rust testing. The regression must be run via Bazel and reviewed before merge.
This is a partial mitigation, not
Fixes #4756. The durableCodexDiscoveryFrontierstill lacks a serialized mid-scan directory position: process-retainedReadDiriterators are lost on restart, so repeated traversal of an unfinished history remains. The 4+ hour re-import, 125–160 ms profile writes, multi-provider continuation, and end-to-end bounded-write targets are unproven. Keep #4756 open. The separate projection-queue repair is in #4793. Both issue-specific PRs are temporarily ready for review so the repository's normally skipped Bazel Linux Rust and Clippy jobs can execute; this does not authorize a merge or claim incident acceptance. The exact-head verification run for this PR is #42505092291. Protected #4776 and Kev/decision-model PRs remain draft, and release #4763 is read-only.Changes are restricted to the Codex discovery source and its test module. No CI policy, dependency, release, infrastructure, production, credential, or other owner branch changes.