Skip to content

perf(sessions): avoid idle scans of deferred relation receipts - #4810

Open
ScriptedAlchemy wants to merge 11 commits into
masterfrom
fix/4481-relation-recovery-idle-probe
Open

ScriptedAlchemy wants to merge 11 commits into
masterfrom
fix/4481-relation-recovery-idle-probe

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Scope — Refs #4481 (partial, not closure)

The idle daemon's retained-summary pass calls relation-receipt recovery even when every pending receipt is asleep until a future retry deadline. The existing recovery SELECT orders by created_at and SQLite chooses idx_session_relation_receipts_pending, walking the sleeping pending prefix despite finding no due rows.

Fix: Use the existing covering idx_session_relation_receipts_recovery_due through a cheap MIN(recovery_next_attempt_at) eligibility probe in the same snapshot. When there is no due work, return a truthful empty page with the persisted next deadline, without reading or joining the old deferred journal rows. Keep creation-order processing for actually due work and preserve cancellation, typed errors, and the existing delayed retry outcomes.

Pagination audit: The shared bounded_query_limit(limit) already requests limit + 1 rows for its has_more sentinel. Review found that the first patch accidentally added an additional increment, fetching limit + 2; follow-up commit b1be997 removes that double padding. The existing pagination semantics are retained and covered by the two-page regression; no pre-existing has_more defect is claimed.

Regression coverage

Extend the existing real registered-store recovery journey in crates/tracedecay-session-runtime/src/lcm_effects/tests.rs:

  • A persisted future retry is not processed, the page remains idle, and the next retry deadline is retained.
  • Two due receipt generations are recovered in distinct limit=1 pages; the first page must report has_more=true, the second false.
  • After both settle, an additional pass reports zero work and no invented backlog.
  • Existing malformed-journal and cancellation regressions remain unchanged.

Only two files change, with no schema/dependencies/configuration changes. This is separate from owner PR #4808 (LCM dirty-invalidation backoff) and peer-owned projection, storage, retention and recovery PRs.

Measured SQL evidence and verification limit

A disposable synthetic SQLite fixture containing 75,000 pending, future-deferred relation receipts, using the production index definitions, reported a median of 9.03 ms for the existing creation-order empty candidate SELECT versus 0.0051 ms for the covering-index MIN probe (25 repetitions; local host). EXPLAIN showed the former using idx_session_relation_receipts_pending and the latter using idx_session_relation_receipts_recovery_due. These are isolated query measurements, not proof of the reported macOS 170–300% idle CPU cause or an end-to-end daemon benchmark.

The query/page code was reviewed against bounded_query_limit on the exact branch head; no workspace Bazel run was available. Exact-head repository Bazel Rust tests, CI, independent review and a native macOS idle/24-hour store growth acceptance run remain required; the authoring environment has no Bazel or native daemon. No Cargo was used. Do not merge until nonvacuous Bazel tests and review pass.

#4481 stays open until idle CPU, repeated retries and bounded growth are proven against master. Held draft #4776, Kev/decision-model work and read-only release #4763 were not modified.

Probe the indexed next relation recovery deadline before reading the
creation-order journal, and detect one extra due receipt so the bounded
recovery worker continues without missing backlog.

Extend database-backed recovery regressions for deferred idle reads,
due-page continuation, and settled quiescence.

Refs #4481
@changeset-bot

changeset-bot Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: b0f6277

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

The shared bounded_query_limit helper already requests one sentinel
row for has_more. Do not pad the caller's limit a second time.

Refs #4481
The bounded query helper already fetched limit plus one before this PR.
Keep the behavior regression and describe what it actually proves.

Refs #4481
The one-row recovery pagination regression reused an applied receipt while
setting its state to pending. The actual SQLite schema refuses that mixed
state unless graph_watermark and applied_at are cleared in the same update.
Restore the valid pending transition without weakening the receipt constraint
or the two-page recovery and idle-queue assertions.

Refs #4481

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Repair three hosted automation_runner_test failures by supplying the
configured executable identity required by the production prefilter.
Use the same identity for the prefilter and retained runner, preserving
manual-schedule, exact-ledger, live-lock and lock-race assertions.

Add a production-prefilter regression for unconfigured refusal,
deduplication and same-occurrence recovery after binding, with zero
backend calls and no effectful terminal. Do not relax backend admission,
change timeouts, or alter the graph publication changes.

Refs #4532

Copy link
Copy Markdown
Owner Author

Review/ownership blocker before merging this #4481 slice: the PR head advanced from f085c0f to 1430ba1 while this issue was being checked. The new commit is test(automation): bind scheduler fixtures to backend identities (Refs #4532), changing only crates/tracedecay/tests/automation_runner_test/jobs.rs. It addresses the separately failing automation-runner baseline, not this PR's relation-recovery source/tests, and introduces a third file into the previously two-file issue-specific diff. The previous Bazel attempt passed the affected session-runtime/session-temporal-store/LCM targets and Windows, but its exact-head evidence applies to f085c0f, not the new head. The new run must finish before current-head claims. The new fixture tests appear behaviorally motivated, but Refs #4532 names the separately owned graph-publication issue; confirm the proper owner and keep unrelated CI fixture repair independently reviewable rather than quietly mixing acceptance. No force pushes, merges, protection bypasses, or changes to the peer fix are authorized here. Issue #4481 remains open pending actual macOS daemon CPU and 24-hour store evidence.

A synchronous OnceLock wait inside the asynchronous historical-ingest
fixture blocked the worker that must poll live projection concurrently.
Use a durable async watch latch to preserve release ordering while
exercising the real interleaving and supersession journey.

Refs #4481

Copy link
Copy Markdown
Owner Author

Published b0f6277 (Refs #4481): the history-interleaving test fixture had introduced OnceLock::wait() inside its async run_pass, blocking the very native worker required to poll live projection alongside in-flight history. The fixture now uses a durable async tokio::sync::watch release latch, preserving the release-before-poll guarantee without pinning the worker thread. The retained real scheduler supersession test still checks both named/aggregate joins. This fixes a test behavior defect, not an established production idle-CPU cause. Exact-head CI is pending; no Bazel or macOS daemon run is available locally. The PR's unrelated automation/dashboard/storage fixture changes still need independent ownership review; do not treat previous head checks as validation. #4481 remains open.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant