[ScriptedAlchemy]

Issue-specific source repair — Refs #4152

This PR remains partial and does not close #4152. It supersedes historical #4802's commitlint-broken history with a clean branch based on master; no commits were rewritten and all later updates were exact expected-old-head, nonforce fast-forwards.

Production changes

• Remove observation_projection_rebuild_preexisting_outputs, an ephemeral ownership table that was allocated on the long-lived projection writer and could retain unlinked SQLite TEMP pages even after being dropped. The collision preflight reads staged output and current/foreign ownership from the existing indexed durable tables, before clearing ownership.
• Correct canonical-body comparisons: a large raw message stores content = NULL and a derived snippet while its complete body lives in session_canonical_bodies. Comparing the snippet to a staged complete body falsely rejects unchanged foreign/shared outputs. The preflight now walks matching candidates in bounded keyset pages, hydrating compacted protected bodies through the existing hash-verifying reader. Missing/corrupt bytes fail closed; ordinary inline collision and Hermes ownership rules remain in effect. No new TEMP table or shadow cache is introduced.

Regressions

• Registered global-DB preflight test confirms typed rejection of conflicting foreign bodies and absent/uncreated messages; compatible foreign bodies pass; no named TEMP table or writer TEMP-page growth is left; refusal rolls back without touching committed body/location or leaving staging; PRAGMA integrity_check remains clean. This test previously passed nonvacuously in the Linux Bazel log for run 42454900873, though the full target timed out.
• A second registered-SQLite regression commits 129 large compacted bodies, verifies compatibility across the keyset page boundary, rejects a last-page changed tail beyond its unchanged snippet, refuses corrupt/missing canonical bodies, and verifies all committed data, metadata, staging rollback, database integrity, and foreign-key integrity after savepoint-scoped faults.
• The shipped-allocator integration test now exercises MEMORY and FILE TEMP databases with live main/TEMP integrity checks, cross-thread frees/collections, eight rollback cycles, actual Linux unlinked TEMP and sorter-file lifetime, comparison with the production spill census, connection reclamation, and exact data persistence after reopening. Its glibc allocator-routing positive control remains.

Verification and remaining work

• Independent SQLite 3.46.1 probes passed the 129-body comparison cases and the 1,024-row TEMP/sorter and rollback lifetime scenarios. They do not prove the Rust/Bazel tests, mimalloc, or native macOS.
• CI on earlier exact head 42464926379 found only formatting mismatches in the two affected files. Commit a9c68c2 applies precisely the Bazel rustfmt diff without changing behavior. Fresh exact-head CI is pending; no fresh Bazel Rust pass is claimed.
• Required nonvacuous Bazel verification: //crates/tracedecay-cli:c_library_allocator and the named output_preflight regressions under //crates/tracedecay-global-db:unit_test, then affected full targets and platform checks. There is no Bazel executable or working repository checkout in this run environment. No Cargo was used.
• Original in-memory SQLite page-overwrite causality remains unproven. The proven mimalloc owner-heap UAF was removed by merged fix(memory): drop mimalloc owner heaps that crash on delete (#4152) #4173, but this did not prove it caused the in-memory page damage. Other TEMP ownership sets are isolated on mixed DRAFT fix: consolidate follow-up application issue repairs #4776 and must not be bypassed, merged, or auto-merged. Root-cause recurrence/allocator evidence, remaining spill bounds, safe lifetime reclamation and master-proven acceptance are still required before closing daemon: in-memory TEMP page corruption suspect and ~1 GB unlinked SQLite temp files #4152.

No production or infrastructure actions, tags, credentials, release #4763 changes, CI weakening, or schedule changes.

[changeset-bot]

⚠️ No Changeset found

Latest commit: a9c68c2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ScriptedAlchemy]

Refs #4152 — independently reviewed the isolated one-file preflight replacement at 1c3344e against the old materialized ownership flags and the current indexed provenance/staged-output predicates. The boolean conditions preserve the important cases: absent output requires staged creation; a present output can differ only if exclusively created by the current projector. The committed-foreign-body regression executes all three preflight paths plus rollback/integrity and has a nonvacuous passed line in this head's Linux Bazel log (observation_projection::rebuild::activation_tests::output_preflight_rejects_unowned_changes_without_retaining_temp_storage ... ok, run 42454900873, Test Linux fleet).

Not an approval/merge clearance: that same Linux job is red: //crates/tracedecay-global-db:unit_test times out after 300 s in the concurrent 499-test target; automation scheduler and native integration tests also fail outside this diff. Windows is still running and macOS was skipped in this workflow. Keep the checks intact and resolve/attribute those failures through their owners; do not inflate timeouts or treat one test pass as a green target.

This isolated fix stops one persistent TEMP ownership table, but PRAGMA temp.page_count only tests TEMP-database allocation on that writer: it does not census separate unlinked sort files, prove reclaimed bytes on connection retirement, or reproduce the original in-memory SQLite page overwrite. These are still outstanding acceptance requirements. Keep Refs #4152 and the issue open until the remaining spill/reclamation/corruption evidence is actually proven on master. Mixed #4776 remains DRAFT/no merge/no auto-merge.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[ScriptedAlchemy]

Published source repair and real storage regressions — Refs #4152

Pushed two commits personally, each by an exact expected-old-head, nonforce fast-forward. Current head: d5a9ef5e8c741eef529c5518851a88b5100fc182.

Production defect fixed: compacted bodies were compared with snippets

d5a9ef5 fixes a concrete rebuild-preflight bug. upsert_projection_raw_message stores large bodies in session_canonical_bodies, with lcm_raw_messages.content = NULL and only a shortened placeholder in the raw row. Rebuild staging hydrates the full body. Comparing COALESCE(content, placeholder_text, '') with that full staged text therefore rejected unchanged unowned/shared outputs.

The preflight now pages candidate identities using the staged output key and hydrates only compacted protected bodies through the existing verified read_message path. It checks actual decoded bytes and stored metadata, not merely an unverified matching hash. Missing/corrupt bodies fail closed as storage errors. Existing sole-projector ownership and Hermes rules remain unchanged. No TEMP table is reintroduced, and only one candidate body pair is hydrated at a time.

The added registered-SQLite regression commits 129 actual compacted messages before rebuilding, accepts matching bodies across the page boundary, rejects a changed tail on the final message even though its snippet is unchanged, and refuses a corrupted or missing persisted body behind an unchanged hash. Faults are savepoint-scoped; foreign keys are deferred, not disabled. After rollback, a fresh snapshot verifies every committed message and its metadata, no abandoned generation, all integrity-check results, and foreign-key integrity. This is a regression for the verified preflight defect, not a reproduction of the original allocator corruption.

Real allocator / TEMP / sorter lifetime coverage

ad4c9b7 extends the existing c_library_allocator integration test while retaining its glibc-routing positive control. It exercises MEMORY and FILE TEMP databases with the shipped allocator configuration, live main/TEMP integrity checks, cross-thread frees and allocator collection, eight rollback cycles, and exact durable-row comparison. On Linux it requires real unlinked TEMP and sorter files, cross-checks their bytes with the production census without opening/duplicating descriptors, verifies statement/connection reclamation, and reopens the same durable database to prove data preservation. Other Unix hosts execute the integrity/data checks, not the Linux FD observations.

Verification actually performed

• Inspected the published source diff: the second commit changes only rebuild.rs; the first changes only the existing allocator integration test. No held fix: consolidate follow-up application issue repairs #4776 changes were copied.
• Independent Python SQLite 3.46.1 probe: 1,024 rows preserved through eight rollback cycles in each TEMP mode; FILE mode produced 4,726,784 TEMP bytes and 12,594,085 sorter bytes. Sorter files disappeared after statement completion, TEMP files after connection close; dropping the TEMP table alone retained its file.
• Independent SQL/hydration probe: the old predicate rejects matching canonical data; the new candidate flow accepts all 129 matching rows in two pages, rejects the final-row conflict and both storage faults, and uses indexed seek plans without a TEMP B-tree in that fixture.

These probes are supporting mechanism evidence, not Rust/Bazel, mimalloc, or native-macOS test passes. This environment has neither a repository checkout nor Bazel. No Cargo was used. Exact-head workflow 42464926379 is queued; its native=none title is not evidence that native Rust tests will execute.

Required Bazel verification on this head:

bazel test //crates/tracedecay-cli:c_library_allocator --test_output=all
bazel test //crates/tracedecay-global-db:unit_test --test_arg=output_preflight --test_output=all

Confirm the named tests actually execute with nonzero counts, then run the affected target and applicable platform checks. Prior full-target timeouts and unrelated CI failures are not claimed fixed by these commits.

#4152 remains open: original in-memory corruption causality, remaining runtime spill/reclamation acceptance, native evidence, and master acceptance are still outstanding. #4776 is still DRAFT and unmerged; no merge/auto-merge, release, infrastructure, credentials, tags, or schedule changes were made.

[ScriptedAlchemy]

Refs #4152 — exact-head source/behavior review (a9c68c2). Inspected the current rebuild preflight against master ownership predicates and the canonical LCM reader. The indexed keyset loop drops its result cursor before verified per-body hydration, keeps the current-projector/foreign-projector decision ahead of mutation, and rejects absent outputs without staged creation. For compacted inline content, message_body_from_record_row verifies bytes against the canonical hash; the new 129-message regression covers a beyond-first-page changed tail, missing/corrupt canonical rows and transactional rollback. The allocator test exercises real FILE/MEMORY TEMP pages, unlinked sorter lifetime and exact durable data after rollback/reopen, without changing production allocator routing.

Not an approval or merge clearance. The current exact-head run 42468670167 is still queued; the two new Rust regressions have no nonzero Bazel result on this head, nor a native macOS pass. The prior Linux full target timed out, despite one preflight test passing on an older head. This PR does not prove recurrence/causation of the original in-memory page overwrite or bound all long-lived sorter/TEMP spills; the separate retained/cleared TEMP cache change remains held on mixed DRAFT #4776 and must not be moved to bypass its hold. Keep Refs #4152, issue OPEN, and do not merge until applicable exact-head checks/reviews and master acceptance are established.

[ScriptedAlchemy]

Refs #4152 — fresh exact-head Bazel evidence for a9c68c2ba2b203e2a1e7d005a409f89b81c41274.

The Linux fleet job in run 42468670167 executed and passed, without a (cached) annotation:

• //crates/tracedecay-cli:c_library_allocator — PASSED in 1.2 s (single-test integration binary containing the FILE/MEMORY TEMP corruption and real unlinked spill/reclamation regression).
• //crates/tracedecay-global-db:unit_test — PASSED in 105.4 s (contains the output-preflight and 129-canonical-body regressions).
• Repository gates, Clippy, and Dashboard also succeeded.

The fleet command used --test_output=errors, so successful individual test names/counts are not printed in this job log. These are non-cached successful Bazel targets, not an independently collected per-test stdout assertion. No local Bazel or native macOS pass is claimed.

New exact-head CI blocker: the same Linux fleet job is red on three other targets, none modified in this PR:

• //crates/tracedecay-session-runtime:unit_test: a_retryable_projection_publishes_while_summary_admission_is_blocked (zero-progress retry assertion);
• //crates/tracedecay:runtime_acceptance_suite: LSP frame acknowledgement DeadlineExceeded;
• //crates/tracedecay:automation_runner_test: scheduler due/lock/backend classification assertions.

Do not weaken CI or expand this issue's source ownership to chase those failures without a regression-backed attribution. The other Linux job is queued, Windows remains in progress, and macOS was skipped on this run. No full CI-green or merge clearance is claimed.

The separate retained/cleared TEMP tables still live on master and their removal is held in DRAFT #4776; leave that branch DRAFT and unmerged, without copying its held work. Original in-memory overwrite causality and end-to-end bounded spill/reclamation evidence remain unproven. Keep #4152 OPEN and Refs #4152 until every acceptance criterion is demonstrated on master.

[ScriptedAlchemy]

Refs #4152 — new exact-head validation blocker at a9c68c2ba2b203e2a1e7d005a409f89b81c41274.

CI run 42468670167 has now finished failed. Its Windows job was cancelled during Bazel build, after logging ERROR: build interrupted / The operation was canceled; it never reached native tests. This is not evidence of a Windows compilation or behavioral failure, but it leaves Windows coverage unproven. macOS was skipped. The Linux fleet's //crates/tracedecay-cli:c_library_allocator and //crates/tracedecay-global-db:unit_test still passed on this head; the three other red Linux targets remain owned outside #4152. Repository gates, Clippy and Dashboard passed.

Read current root AGENTS.md, the issue and all three discussions, master accepted allocator/sort-spill/census code, current PR review/threads and overlapping PRs. No new issue-specific code defect was established by CI, so no speculative edit, gate weakening, timeout increase or merge was made. The remaining TEMP retained/cleared table removal and physical-inode-deduplicated spill census are present on held DRAFT #4776 only; master still lacks them. The maintenance footprint file also overlaps open #4799. Do not copy held work or overwrite that owner.

Issue #4152 remains OPEN: original memory-overwrite causality, remaining spill/reclamation acceptance and master-proven behavior are unresolved. #4776 and Kev/decision work stay DRAFT/unmerged; release #4763 and all schedules remain untouched.

[ScriptedAlchemy]

Refs #4152 — verified safe-reclamation blocker on the still-held projection writer TEMP schema (not a new source fix).

I ran an isolated SQLite 3.46.1 file-backed probe, without touching operator data:

• A 1,500-row TEMP table allocated a real unlinked etilqs_* file of 6,922,240 bytes.
• After DROP TABLE, the file and 1,690 allocated TEMP pages remained. Neither PRAGMA shrink_memory nor VACUUM temp reclaimed it; PRAGMA temp.auto_vacuum=FULL after allocation also did not.
• Switching PRAGMA temp_store=MEMORY did close the unlinked file and zero TEMP pages, but it also dropped an independently checked live TEMP table and its rows. The separately committed main-database row survived.

Therefore an in-place temp_store reset is not safe to prescribe on the current master projection writer, which still creates the two retained/cleared ownership TEMP tables in clear_active_projection. Resetting while they contain data would silently destroy that connection-local working set. An eventual reclamation path needs proof that no live TEMP objects/cursors or transaction depend on the connection, then verified main-store integrity and physical FD disappearance. Don't replace this with an unchecked reset, or copy the removal from held mixed DRAFT #4776 to bypass its hold.

I independently retried only the previously cancelled Windows CI job on PR #4812's unchanged head a9c68c2 (no CI policy changes); the replacement job entered the setup/build sequence. The existing exact-head Linux Bazel targets covering the issue passed, but the original page-overwrite cause, complete live-daemon spill bounds, and master acceptance remain unresolved. Keep Refs #4152, issue OPEN and PR unmerged; no production, release, tags, infra, credentials, or other schedule changed.