Repository navigation
DDIR: reuse join presentation scratch across keys - #941
Merged
frankmcsherry merged 1 commit intoOct 18, 2026
Merged
Conversation
Keep up to four borrowed run descriptions inline, and reuse each side's cross-chunk merge heads across keys. This removes two run-description allocations per matching key and repeated compound-value merge scratch allocation without changing presentation or consolidation. Keep the regression for seven chunks, cancellation and successive keys, which exercises both inline-capacity spilling and scratch reuse.
This was referenced Oct 17, 2026
frankmcsherry
added a commit
that referenced
this pull request
Oct 18, 2026
* DDIR: reuse join presentation scratch across keys Keep up to four borrowed run descriptions inline, and reuse each side's cross-chunk merge heads across keys. This removes two run-description allocations per matching key and repeated compound-value merge scratch allocation without changing presentation or consolidation. Keep the regression for seven chunks, cancellation and successive keys, which exercises both inline-capacity spilling and scratch reuse. * DDIR: batch join key validation over borrowed columns Check all matched identifier-run endpoints in a block through Corgi's borrowed equality API, omitting self and duplicate endpoint checks. Batch collision-pair filtering too, and delete the per-key one_key helper. Keep primitive identifiers streaming, without unnecessary block metadata. Retain structural value merging and collision staging as fallbacks. Add cross-side/chunk collision coverage for symmetric and both lopsided joins. Build on the scratch reuse in #941; pin the API from WIP #105. * DDIR: share batched key-run validation with reduction Move join's iterator-based endpoint check into corgi/chunk.rs. The reduce PR carries the identical helper so each PR builds independently and their merged source retains one implementation. Pin the stable-order borrowed equality implementation from WIP #105. * DDIR: batch comparisons directly by source column pair * DDIR: pin merged indexed Corgi comparisons
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Join presentation allocated two run-description vectors per matching key, plus merge-head scratch for compound values spread across chunks. Keep up to four borrowed run descriptions inline and reuse each side's merge-head capacity across keys. Larger run sets still spill; the presentation algorithm, payload columns and consolidation are unchanged.
The regression spans seven chunks, compound values, cancellation and successive keys, exercising both spilling and scratch reuse. This PR changes only
interactive/src/corgi/join.rs.An isolated 50,000-key control, with two values per key, measured:
Apple M4, System allocator, scalar
u64time; medians of three alternating runs with 21 timing repetitions each, and a separate allocation pass. The timer covers join presentation and bridge checks over prebuilt native columns, excluding ingestion, sorting, proxy matching, projection and scheduling. These are historical stage measurements, not a whole-query speedup claim.Validation:
cargo test -p interactive --lib --test corgi_backend --test corgi_chunk_mergepasses (one existing ignored scale test);git diff --checkpasses.Based directly on current
master-next(0b41ba7), with the existing Corgi pin. Independent of the reduction PR and the proposed scalar-comparator refresh. Benchmark programs and raw reports are retained in the local audit, outside this PR.