Repository navigation
DDIR: reuse prepared lanes and known value groups in join - #949
Open
frankmcsherry wants to merge 4 commits into
Open
frankmcsherry wants to merge 4 commits into
frankmcsherry wants to merge 4 commits into
Conversation
Keep up to four borrowed run descriptions inline, and reuse each side's cross-chunk merge heads across keys. This removes two run-description allocations per matching key and repeated compound-value merge scratch allocation without changing presentation or consolidation. Keep the regression for seven chunks, cancellation and successive keys, which exercises both inline-capacity spilling and scratch reuse.
Check all matched identifier-run endpoints in a block through Corgi's borrowed equality API, omitting self and duplicate endpoint checks. Batch collision-pair filtering too, and delete the per-key one_key helper. Keep primitive identifiers streaming, without unnecessary block metadata. Retain structural value merging and collision staging as fallbacks. Add cross-side/chunk collision coverage for symmetric and both lopsided joins. Build on the scratch reuse in TimelyDataflow#941; pin the API from WIP TimelyDataflow#105.
Move join's iterator-based endpoint check into corgi/chunk.rs. The reduce PR carries the identical helper so each PR builds independently and their merged source retains one implementation. Pin the stable-order borrowed equality implementation from WIP TimelyDataflow#105.
This was referenced Oct 17, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Join presentation currently rechecks primitive storage per key and rediscovers equal structured-value runs with one
compare_atper row. Prepare borrowed lanes/classification per block and retain equal-(key,value) ends from existing sort/consolidation and advancement. Readers use known ends, clipped to their selected range; unknown chunks retain the selected-row fallback and never trigger a whole-chunk scan.Stacked on #944/#941; own change is c06a562c. Requires WIP #109 on #105 and pins its exact head. Re-pin to WIP master after those dependencies merge.
Native M4 controls (4096 keys,8 rows/key,2 values/key,one chunk/side; preparation/advancement/consolidation included, source construction excluded): List8 376→72 ns/key; List83072→71; integer77→75. Instrumented structured comparisons57,344→0, with36 remaining capability/ID source visits. Four-chunk merge-head comparisons remain. IC6/BI5 whole-query binding is neutral (4.337→4.346 /20.843→20.874 ms).
Costs:24-byte chunk header; irregular structured groups retain an offset array. Singleton constructor controls add about0.3–2.8 ns/row, and one integer-pair presentation control rises93.9→98.5ns/key. A rejected lazy whole-chunk cache made first sparse nested-list lookup112µs→24ms; retaining existing metadata instead gives25µs. These are native measurements and boundary counts, not measured MZ decompression.
Validation: actual git pin,
cargo test -p interactivepasses162tests with5existing ignored;cargo clippy -p interactive --lib --testscompletes with repository warnings. Focused tests cover cancellation, selected/clipped ranges, true hash collisions, advancement, unknown metadata, retiming and TARGET output splits.72fresh cold/warm sparse controls pass; maintained whole-query answers match their oracle. Raw controls are kept outside this commit.