Repository navigation
DDIR: validate join keys in batches - #944
Merged
frankmcsherry merged 6 commits intoOct 18, 2026
Merged
frankmcsherry merged 6 commits into
frankmcsherry merged 6 commits into
Conversation
Keep up to four borrowed run descriptions inline, and reuse each side's cross-chunk merge heads across keys. This removes two run-description allocations per matching key and repeated compound-value merge scratch allocation without changing presentation or consolidation. Keep the regression for seven chunks, cancellation and successive keys, which exercises both inline-capacity spilling and scratch reuse.
Check all matched identifier-run endpoints in a block through Corgi's borrowed equality API, omitting self and duplicate endpoint checks. Batch collision-pair filtering too, and delete the per-key one_key helper. Keep primitive identifiers streaming, without unnecessary block metadata. Retain structural value merging and collision staging as fallbacks. Add cross-side/chunk collision coverage for symmetric and both lopsided joins. Build on the scratch reuse in TimelyDataflow#941; pin the API from WIP TimelyDataflow#105.
This was referenced Oct 17, 2026
Move join's iterator-based endpoint check into corgi/chunk.rs. The reduce PR carries the identical helper so each PR builds independently and their merged source retains one implementation. Pin the stable-order borrowed equality implementation from WIP TimelyDataflow#105.
This was referenced Oct 17, 2026
…-keys # Conflicts: # interactive/src/corgi/join.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validate structured join keys by collecting matched identifier-run endpoints for a whole block, grouping their indices by source-column pair, and calling
equal_idxonce per group. Omit self comparisons, duplicate singleton endpoints and the hash lane. Delete the per-keyone_keycheck. Collision output-pair filtering uses the same indexed interface while retaining output order.Exact primitive identifiers keep the streaming path. Structured value merging and rare collision staging retain their existing comparison fallback. The comparison path reads borrowed columns without gathering payloads.
Builds on #941 and shares an identical iterator-based
chunk.rskey-run validator with #943. Pins merged WIP master 2c9a8ce from Corgi #105. No dependency on the reduction PR or the scalar-comparator proposal.Validation: 181 interactive/server tests pass with five existing ignores; the combined #943-#947 stack passes 191 with five existing ignores. Workspace Clippy passes with existing repository warnings. Regressions exercise forced collisions, disagreement across sides/chunks, symmetric joins and both lopsided directions; existing tests cover cancellation, product times and multiple blocks.
The earlier isolated presentation control improved pair-key symmetric joins from 450.8 to 94.3 ns/key and list-key joins from 297.3 to 104.9; those numbers precede this interface simplification and were not rerun. Fresh IC6/BI5 controls for #943 verify that replacing the pooled API with indexed calls preserves that consumer's query gains. No new whole-query join speedup is claimed. Benchmark artifacts remain outside the code diff.