Repository navigation
DDIR: move whole chunks during extraction - #950
Open
frankmcsherry wants to merge 5 commits into
Open
frankmcsherry wants to merge 5 commits into
frankmcsherry wants to merge 5 commits into
Conversation
Keep up to four borrowed run descriptions inline, and reuse each side's cross-chunk merge heads across keys. This removes two run-description allocations per matching key and repeated compound-value merge scratch allocation without changing presentation or consolidation. Keep the regression for seven chunks, cancellation and successive keys, which exercises both inline-capacity spilling and scratch reuse.
Check all matched identifier-run endpoints in a block through Corgi's borrowed equality API, omitting self and duplicate endpoint checks. Batch collision-pair filtering too, and delete the per-key one_key helper. Keep primitive identifiers streaming, without unnecessary block metadata. Retain structural value merging and collision staging as fallbacks. Add cross-side/chunk collision coverage for symmetric and both lopsided joins. Build on the scratch reuse in TimelyDataflow#941; pin the API from WIP TimelyDataflow#105.
Move join's iterator-based endpoint check into corgi/chunk.rs. The reduce PR carries the identical helper so each PR builds independently and their merged source retains one implementation. Pin the stable-order borrowed equality implementation from WIP TimelyDataflow#105.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CorgiChunk::extractcurrently copies the key/value/time/diff columns even when every row belongs to the same side of the frontier. Move those whole chunks instead, preserving their known value-group metadata. The residual update is unchanged; partial extraction still gathers both pieces.Stacked on #949 (and its #941/#944 predecessors). Review this PR's own change at 9092577a; the upstream base is
master-nextbecause the dependency branch is in the fork. The inherited Corgi pin is WIP #109's3d7f3b4cand must be re-pinned after that lands.Validation:
cargo test -p interactive --lib --test corgi_backendpasses (78 library + 22 backend tests). The new test checks shared allocation identity, retained groups, value/time/diff equivalence, frontier/residual behavior, and empty chunks.Three alternating native mini-02 M4/System-allocator controls with 262,144 rows give all-ship flat/nested structured values 1.331/9.106 → 0.101/0.103 ms and all-keep 1.876/10.106 → 0.747/0.723 ms. Partial selection is slightly slower: 2.407 → 2.452 ms flat (+1.9%) and 9.858 → 9.940 ms nested (+0.8%). Partial allocation counts/bytes are identical. Whole-side payload gather calls become 1 → 0; partial remains 2 → 2 (counts derived from the source). These are native Corgi measurements, with construction excluded and returned-output destruction included.