Repository navigation
DDIR: reuse known group bounds in single-chunk advancement - #952
Open
frankmcsherry wants to merge 6 commits into
Open
frankmcsherry wants to merge 6 commits into
frankmcsherry wants to merge 6 commits into
Conversation
Keep up to four borrowed run descriptions inline, and reuse each side's cross-chunk merge heads across keys. This removes two run-description allocations per matching key and repeated compound-value merge scratch allocation without changing presentation or consolidation. Keep the regression for seven chunks, cancellation and successive keys, which exercises both inline-capacity spilling and scratch reuse.
Check all matched identifier-run endpoints in a block through Corgi's borrowed equality API, omitting self and duplicate endpoint checks. Batch collision-pair filtering too, and delete the per-key one_key helper. Keep primitive identifiers streaming, without unnecessary block metadata. Retain structural value merging and collision staging as fallbacks. Add cross-side/chunk collision coverage for symmetric and both lopsided joins. Build on the scratch reuse in TimelyDataflow#941; pin the API from WIP TimelyDataflow#105.
Move join's iterator-based endpoint check into corgi/chunk.rs. The reduce PR carries the identical helper so each PR builds independently and their merged source retains one implementation. Pin the stable-order borrowed equality implementation from WIP TimelyDataflow#105.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Single-input
CorgiChunk::advancecurrently discards existing equality-group bounds and scans the payload withgroup_boundsagain. Preserve those bounds through the owned/shared input paths and materialize their offsets instead. Unknown and multi-input paths keep their existing discovery; an unfinished giant-group carry retains known metadata without advancing its times.Stacked on #950 and its #949/#944/#941 predecessors. Review the own change at 2246a5a9; the upstream base is
master-nextbecause the dependency branch is in the fork. The inherited Corgi pin remains WIP #109's3d7f3b4c.Validation: 79 library + 22
corgi_backendtests pass; Clippy passes with repository warnings. The independent Product/PointStamp oracle now covers structured values, owned/shared input, cancellation and partial advancement. A focused test checks repeated unfinished carries and final cancellation with known/unknown metadata.Three alternating native M4/System-allocator controls over 65,536 rows include bounds materialization, public advancement and output destruction. Owned known flat/nested/list-of-pairs costs are 1.762/11.314/4.698 → 1.016/5.780/1.191 ms. Unknown/multi/native-integer controls have mixed −2.8% to +2.5% deltas with identical allocation counts/bytes. An irregular three-value control improves 1.649 → 0.854 ms, but peak tracked live request bytes rise 8.126 → 8.313 MiB (+192 KiB) while retained offsets coexist with their materialized copy. Construction is excluded; a separate control-only accessor prepares ownership/metadata without changing payload and is absent from this commit.