Skip to content

DDIR halfjoin: validate uniform key ranges before fanout - #948

Draft
frankmcsherry wants to merge 2 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-halfjoin-range-validation
Draft

frankmcsherry wants to merge 2 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-halfjoin-range-validation

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

For a hashed identifier range, batch the arranged-key endpoint check and eligible update/representative checks, then emit time-eligible matches without checking every output candidate's key. Sorted real keys make equal endpoints a certificate for the range; genuine collisions retain the bounded fallback. Singleton ranges retain the pooled cross-chunk comparison path.

Targets master-next and includes #945 as a dependency. Review this PR's own change at 4e8cd894. Validation runs within each run/arranged-chunk pair; arranged-chunk seeks advance once after all runs in the block. The range proof and fallback share indexed equality grouped by source-column pair through equal_idx, preserving request order.

Validation on mini-02, Rust 1.95: cargo test --locked --offline -p interactive -p ddir-server passes 193 tests with five existing ignores; cargo clippy --locked --offline -p interactive -p ddir-server --all-targets passes with repository warnings. The halfjoin integration test compares delta joins and recursive reachability with binary joins over 24 update/retraction ticks, with one and three workers.

Remains a draft because earlier native M4 controls showed a fanout/singleton tradeoff: pair keys with fanout 64 took 1,705 → 1,158 ns/key, while singleton-list keys regressed 49.1 → 53.7 ns/key. Triangle load was roughly neutral (78.8 → 79.6 ms). These measurements predate the per-run traversal; benchmarks were not rerun for this rebase.

Use bounded indexed equality grouped by source-column pair over borrowed keys. Preserve candidate order, strict time eligibility and signed differences; cover collisions and mixed integer storage.
@frankmcsherry
frankmcsherry force-pushed the ddir-halfjoin-range-validation branch from d6b172a to 4e8cd89 Compare October 19, 2026 01:36
@frankmcsherry
frankmcsherry changed the base branch from ddir-half-join to master-next October 19, 2026 01:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant