Skip to content

Avoid rebuilding unchanged frontiers for positive count updates - #829

Draft
frankmcsherry wants to merge 2 commits into
TimelyDataflow:masterfrom
frankmcsherry:codex/progress-frontier-counts
Draft

frankmcsherry wants to merge 2 commits into
TimelyDataflow:masterfrom
frankmcsherry:codex/progress-frontier-counts

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Adding a positive count at a timestamp already in a MutableAntichain frontier rebuilds that frontier, scanning all retained timestamps even though the addition cannot change the frontier.

Decide whether to rebuild from the update sign: a positive update can change the frontier only if no current frontier element is less than or equal to its time; a negative update can change it only if its time is in the frontier. Zero updates do nothing. The check stays in the existing helper, preserving the factoring introduced by #797. Counts retain the existing ChangeBatch compaction policy.

This occurs in Differential's TraceBox compaction adjustment, which adds the new holds before removing the old holds, and when cloning shared trace readers. In one instrumented run of the existing Differential BFS example (10k vertices, 100k initial edges, ten waves of 1k separately timestamped replacements, four workers), this predicate would avoid 9,808 of 71,806 baseline rebuilds, including 6,120 in the compaction-addition calls. Avoided rebuilds were small: at most 27 raw batch entries. The existing Timely PageRank example also exercises positive additions at current frontier elements.

The trace-compaction example also has an independent batching opportunity: it calls update_iter separately for additions and removals without observing the intermediate frontier. Combining those iterators can already eliminate an intermediate rebuild. The diagnostic counts above include that caller pattern and should not be interpreted as savings beyond improved caller batching. Trace cloning (which only adds holds) and positive additions across propagation rounds remain distinct cases. This PR preserves the existing rule of at most one frontier rebuild after consuming the entire iterator.

Seven interleaved, uninstrumented release runs of existing Timely PageRank, hash-join, barrier, and Differential BFS examples showed median execution times within 2% of baseline. These runs establish occurrence, not a material application speedup. Repeated additions with a large retained timestamp population can still avoid quadratic scan work, but that large-state case was not established in these application runs. No benchmark example is added.

The regression test compares frontiers, emitted changes, and retained counts against a counted-set oracle for 2,000 signed batches each over scalar and product timestamps, including negative counts, zeros, and frontier retreat.

Validation: workspace tests, doctests, and Clippy passed before removal of the standalone benchmark; production and test code are unchanged. Existing application examples were additionally rebuilt and run against this independent draft.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant