Repository navigation
Triggered plans: one counter per chunk, per-peer completion counters - #616
Merged
Merged
Conversation
ariscript
requested review from
BKP,
artulab,
drprajap,
efric,
mawad-amd,
neoblizz and
nirvedhmeshram
as code owners
October 29, 2026 19:04
This was referenced Oct 29, 2026
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
Same issue as the Triton path: the loop bound is loaded at runtime, but the Pythonrange(...)in… · New In Triton,range(...)must have a compile-time constant bound; `tl.load(self.view +… · Newtl.constexpris meant to be used as a parameter annotation in Triton kernel signatures, not… · NewH_SENDS_TOis stored in a single int64 header word (bitmask), which cannot represent `world_size… · New
What changed in this PR
Updates triggered-plan gate layout and device/host synchronization to use contiguous int32 counters (per-chunk marks + arrivals, and per-peer completion), enabling single-atomic mark_ready and local-only wait() progress checks.
Changes:
- Replace per-chunk “ready word + counter” gate rows with int32 counters/arrivals and per-peer completion counters (ABI v2).
- Add device APIs
TriggeredView.sent()/wait_sent()and host-side completion signaling per peer queue. - Update unit tests and docs to reflect the new gate tensor layout and semantics.
| File | Description |
|---|---|
iris/experimental/plan/triton.py |
Switches device ABI/layout to int32 gates and adds sent()/wait_sent() logic. |
iris/experimental/plan/gluon.py |
Mirrors Triton TriggeredView changes for Gluon device code. |
iris/experimental/plan/triggered.py |
Allocates/slices new gate tensor, adds round limit check, and switches wait logic to local-only gates. |
iris/experimental/plan/kernels.py |
Replaces progress kernel with local-only wait kernel; updates release semantics. |
iris/experimental/plan/armer.py |
Arms chains polling counter thresholds and signals int32 arrivals + per-peer completion. |
tests/unittests/test_plan.py |
Adds coverage for external counter increments, wait_sent, and round limit; updates arrivals assertions. |
tests/unittests/test_plan_view.py |
Updates view tests for new counter/arrival layout and adds sent()/wait_sent() coverage. |
tests/unittests/test_plan_view_gluon.py |
Updates Gluon view tests similarly and adds sent()/wait_sent() coverage. |
docs/conceptual/triggered-plans.md |
Updates conceptual model and memory-ordering text for counter-based gating and completion counters. |
docs/reference/host/plan.md |
Exposes new device methods sent/wait_sent in API docs. |
iris/experimental/plan/__init__.py |
Updates module docstring to mention new device APIs. |
Review effort: Lite
ariscript
force-pushed
the
arprakas/plan-gate-counters
branch
from
October 30, 2026 22:43
53d8704 to
8accc21
Compare
…iggered plans Gates become one int32 symmetric tensor: a counter and an arrival counter per chunk, then a completion counter per peer. - Chains poll the chunk's counter against writers x round, so mark_ready is a single release atomic: no last-writer election and no ready word. plan.counters is contiguous int32, so a kernel that signals per tile (a tritonBLAS epilogue) can increment it directly. - Every round, a rank's chains to each peer end with an ATOMIC ADD32 on the sending rank's completion counter for that peer. wait() then reads only local words; the progress kernel's remote reads and armed table are gone. TriggeredView gains sent() / wait_sent(). - Arrival signals are 32-bit (ADD32, 0x0f). release_kernel raises the counter to writers x round with atomic_max, so close() opens partially marked gates. - Round limit: writers x round <= 2^31 - 1. Tests pass at 2 and 4 ranks on MI300X (torch allocator). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The device view keeps the peers a rank sends to as bits of one int64 word; at 64 ranks building it failed with an int64 overflow from torch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nment Assigning a Python int to an element of the CUDA handle tensor copies from the host and blocks until the stream drains, so start() (and close()'s release) waited for all queued GPU work. fill_ on a one-element view is queued like any kernel. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ariscript
force-pushed
the
arprakas/plan-gate-counters
branch
from
November 2, 2026 17:37
8accc21 to
8457221
Compare
…eue separately The view now ends with one word per rank, 1 if this rank's chains go to it, read by sent() and the wait kernel; the int64 bitmask and its 63-rank limit are gone. Every round queues each peer's completion signal separately, even after a failed arm, and the original arm error is the one raised. A peer whose signal cannot be queued is no longer waited for, and the plan's memory is kept for the life of the process (with a warning) rather than freed while its copies may still run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The wait kernel treated this rank's own chunks as done and relied on the completion counters, which only exist when the rank has peers. With none, or with the producer on another stream, wait() could reach the mark-count check before the producer finished and report a wrong count. It now polls each own chunk's counter for writers x round (a local read). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d when the round is set - Driving the counters: a gpu-scope release is enough on gfx942/gfx950 (it writes back L2 too); the barrier must wait for stores, which tl.debug_barrier only does on Triton with triton-lang/triton#9025; and which tritonBLAS map_type pairs with which schedule builder, since a mismatched chunk numbering ships chunks before their tiles are written, undetected. - wait_sent: only after all of this rank's chunks are marked, so inside the producer every writer CTA must be resident; new test calls it from a persistent producer. - start() sets the round with work queued on the current stream: kernels taking the handle must be on that stream or wait for it. - Why counters stop at 2^31 - 1; the external-counter test says it mirrors tritonBLAS and relies on a store-waiting barrier. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ariscript
force-pushed
the
arprakas/plan-gate-counters
branch
from
November 3, 2026 17:18
8457221 to
4f1b311
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


First of three stacked follow-ups to #573, in the order agreed on that PR: gate layout (this PR),
then rounds (#617), then destinations and arrival groups (#618). They address @dsidler's review points
1 and 4.
What changes
counter per peer.
writers × round, somark_readyis a singlerelease atomic. There is no last-writer election and no separate ready word.
plan.countersis contiguous int32, so a kernel that signals per tile can increment it directlyinstead of calling
TriggeredView.mark_ready, for example a tritonBLASEpilogueConfig(counter_buffer=...)(Add SignalView and WaitView tritonBLAS#81). That works only when the kernel's counter numbering matches the schedule (only
some
map_type/ schedule-builder pairs do) and its barrier waits for stores (Triton with[Backend] Add a new TritionGPU barrier triton-lang/triton#9025); the docs spell out both.
counter for that peer:
plan.wait()now reads only local memory; the progress kernel's remote reads are gone.TriggeredView.sent()/wait_sent(), for example beforeoverwriting their source.
is the one raised. A peer whose signal cannot be queued is no longer waited for, and the plan's
memory is kept (with a warning) rather than freed while its copies may still run.
wait()also waits until this rank's own chunks have all their marks, so it never checks the markcount before the producer has finished.
start()sets the round with afill_queued on the current stream. It used to assign an element ofthe handle tensor, which copies from the host and waited for all queued GPU work. Kernels that take
the handle must run on that stream, or wait for it.
writers × roundmust stay at or below 2³¹ − 1 (over 7M rounds at 304 writers).Testing (MI300X, torch allocator)
tests/unittests/test_plan.pyat 2 and 4 ranks. New tests:plan.counterswith a plain gpu-scopeatomic_addafterdebug_barrier, as the tritonBLAS epilogue does: 8 writers on one 4 MiB chunk, 10 rounds,no stale data;
wait_sentbefore reusing the source, from a second kernel and from inside a persistent producer;sent()past 64 ranks.