Skip to content

Barrier after the wipe in test_all_gather_sdma_in_place - #623

Merged
nirvedhmeshram merged 1 commit into
mainfrom
arprakas/fix-sdma-in-place-test
Nov 4, 2026
Merged

nirvedhmeshram merged 1 commit into
mainfrom
arprakas/fix-sdma-in-place-test

Conversation

@ariscript

Copy link
Copy Markdown
Member

test_all_gather_sdma_in_place wipes out and refills this rank's slice before each call, with no
barrier before the all-gather. A faster rank can then start the next call and copy into a slower
rank's out before that rank's wipe, which erases it, and the slower rank finds an empty slice (the
same hazard example 34 had). _gather_and_check already barriers at that point.

It failed this way in CI on #617 and #618 (4 runs, 3 of them at 8 ranks), leaving the remaining ranks
hung until the NCCL timeout.

🤖 Generated with Claude Code

The test wipes out and refills this rank's slice before each call, with no barrier before the
all-gather, so a faster peer's copies for the next call could land before a slower rank's wipe and
be erased (the same hazard example 34 had). _gather_and_check already barriers there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings November 3, 2026 17:34
@github-actions github-actions Bot added in-progress We are working on it iris Iris project issue labels Nov 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

No open issues 🎉

What changed in this PR

Adds synchronization to prevent a race in test_all_gather_sdma_in_place where faster ranks can start the next all-gather and overwrite a slower rank’s out buffer before it wipes/refills, leading to incorrect assertions and CI hangs/timeouts.

Changes:

  • Insert a ctx.barrier() between refilling the local slice and invoking all_gather to force all ranks to finish wipe/refill before any rank begins the next collective.
File Description
tests/​ccl/​test_all_gather_sdma.py Adds a barrier to prevent cross-iteration buffer overwrite races in the in-place SDMA all-gather test.

Review effort: Lite

@nirvedhmeshram
nirvedhmeshram enabled auto-merge (squash) November 3, 2026 17:52
@nirvedhmeshram
nirvedhmeshram merged commit 3be1c65 into main Nov 4, 2026
26 of 27 checks passed
@nirvedhmeshram
nirvedhmeshram deleted the arprakas/fix-sdma-in-place-test branch November 4, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

in-progress We are working on it iris Iris project issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants