[xingjiangjun]

Summary

Add native NCCL/MCCL AllToAll using grouped Send/Recv, following the existing CCL framework and example conventions.

This PR is a follow-up to #65, originally contributed by @GordonYang1. It rebases and refines the original CCL AllToAll implementation and aligns the examples with existing project implementations.

Rebased onto upstream 52a48aa (#86). The current diff contains eight files; the MPI backend, MPI example, and public example utilities are unchanged. The latest review fixes are in 21eb58a.

Changes

• Shared CCL implementation
  • Keep the peer loop, offsets, communicator/type checks, overflow checks, and balanced group error handling in src/backends/ccl/common/impl/all_to_all.h.
  • NCCL/MCCL operator specializations inherit the common implementation. Reuse upstream Send/Recv and add only thin GroupStart/GroupEnd SDK forwarding methods to the backend API files.
  • Require a matching native communicator; do not fall back to MPI.
• Base validation
  • Return success for zero count before HasInvalidArgs(), following AllGather.
• Examples
  • Follow native thread-per-GPU examples and MPI AllToAll's source/destination block pattern; reuse Validator::ValidateResult() and Metrics::Print().
  • Bootstrap Hybrid native communicators through host-memory MPI Broadcast of the Unique ID.
  • Return failure when validation fails. Block-constant data removes the previous element-index encoding and its count-dependent Float32 precision restriction.

Platform and Backend Affected

Affected shared implementations are listed below; tested platforms are listed separately.

Platform

• CPU — shared base validation and host bootstrap
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU
• HYGON DCU
• T-Head PPU
• Ascend NPU

Backend

• OpenMPI — Hybrid bootstrap; backend implementation unchanged
• MPICH
• NCCL/RCCL
• MCCL
• CNCL
• HCCL

Performance Impact

• No performance impact — no changes to existing collective implementations are intended
• Performance improved
• Performance regression possible

N/A: this is a functionality PR; example timings are not controlled performance comparisons.

Known Issues & Future Work

• Native in-place AllToAll is unsupported.
• RCCL, Moore Threads, Iluvatar, and T-Head runtime behavior, all datatypes, and all failure paths are not exhaustively validated.
• The unchanged OpenMPI backend retains its existing FP16/BF16 limitations.

Test Results

Environment	Tests	Result
NVIDIA NCCL	Native AllToAll: 1 rank/count 1; 2 ranks/count 16 and 262144 (UInt32, count per peer)	3/3 PASS
MetaX MCCL	Same Native AllToAll cases	3/3 PASS
NVIDIA OpenMPI + NCCL / MetaX OpenMPI + MCCL	Hybrid AllToAll, 2 ranks, Float32, count 262144 per peer	Both PASS
NVIDIA NCCL / MetaX MCCL	AllReduce, AllGather, ReduceScatter, Send/Recv, 2 GPUs	4/4 PASS on each platform
NVIDIA + MetaX OpenMPI	Nine MPI examples, 2+2 GPUs, OpenMPI 4.1.6 and UCX/TCP	9/9 PASS

The nine MPI examples are AllGather, AllReduce, AllToAll, Broadcast, Gather, Reduce, ReduceScatter, Scatter, and Send/Recv. Validation output and process exit codes were checked. AllToAll parameter/status checks also passed on both native backends. Compilation and local clang-format 16 checks passed.

Both platforms used verified source snapshots. The final Native example was rebuilt and rerun after its cleanup; Hybrid tests preceded that Native-only change and its source was unchanged. The committed source matches the final tested snapshot; tests were performed before creating the commit, not rerun solely to change its SHA.

Heterogeneous runs used the OpenMPI backend with native NCCL/MCCL disabled. RDMA/RoCE is not claimed. Send/Recv ranks 0 and 1 were both on NVIDIA. These are correctness checks, not controlled performance measurements.

Current build/test logs and per-rank exit codes are archived locally and have not yet been uploaded. The previous test-log attachment documents the earlier version, not these reruns.

Test Involved Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU
• HYGON DCU
• T-Head PPU
• Ascend NPU

Test Involved Backend

• OpenMPI
• MPICH
• NCCL/RCCL — NCCL only
• MCCL — MetaX only
• CNCL
• HCCL

---

Checklist

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat: …, fix(nccl): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests). Three commits preserve the original contribution and review revisions; squash disposition is left for merge.
• No stray merge commits from master — rebased onto 52a48aa, including feat: support MCCL's AllGather, ReduceScatter, Send, and Recv #86.
• No fixup! / squash! / wip commits remain.

Scope and Design

• Changes are minimal — no unrelated modifications were introduced (CONTRIBUTING.md §Code/General).
• No new dead code or debug-only prints; standard example output and the existing base-layer TODO are retained.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene

• The code is self-explanatory; comments were added only where the intent or rationale is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, inconsistent indentation, or mixed formatting styles remain.
• Identifiers referenced in comments or error messages are wrapped in Markdown backticks (e.g. the `AllReduce` implementation) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly. Examples follow the existing repository conventions; strict compliance is not claimed.
• clang-format (version 16, per .github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.
• N/A: no explicit exception throwing was added; native failures follow existing Api::Check/ReturnStatus conventions rather than new assertion-based handling.
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• N/A: no constructors with member initializer lists were added.
• Exactly one blank line between classes, between classes and functions, and between functions (CONTRIBUTING.md §C++).
• Exactly one blank line between members (functions and variables) within a class (CONTRIBUTING.md §C++).
• Exactly one blank line before and after the contents of a namespace (CONTRIBUTING.md §C++).

Python Specific (if Python files changed)

N/A: no Python files changed.

Testing

• The listed Native, Hybrid, and nine heterogeneous MPI examples passed; coverage and attachment limitations are stated under Test Results.

Build, CI, and Tooling

• N/A: no new backend/device or auto-detection changes.
• Both CI workflows (clang-format.yml, ruff.yml) passed on 21eb58a.

Documentation

• README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• N/A: no released public API signature changes; changes to the proposed PR behavior are documented above.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.

[Ziminli]

line198-199 这个打印需要去掉，可以参考其他示例，用 Validator::ValidateResult(). 保证代码复用和打印格式对齐。

[xingjiangjun]

Updated in 1ab5156. In examples/ccl_mpi_hybrid/all_to_all.cc:32–50, each received element is compared with its own expected value, and the resulting error indicators are passed to Validator::ValidateResult(). This preserves exact checking and rejects NaN/Inf while using the standard output format. Each rank reports its own result, and validation failure produces a nonzero exit status. The duplicate custom pass/fail print was removed. Both ranks passed the Hybrid regression.

[Ziminli]

这个 example 的打印和实现形式还是没有对齐。具体说明下：

1. 不需要每个 rank 都打印结果，参考其他 examples；
2. AllToAll 的打印可以参考 mpi/all_to_all 的打印逻辑，大概长这样：

=== Performing AllToAll on GPU Memory ===
Count per peer: 262144 floats
Total per rank: 2097152 floats (8 MB)
Warm-up iterations: 2
Profile iterations: 20

=== AllToAll Results ===
Correct: YES
Sample recv blocks: [src0: 0] [src1: 1000] [src2: 2000] [src3: 3000] 
Time:           4.137 ms
Throughput:     3.31 GB/s (Bus BW)
Alg Bandwidth:  1.89 GB/s
InfiniCCL finalized.

保证 ccl_mpi_hybrid/all_to_all 的输出和这个对齐；
3. PrintMetrics 没有复用。还是一样，参考其他 examples，都是直接复用已有的 Metrics：

// Metrics Reporting (Only from rank 0 for cleaner output)
  if (rank == 0) {
    Metrics metrics{elapsed, total_bytes, size};
    metrics.Print();
  }

4. 参考 ccl_mpi_hybrid/all_reduce.cc，RunAllToAllExample() 里需要留一些分阶段的 comments，比如 // Setup Communicator, // Warm-up and D2H transfer the answer. 这种，不然看着比较乱。

总而言之就是参考其他已有的和对应的 examples，保证实现方式和格式大体上都是对齐的，按理来说这就是个模块化的工作。

[Ziminli]

Communicator fallback 的这些逻辑目前都不需要。实现直接对标其余实现（e.g., all_reduce, reduce_scatter, all_gather, etc.) 即可。

[xingjiangjun]

Removed DeferredAllToAll and communicator fallback in 1ab5156. The backend-specific implementations now require a matching native communicator and validate its handle (src/backends/ccl/{nccl,mccl}/impl/all_to_all.h:28–47). The obsolete common implementation was removed. To keep Hybrid initialization valid, examples/ccl_mpi_hybrid/all_to_all.cc:95–109 now follows Hybrid AllReduce: initialize the MPI communicator, broadcast the Unique ID in host memory, then initialize the native communicator. Native and Hybrid two-rank regressions passed.

[Ziminli]

现在的 ccl AllToAll 实现没有正确利用 ccl 的实现框架。基础实现应该在 backends/ccl/common/ 里，而不是在每个具体的 ccl 后端（e.g., nccl/, mccl) 下都写一遍具体的实现。具体可以参考其他已有的 ccl 通信操作实现，比如 all_reduce。