[Ziminli]

Summary

Support infinicclAvg in HCCL AllReduce and ReduceScatter by submitting native HCCL SUM followed by ACLNN in-place division on the same stream. Previously, the HCCL reduction map rejected AVG. FP16, BF16, FP32, INT8, and INT16 are supported; unsupported AVG datatypes return NotSupported. There is no public API or communicator ABI change.

Breaking build change: WITH_HCCL=ON now requires aclnnop/aclnn_div.h and libopapi, including builds whose applications never request AVG. Basic HCCL probing still checks only HCCL, but actual bbackend configuration requires these additional dependencies.

AVG versus the native HCCL path：

Changes

• Reduction Dispatch

  • Add HcclReductionOp in src/backends/ccl/hccl/reduction_op.h to preserve the AVG flag through the shared CCL adapter while mapping native execution to HCCL_REDUCE_SUM.
  • Share AVG preparation/execution between AllReduce and ReduceScatter in src/backends/ccl/hccl/api.h. Skipping division if native SUM fails.

• Ascend Averaging

  • Implement averaging in src/backends/ccl/hccl/ascend/average.{h,cc} using aclnnInplaceDivMods, with an FP32 divisor for floating inputs and an INT64 divisor with truncation toward zero for INT8/INT16.
  • Prepare before submitting SUM. Reuse thread-local plans keyed by context, stream, output buffer, element count, datatype, and communicator size.
  • Protect cached executor/workspace lifetime with completion events; preserve synchronous null-stream behavior and asynchronous explicit-stream execution.

• Error Handling

  • Add INFINI_CHECK_HCCL and INFINI_CHECK_ACLNN.
  • Keep direct native HCCL statuses returnable and map unsupported AVG types to the public NotSupported result.

• Build Integration

  • Isolate ACLNN/opapi discovery in _infiniccl_find_hccl_average, reusing the existing Ascend SDK hints. Remove invalid QUIET search arguments.
  • Add the ACLNN include path and libopapi linkage in src/CMakeLists.txt.
  • Align the generated public header's closing comment with clang-format.

Platform and Backend Affected

Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU
• HYGON DCU
• T-Head PPU
• Ascend NPU

Backend

• OpenMPI
• MPICH
• NCCL/RCCL
• MCCL
• CNCL
• HCCL

Performance Impact

• No performance impact
• Performance improved
• Performance regression possible

Every AVG call: HCCL first computes SUM, then ACLNN divides the output by the number of ranks. That adds an operator launch and another read/write pass over the output buffer.

First call or cache miss: InfiniCCL also creates the tensor/scalar descriptors, prepares the operator executor, creates a completion event, and allocates workspace if needed.

Cache hit: Matching calls reuse those resources. They still execute the division; caching removes repeated preparation work.

Other reductions: SUM, PROD, MIN, and MAX skip AVG preparation and division.

The previous Ascend/HCCL implementation could not run AVG, so when Ascend is required, this is the cost of enabling AVG rather than a regression from an existing supported AVG path. No controlled performance comparison is reported.

Known Issues & Future Work

• ACLNN/opapi remain mandatory for HCCL configuration. Optional AVG compilation and graceful capability handling on minimal SDK installations are future work.
• INT32/INT64 AVG is disabled because CANN 9.1 division probes lost precision for large values. Unsigned types and FP64 remain disabled pending validation.
• SUM produces an original-datatype result before division, so overflow and rounding can occur before averaging.
• The cache targets eight warm plans per thread but retains in-flight plans; it may grow beyond that target. ACL contexts must outlive cached plans.
• Validation covers Ascend 910C with CANN 9.1.0 and matching A3 operator binaries. Other NPUs/SDK releases, heterogeneous multi-node execution, and full CI remain unverified. Internal failures abort the local process; independently launched peers do not receive a guaranteed communicator-wide abort.
• One development example run reported HCCL resource unavailable during communicator initialization, followed by a cleanup failure. A retry and the final example suite passed; the intermittent failure's root cause is unresolved.

Test Results

Environment: AArch64 Ascend 910C, CANN 9.1.0/A3 container, CMake 3.27.9, native HCCL and local OpenMPI configurations.

Test Involved Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU
• HYGON DCU
• T-Head PPU
• Ascend NPU

Test Involved Backend

• OpenMPI
• MPICH
• NCCL/RCCL
• MCCL
• CNCL
• HCCL

Ascend HCCL + OMPI:
ccl_mpi_hybrid_all_gather.log
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_send_recv.log

Ascend HCCL:
ccl_all_gather.log
ccl_all_reduce.log
ccl_all_reduce_fp16_bf16.log
ccl_reduce_scatter.log
ccl_send_recv.log

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat: …, fix(nccl): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits from master — the branch is rebased cleanly on top of the current master.
• No fixup! / squash! / wip commits remain.

Scope and Design

• Changes are minimal — no unrelated modifications were introduced (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• N/A- Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene

• The code is self-explanatory; comments were added only where the intent or rationale is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, inconsistent indentation, or mixed formatting styles remain.
• Identifiers referenced in comments or error messages are wrapped in Markdown backticks (e.g. the `AllReduce` implementation) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• clang-format (version 16, per .github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.
• No exceptions are thrown. Error paths use assert with messages that include at least __FILE__, __LINE__, and __func__ (CONTRIBUTING.md §C++).
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• Exactly one blank line between classes, between classes and functions, and between functions (CONTRIBUTING.md §C++).
• Exactly one blank line between members (functions and variables) within a class (CONTRIBUTING.md §C++).
• Exactly one blank line before and after the contents of a namespace (CONTRIBUTING.md §C++).

Python Specific (if Python files changed)

N/A — No Python files are changed.

Testing

• All applicable example programs have been built and tested successfully on at least one supported heterogeneous cluster setup.

Build, CI, and Tooling

• N/A- New backends or devices have been added to auto-detection in CMakeLists.txt under if(AUTO_DETECT_DEVICES) or to if(AUTO_DETECT_BACKENDS) if applicable.
• Both CI workflows (clang-format.yml, ruff.yml) are green locally (or expected to be green on CI).

Documentation

• README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• Any user-visible breaking change is called out explicitly under "Summary" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• N/A- Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.