From 7781481ffb1730b71aa149c255da5b867a2f5fc5 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Wed, 14 Oct 2026 20:45:13 +0200 Subject: [PATCH 1/4] Add AGENTS.md guidance for coding agents Add a root AGENTS.md and per-directory guides for dpnp/, dpnp/tests/, dpnp/backend/, dpnp/tensor/, doc/, benchmarks/ and .github/. The rules come from recurring maintainer review comments, and every command, path and identifier in them is checked against the tree. --- .github/AGENTS.md | 13 +++++++ AGENTS.md | 78 ++++++++++++++++++++++++++++++++++++++++++ benchmarks/AGENTS.md | 19 ++++++++++ doc/AGENTS.md | 24 +++++++++++++ dpnp/AGENTS.md | 35 +++++++++++++++++++ dpnp/backend/AGENTS.md | 39 +++++++++++++++++++++ dpnp/tensor/AGENTS.md | 24 +++++++++++++ dpnp/tests/AGENTS.md | 41 ++++++++++++++++++++++ 8 files changed, 273 insertions(+) create mode 100644 .github/AGENTS.md create mode 100644 AGENTS.md create mode 100644 benchmarks/AGENTS.md create mode 100644 doc/AGENTS.md create mode 100644 dpnp/AGENTS.md create mode 100644 dpnp/backend/AGENTS.md create mode 100644 dpnp/tensor/AGENTS.md create mode 100644 dpnp/tests/AGENTS.md diff --git a/.github/AGENTS.md b/.github/AGENTS.md new file mode 100644 index 000000000000..932c44ac9a83 --- /dev/null +++ b/.github/AGENTS.md @@ -0,0 +1,13 @@ +# AGENTS.md - CI Workflows (.github/) + +## Rules for Changes +These keep zizmor (`zizmor.yml`, fails on medium or higher findings) and the OpenSSF Scorecard clean. +- Pin every third-party action to a full commit SHA with a `# ` comment. +- Every workflow declares top-level `permissions` (`read-all`, `contents: read` or `{}`). Grant more per job, with a comment saying why. +- Every `actions/checkout` sets `persist-credentials: false`. +- Never inline untrusted contexts (`github.event.*`, `github.head_ref`, `inputs.*`) in `run:`; pass them through `env:`. Prefer `env:` for every value in new code. +- Every job sets `timeout-minutes`; for a job that calls a reusable workflow, set it inside the called workflow. +- `actionlint` runs in pre-commit. +- `workflows/array-api-skips.txt` lists array-api-tests node IDs skipped by the conformance job; precede each with a comment naming the upstream issue. +- `bump-sycl-deps.yml` rewrites the `DRIVER_PATH`, `OCLCPUEXP_FN` and `TBB_*` pins in `os-llvm-sycl-build.yml` with `sed`; keep their format. +- Dependabot (`dependabot.yml`) bumps actions, `environments/` pins and pre-commit hooks weekly with a 7-day cooldown. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000000..4fa39702508b --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,78 @@ +# AGENTS.md - dpnp + +dpnp is a NumPy-compatible array library for SYCL devices. Its main namespace, `dpnp.fft` and `dpnp.linalg` also conform to the Python Array API standard (2025.12, `doc/reference/array_api.rst`). The Python layer (`dpnp/`) is built on `dpnp.tensor` (Array API, `dpnp/tensor/`) and pybind11 SYCL extensions (`dpnp/backend/extensions/`); `dpctl` provides devices, queues and USM memory. + +## Rules for Changes + +These come from recurring maintainer review comments. Directory-specific rules are in the nearest `AGENTS.md`. + +- Match NumPy, or SciPy for `dpnp.scipy`: shape, dtype, memory layout, edge cases (empty, 0-d, NaN/inf) and error and warning text. Following CuPy instead is an exception. Any deliberate difference is documented in the docstring and covered by a test with a comment. +- Don't break Array API conformance. Where NumPy and the spec differ, raise it in the PR rather than silently picking one (precedent: `ndarray.T` on non-2-D arrays was deprecated to match the spec). +- Code and tests must work on devices without fp64 support (e.g. Iris Xe). Don't hardcode float64/complex128; use `dpnp.default_float_type()` or the dtype helpers in `dpnp/tests/helper.py`. +- Compute follows data: allocate results and temporaries on the inputs' SYCL queue with their coerced USM type, and raise when inputs computed together are on different queues (`get_usm_allocations` raises `ValueError`, `dpnp.tensor` raises `ExecutionPlacementError`). Never move inputs silently. +- Avoid hidden host transfers and queue syncs: `int()`/`bool()` on device arrays, `.wait()`, `asnumpy`, redundant full-size copies. +- Accept both `dpnp.ndarray` and `usm_ndarray`; validate and unwrap with `dpnp.check_supported_arrays_type` and `dpnp.get_usm_ndarray`, not ad-hoc `isinstance` or method calls. +- Every user-visible change, including build, recipe and C-API changes, gets a one-line `CHANGELOG.md` entry (format below). +- A new public function needs an export in `dpnp/__init__.py` (or in its subpackage's `__init__.py`), a numpydoc docstring, an entry in `doc/reference/*.rst`, tests (including `test_sycl_queue.py` and `test_usm_type.py`) and a changelog entry. Don't export names that NumPy/SciPy don't. +- Deprecate before removing: keep the old API for at least one release with a `DeprecationWarning`, and add a `Deprecated` changelog entry and a test for the warning. Removing something NumPy itself has removed can land directly with a `Removed` entry. +- Every new branch and error path has a test; a bug fix comes with a test that fails without it. Use `# pragma: no cover` only for branches the Linux coverage job can't reach (CUDA, HIP, Windows-only) or that can't happen by construction. +- A workaround, skip or xfail gets a comment with the full issue URL so it can be reverted later. In `dpnp/tests/third_party/`, a skip for an unsupported feature needs only its reason string. +- Keep comments short and accurate: one line of why. Don't narrate the change, restate the code, or mention "this PR". +- One PR, one change. Search for an existing helper before adding one. +- New files carry the BSD-3 license header from a neighbouring file with the current year; never change the year in existing files. Test files directly in `dpnp/tests/` and under `dpnp/tests/third_party/` have no header. +- Don't edit generated files: `dpnp/_version.py`, `dpnp/include/dpnp/tensor/_usmarray*.h`, `dpnp/resources/cmake/`, `doc/reference/generated/`, `doc/reference/*.inc`. + +## Verification Before You Push + +```bash +pre-commit run --all-files # the required `Check` job; pylint must be on PATH +python scripts/build_locally.py # in-place build + editable install +python -m pytest -ra dpnp/tests/test_.py +``` + +The build needs the packages in `environments/build_with_oneapi.yml`, `environments/oneapi_pkgs.yml` and `environments/dpctl_pkg.yml`. The `no-commit-to-branch` hook fails on `master` and `maintenance/*`, so work on a branch. Full-suite commands and test env vars are in `dpnp/tests/AGENTS.md`; build options in `dpnp/backend/AGENTS.md`. + +### Changelog format + +Append to the end of the matching `### Added` / `Changed` / `Deprecated` / `Removed` / `Fixed` / `Security` list in the top `## [X.Y.Z] - YYYY-MM-DD` section. Past tense, code in backticks, no trailing period: + +``` +* Added support for the `"nuc"` (nuclear) norm in `dpnp.linalg.cond` [#3084](https://github.com/IntelPython/dpnp/pull/3084) +``` + +### Where the checks live + +| Check | Workflow | Required on master | +| --- | --- | --- | +| pre-commit (`Check`) | `.github/workflows/pre-commit.yml` | yes | +| conda build and tests, Linux and Windows, py3.12-3.14t (`Test (...)`) | `conda-package.yml` | yes | +| Internal Jenkins (`jenkins-test-dpnp-3.13-linux-dev`, `...-3.14-...`) | external | yes | +| Array API conformance of `dpnp`, not yet `dpnp.tensor` (array-api-tests; never fails, results posted as a PR comment) | `conda-package.yml` | no | +| pybind11 examples | `conda-package.yml` | no | +| conda-forge build and tests | `conda-package-cf.yml` | no | +| oneMath build, open-source LLVM SYCL build | `check-onemath.yaml`, `os-llvm-sycl-build.yml` | no | +| Sphinx docs (`make spelling`, `make html`) | `build-sphinx.yml` | no | +| Coverage (Coveralls), zizmor | `generate_coverage.yaml`, `zizmor.yml` | no | + +GitHub-hosted runners have no GPU, so tests there run on a CPU device: a green run says nothing about GPUs or devices without fp64. + +## Reviewing + +Report source-confirmed problems with NumPy compatibility, Array API conformance, compute-follows-data, devices without fp64, SYCL event ordering and lifetimes, error handling and test coverage. Don't report what pre-commit enforces (formatting, import order, spelling), and don't report that `@pytest.mark.slow` tests are skipped in CI: that is by design. + +## Directory Guides + +Read the `AGENTS.md` nearest the files you change. When editing these files, every command, path and name must match the repository; delete what can't be verified rather than soften it. + +- [dpnp/AGENTS.md](dpnp/AGENTS.md): Python layer, adding a function, docstrings +- [dpnp/tests/AGENTS.md](dpnp/tests/AGENTS.md): test helpers, fp64, CuPy tests, running the suite +- [dpnp/backend/AGENTS.md](dpnp/backend/AGENTS.md): pybind11 SYCL extensions, C++ rules, build options +- [dpnp/tensor/AGENTS.md](dpnp/tensor/AGENTS.md): `dpnp.tensor` and libtensor +- [doc/AGENTS.md](doc/AGENTS.md): Sphinx docs +- [benchmarks/AGENTS.md](benchmarks/AGENTS.md): ASV benchmarks +- [.github/AGENTS.md](.github/AGENTS.md): workflow hardening rules + +## Further Reading + +- `doc/quick_start_guide.rst`: build from source, CUDA/HIP targets +- `.github/pull_request_template.md`: the PR checklist diff --git a/benchmarks/AGENTS.md b/benchmarks/AGENTS.md new file mode 100644 index 000000000000..bfed210cdf68 --- /dev/null +++ b/benchmarks/AGENTS.md @@ -0,0 +1,19 @@ +# AGENTS.md - Benchmarks (benchmarks/) + +ASV suite that times dpnp only; `README.md` has the full setup. + +## Rules for Changes +- NumPy is not timed: `executor` stays the single value `'dpnp'`. Don't edit the leftover `numpy` branches in `setup`: ASV versions a benchmark by the source of `setup` and `time_*`, so any edit makes its results incomparable with the recorded history. +- Keep `params` static and their values plain strings, numbers or tuples; skip unsupported devices in `setup` with `SkipNotImplemented`. +- Timed bodies block on the result (`self.sync(...)` or `dpnp.synchronize_array_data`); otherwise they time only the submission. +- Parameter axes shared by two or more modules live in `benchmarks/benchmarks/_utils.py`. +- A new non-ASV dependency goes in both `requirements.txt` (what CI installs) and the `benchmark` extra in `pyproject.toml`; ASV itself is only in the extra. + +## Verification +From `benchmarks/`, against an environment that already has dpnp: +```bash +asv machine --yes +asv check --python=same +asv run --python=same --launch-method spawn --quick --bench bench_elementwise +``` +Always pass `--launch-method spawn`: the SYCL runtime is not fork-safe. Don't `pip install ".[benchmark]"`; it rebuilds dpnp. diff --git a/doc/AGENTS.md b/doc/AGENTS.md new file mode 100644 index 000000000000..be6bb453f511 --- /dev/null +++ b/doc/AGENTS.md @@ -0,0 +1,24 @@ +# AGENTS.md - Documentation (doc/) + +## Layout +- `conf.py`: Sphinx config (numpydoc, autosummary, furo). It registers the dpnp `Limitations` docstring section, documents the `DPNP*Func` wrapper objects as functions, and regenerates `reference/comparison_table.rst.inc` from `comparison_generator.py` on every build +- `reference/.rst`: `.. autosummary::` lists under `.. currentmodule::` (`dpnp` or the subpackage); stubs are generated into `reference/generated/` +- `known_words.txt`: spelling allow-list +- `quick_start_guide.rst`: install and build-from-source instructions + +## Rules for Changes +- A new public function goes in the autosummary block of the matching `reference/.rst`; ufuncs also go in `reference/ufunc.rst`, `ndarray` members in `reference/ndarray.rst`. These pages already list many NumPy names that dpnp lacks, so check before adding a duplicate. +- A new namespace needs its own `reference/.rst` and a toctree entry: in `reference/routines.rst` for NumPy namespaces (model: `reference/fft.rst`), in `reference/scipy.rst` for SciPy ones (model: `reference/scipy_linalg.rst`). +- Move a deprecated function from its reference page to `reference/_deprecated.rst` (an orphan page that keeps its stub) and remove cross-links to it. +- A new wrapper base class (not a subclass of the existing `DPNP*Func` classes) must be added to `conf.py` and `_can_document_member`, or autodoc skips its instances. +- Update `quick_start_guide.rst` (and `README.md`) when build options or supported Python versions change, and `reference/array_api.rst` when the Array API version or the conforming namespaces change. +- Add new legitimate words to `known_words.txt`. +- Warnings don't fail the build (no `-W`), so check the log and the rendered page yourself. + +## Verification +```bash +cd doc +make spelling +make html +``` +This needs an importable dpnp build and the packages in `environments/building_docs.yml` (including CuPy, for the comparison table). CI (`build-sphinx.yml`) publishes builds of PRs from IntelPython/dpnp branches at `https://intelpython.github.io/dpnp/pull//index.html`; fork PRs get an artifact instead. diff --git a/dpnp/AGENTS.md b/dpnp/AGENTS.md new file mode 100644 index 000000000000..81338636ee14 --- /dev/null +++ b/dpnp/AGENTS.md @@ -0,0 +1,35 @@ +# AGENTS.md - Python layer (dpnp/) + +## Layout +- `dpnp_iface_.py`: public functions, grouped like NumPy's routines reference +- `dpnp_array.py`: `dpnp.ndarray`, a wrapper over `dpnp.tensor.usm_ndarray`; `get_array()` unwraps, `_create_from_usm_ndarray()` wraps +- `dpnp_algo/dpnp_elementwise_common.py`: ufunc wrapper classes (`DPNPUnaryFunc`, `DPNPBinaryFunc`, two-output and specialised variants) +- `dpnp_utils/`: shared helpers (`get_usm_allocations`, `map_dtype_to_device`, reduction wrappers) +- `linalg/`, `fft/`, `random/`, `scipy/`, `lib/`: subpackages, each exporting through its own `__init__.py` +- Legacy, don't extend: the Cython `dpnp_algo` path and `dpnp_backend_c` (still used by `dpnp.partition` and `dpnp.random`), and the NumPy fallback `call_origin`. New kernels go into pybind11 extensions or `dpnp.tensor`. + +## Adding a Public Function +1. Implement it in the matching `dpnp_iface_*.py` or subpackage module. Iface modules have no `__all__`. +2. Export it in `dpnp/__init__.py` in two places, both under its NumPy routine category: the `from .dpnp_iface_xxx import (...)` block and the `__all__ += [...]` block. Functions in `linalg`, `fft`, `random`, `scipy.*` and `lib` are exported from that subpackage's `__init__.py` instead (import block and `__all__`). A new subpackage also goes in `packages=` in `setup.py`. +3. Add it to `doc/reference/` (see `doc/AGENTS.md`), tests (see `dpnp/tests/AGENTS.md`) and `CHANGELOG.md`. + +## Patterns +- Ufuncs are module-level wrapper instances, e.g. `cbrt = DPNPUnaryFunc("cbrt", ti._cbrt_result_type, ti._cbrt, _CBRT_DOCSTRING, mkl_fn_to_call=..., mkl_impl_fn=...)`. The first argument is the public name, with no leading underscore. Kernels come from `dpnp.tensor._tensor_elementwise_impl` or `dpnp.backend.extensions.ufunc._ufunc_impl`; the oneMKL VM path runs first when `mkl_fn_to_call` is given and supported. +- Other functions: `dpnp.check_supported_arrays_type(...)`, unwrap with `dpnp.get_usm_ndarray`, call `dpnp.tensor` (`import dpnp.tensor as dpt`) or an extension, and wrap with `dpnp_array._create_from_usm_ndarray`. Pass `out` straight to the `dpnp.tensor` or extension call when it accepts one; otherwise `dpnp.get_result_array(res, out)` copies the result into it. +- Get the result queue and USM type with `get_usm_allocations([...])` (raises `ValueError` on mixed queues) or `dpt.get_execution_queue(...)` (returns `None`; raise `ExecutionPlacementError`). Creation functions take `device`, `usm_type` and `sycl_queue` (normalised by `dpnp.get_normalized_queue_device`); `*_like` functions inherit them from the input. +- Extension calls are ordered with `_manager = dpctl.utils.SequentialOrderManager[exec_q]`: pass `depends=_manager.submitted_events`, then `_manager.add_event_pair(ht_ev, ev)`. Don't add blocking waits. +- Use `dpnp.default_float_type(...)` or `map_dtype_to_device(...)` for the default float type. +- Deprecated ufuncs use `DPNPDeprecatedUnaryFunc(..., deprecated_msg=...)`. +- Unsupported NumPy keywords raise `NotImplementedError` (`dpnp.check_limitations`) and are listed under `Limitations` in the docstring. +- The Array API spec version is set in three places that change together: `__array_api_version__` in `dpnp/tensor/_array_api.py`, `ARRAY_API_TESTS_VERSION` in `.github/workflows/conda-package.yml` and `doc/reference/array_api.rst` (plus the version test in `dpnp/tests/test_ndarray.py`). Inspection utilities come from `__array_namespace_info__` in `dpnp_array_api_info.py`. +- `dpnp.tensor` replaces `dpctl.tensor`, which no longer exists here; `SyclQueue`, `SyclDevice`, `dpctl.memory` and `SequentialOrderManager` still come from `dpctl`. + +## Docstrings +numpydoc, rendered by `doc/conf.py`. Model a new docstring on an existing function; `doc/docstring_template.py` is outdated. +- Sections in the order `doc/conf.py` renders them: summary, ``For full documentation refer to :obj:`numpy.`.``, Parameters, Returns, Raises, Warns, Warnings, Limitations, See Also, Notes, References, Examples. +- Array parameters are typed `{dpnp.ndarray, usm_ndarray}`, optional ones `{None, dpnp.ndarray, usm_ndarray}, optional`; the return is `out : dpnp.ndarray`. +- Every keyword with a default ends with a blank line and ``Default: ````.`` +- Literals in double backticks (``` ``None`` ```), argument names in single backticks; link with `:obj:`. +- Copy the `device` / `usm_type` / `sycl_queue` parameter text from an existing creation function such as `dpnp.empty`. +- Examples start with `>>> import dpnp as np`. A docstring that contains `\` is raw (`r"""`). +- An `ndarray` method that wraps a function has a one-line summary, a blank line, then ``Refer to :obj:`dpnp.` for full documentation.`` diff --git a/dpnp/backend/AGENTS.md b/dpnp/backend/AGENTS.md new file mode 100644 index 000000000000..2297cc536078 --- /dev/null +++ b/dpnp/backend/AGENTS.md @@ -0,0 +1,39 @@ +# AGENTS.md - C++ Backend (dpnp/backend/) + +## Layout +- `extensions//`: one pybind11 module `__impl` per directory (`blas`, `fft`, `indexing`, `lapack`, `sparse`, `statistics`, `ufunc`, `vm`, `window`), imported as `dpnp.backend.extensions..__impl` +- `extensions/common/ext/`: dispatch helpers (`init_dispatch_vector`, `init_dispatch_table`, `DispatchTable`) and validation helpers (`validation_utils.hpp`) +- `extensions/elementwise_functions/`: copy of the libtensor elementwise Python-API helpers, compiled into `ufunc` and `vm` +- `kernels/{elementwise_functions,indexing,statistics,window}/`: header-only SYCL functors in namespaces under `dpnp::kernels` +- Legacy: `src/`, `include/` and `kernels/dpnp_krnl_*.cpp` build `dpnp_backend_c` (still used by `dpnp.partition` and `dpnp.random`); `tests/` (GTest, `-DDPNP_BACKEND_TESTS=ON`) and `examples/` are not built in CI +- `cmake/Modules/`: vendored `IntelSYCL`, `MKL`, `TBB` and `oneDPL` config modules + +## Adding a ufunc Kernel (model: `fabs`) +1. Functor in `kernels/elementwise_functions/.hpp`. +2. Binding in `extensions/ufunc/elementwise_functions/.{hpp,cpp}`: output-type map built from `td_ns::TypeMapResultEntry<...>` ending in `td_ns::DefaultResultEntry`, `MACRO_POPULATE_DISPATCH_VECTORS()`, and `init_(m)` defining `_` and `__result_type`. +3. Register `init_(m)` in `extensions/ufunc/elementwise_functions/common.cpp` and add the source to `_elementwise_sources` in `extensions/ufunc/CMakeLists.txt`. +4. Wrap it in Python (see `dpnp/AGENTS.md`). + +A new extension copies an existing extension's `CMakeLists.txt` (there is no helper function), keeps `add_dependencies( _usmarray)`, and is added with `add_subdirectory` in `dpnp/CMakeLists.txt`. + +## Rules for Changes +These also apply to `dpnp/tensor/libtensor/`. +- C++17. Include what you use (``, ``, ...), use `std::`-qualified fixed-width types (`std::int64_t`, `std::size_t`), make by-value parameters and locals `const` (`constexpr` where possible), and pass non-output parameters as `const T &`. +- Headers use `#pragma once`. +- Python-facing functions take `sycl::queue &exec_q` and `const std::vector &depends`, and return `std::pair` (keep-alive host-task event, compute event). Keep arguments alive with `dpnp::utils::keep_args_alive`. +- Validate inputs before launching: queue compatibility, contiguity, writability, memory overlap (including the same logical tensor), dtype and ndim. Reuse the helpers neighbouring files use: `check_queue`, `check_c_contig`, `check_no_overlap`, `check_writable` from `extensions/common/ext/validation_utils.hpp` (`indexing`, `statistics`, `ufunc`), or `dpnp::utils::queues_are_compatible`, `dpnp::tensor::overlap::MemoryOverlap` and `dpnp::tensor::validation::CheckWritable` (other extensions and libtensor). Don't silently accept strided input. +- Throw `py::value_error` for bad arguments and `std::runtime_error` for internal failures; catch oneMKL and SYCL exceptions and rethrow them. +- Guard double, complex and half with `dpnp::tensor::type_utils::validate_type_for_device(q)`; don't use double literals or casts in kernels that must run without fp64. +- `blas`, `fft`, `lapack` and `sparse` build against either oneMKL or oneMath (`USE_ONEMATH`); `vm` is oneMKL-only and reports availability through `_is_available()`. +- Release the GIL around blocking oneMKL calls (`py::gil_scoped_release`). +- A module that may launch more than `INT_MAX` work-items needs `-fno-sycl-id-queries-fit-in-int` in both `target_compile_options` and `target_link_options` (DPC++ 2026.2 and later check it at link time). +- clang-format 23 formats C++ (`.clang-format`: LLVM base, 4-space indent, 80 columns); gersemi formats CMake at 88 columns, except `cmake/Modules/`. + +## Build +```bash +python scripts/build_locally.py # Release, Ninja, icx/icpx, then pip install -e +python scripts/build_locally.py --clean --debug # remove _skbuild first; Debug build +python scripts/build_locally.py --onemath # oneMath instead of oneMKL +python scripts/build_locally.py --target-cuda # plus NVIDIA sm_50 (or --target-cuda=sm_80); --target-hip= for AMD +``` +`--target-cuda`, `--target-hip` and `--onemath-dir` imply oneMath. Extra CMake options go through `--cmake-opts="..."`; the `--onemkl_interfaces*` options are deprecated. The in-place build copies the `.so` files into the source tree; they are gitignored. diff --git a/dpnp/tensor/AGENTS.md b/dpnp/tensor/AGENTS.md new file mode 100644 index 000000000000..b3cbc46c64ba --- /dev/null +++ b/dpnp/tensor/AGENTS.md @@ -0,0 +1,24 @@ +# AGENTS.md - dpnp.tensor (dpnp/tensor/) + +## Purpose +`dpnp.tensor` is the Array API namespace with `usm_ndarray`, migrated from `dpctl.tensor` in 0.20.0. `dpnp.ndarray` and most dpnp functions are built on it. + +## Layout +- `*.py`: Array API functions; `_elementwise_funcs.py` holds the elementwise function objects +- `*.pyx`, `*.pxd`: Cython modules, all built by glob. `_usmarray.pyx` defines `usm_ndarray`; its `public api` generates `dpnp/include/dpnp/tensor/_usmarray*.h` +- `libtensor/include/kernels/`, `libtensor/include/utils/` (`type_dispatch*.hpp`, `type_utils.hpp`, ...), `libtensor/source/`: C++ for `_tensor_impl`, `_tensor_elementwise_impl`, `_tensor_reductions_impl`, `_tensor_sorting_impl`, `_tensor_accumulation_impl` and `_tensor_linalg_impl` +- Tests: `dpnp/tests/tensor/` + +## Rules for Changes +- `dpnp.tensor` follows the Array API standard, with no NumPy extensions; `dpnp` builds NumPy compatibility on top of it and also stays Array API conformant. Put NumPy-only behaviour in the `dpnp` layer unless the kernel itself is wrong. +- A libtensor change also changes `dpnp`: run `dpnp.tests.tensor` and the affected `dpnp/tests/test_*.py`. +- Many docstrings here still say `dpctl.tensor`; write `dpnp.tensor` in new code and don't copy the old name. +- C++ rules are in `dpnp/backend/AGENTS.md`. libtensor files add a `/// \file` block after the license header. +- New elementwise function: kernel in `libtensor/include/kernels/elementwise_functions/.hpp`; `populate__dispatch_vectors` and `init_` in `libtensor/source/elementwise_functions/.cpp`; register it in `libtensor/source/elementwise_functions/elementwise_common.cpp`; list the source in `dpnp/tensor/CMakeLists.txt`; add the Python object to `_elementwise_funcs.py` and export it in `__init__.py`. +- Tests in `dpnp/tests/tensor/` get a queue with `get_queue_or_skip()` and skip unsupported dtypes with `skip_if_dtype_not_supported(dt, q)` (`dpnp/tests/tensor/helper/`). + +## Verification +```bash +python -m pytest -n auto -ra --pyargs dpnp.tests.tensor +``` +The Array API conformance job in `conda-package.yml` tests only `dpnp` (`ARRAY_API_TESTS_MODULE: 'dpnp'`); a check for `dpnp.tensor` itself is not added yet, so verify spec compliance of `dpnp.tensor` changes against the spec. A libtensor change can still move the `dpnp` results, and the job never fails the build, so read its PR comment. diff --git a/dpnp/tests/AGENTS.md b/dpnp/tests/AGENTS.md new file mode 100644 index 000000000000..bb78e2ee3a82 --- /dev/null +++ b/dpnp/tests/AGENTS.md @@ -0,0 +1,41 @@ +# AGENTS.md - Tests (dpnp/tests/) + +## Layout +- `test_.py`: one file per dpnp module, comparing dpnp with NumPy; `ndarray` methods and attributes go in `test_ndarray.py` +- `test_sycl_queue.py`, `test_usm_type.py`: compute-follows-data, i.e. the result's queue and USM type follow the inputs +- `test_strides.py`: strided and negative-stride inputs; `test_umath.py` tests every `numpy.ufunc` that dpnp has, automatically +- `third_party/cupy/`, `third_party/cupyx/`: CuPy's tests ported with `import dpnp as cupy`; `numpy_cupy_*` decorators run each test with `xp=dpnp` and `xp=numpy` +- `tensor/`: `dpnp.tensor` tests (see `dpnp/tensor/AGENTS.md`) +- `skipped_tests.tbl` (non-GPU default device), `skipped_tests_gpu.tbl` (GPU, or `DPNP_QUEUE_GPU=1`), plus `skipped_tests_cuda.tbl` on CUDA: one exact node ID per line, chosen by `conftest.py`; there is no comment syntax +- `tests_perf/` is never collected + +## Rules for Changes +- Compare with `assert_dtype_allclose(result, expected)` and build inputs with `generate_random_numpy_array` (seeded) from `helper.py`. Don't hand-write dtype lists or use unseeded `numpy.random`. +- Compare against NumPy, or SciPy for `dpnp.scipy`. Where the reference's result depends on how it was built, assert the value the spec gives instead and say why in a comment. +- Parametrize dtypes with `get_all_dtypes()`, `get_float_dtypes()`, `get_float_complex_dtypes()` and similar. They drop float64/complex128 on devices without fp64, so build the NumPy and dpnp inputs with the same parametrized dtype. Never hardcode a float64 NumPy reference: it passes on CPU and fails on GPUs without fp64. Guard fp64-only tests with `has_support_aspect64()`. +- `get_all_dtypes()` includes `None`; pass `no_none=True` unless the default dtype is under test. `get_integer_dtypes()` returns only int32/int64 unless `DPNP_TEST_ALL_INT_TYPES` is set. +- `dpnp/tests/__init__.py` replaces `numpy.testing.assert_*` with wrappers that accept dpnp arrays and default to `strict=True` (dtype and shape must match) with NumPy 2 on fp64 devices. Pass `strict=False` only when a dtype difference is expected. +- Don't copy dpnp results to the host just to compare two dpnp arrays. +- Group tests in a `TestXxx` class with shared parametrizations; extend existing tests rather than duplicating them. +- Error cases run on both libraries: parametrize over `xp` in `[dpnp, numpy]` so both must raise. Gate NumPy-version-dependent behaviour with `numpy_version()` (or `testing.with_requires` in CuPy tests). +- Compute-follows-data error tests expect `(ValueError, ExecutionPlacementError)`: dpnp raises either, depending on the helper. +- A new function is added to the matching parametrized lists in `test_sycl_queue.py` and `test_usm_type.py` (`test_1in_1out`, `test_2in_1out`, ...); elementwise ones also to `test_strides.py`. +- Markers are strict and only `slow` and `multi_gpu` are registered. Tests that need huge allocations are `@pytest.mark.slow`; they never run in CI by design. +- Skip lists are for device- or backend-specific failures only; use the full parametrized node ID. +- CI runs the installed package (`--pyargs dpnp.tests`), so a new test directory or data file needs a matching glob in `package_data` in `setup.py`, or CI silently skips it. + +## CuPy Tests (third_party/) +- They check parity with CuPy and mirror upstream: no dpnp-only tests (those go in `test_*.py`), and code that doesn't apply is commented out, not deleted. +- When a feature lands, remove its `pytest.mark.skip("... is not supported yet")` and any matching `.tbl` entries. +- Use `type_check=has_support_aspect64()` in `numpy_cupy_*` decorators. + +## Running +```bash +python -m pytest -ra dpnp/tests/test_mathematical.py # one file, in-place build +SKIP_TENSOR_TESTS=1 python -m pytest -n auto -ra --pyargs dpnp.tests # as CI; -n needs pytest-xdist +python -m pytest -n auto -ra --pyargs dpnp.tests.tensor # tensor tests, a separate CI step +python -m pytest -ra -m slow dpnp/tests/test_fft_gil.py # slow tests only +``` +- `-m "slow or not slow"` still skips slow tests: `conftest.py` checks the `-m` expression for the substring `"not slow"`. +- Env vars in `config.py` are read with `bool(os.getenv(...))`, so any non-empty value, including `0`, turns them on; only `SKIP_TENSOR_TESTS` is parsed as an integer. +- `python -m dpctl -f` lists the available devices. From c9b533da5a8e9cbf485020462be85b8ac0b43da1 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Wed, 14 Oct 2026 21:59:40 +0200 Subject: [PATCH 2/4] Update test guidance for the fixes in #3103 -m "slow or not slow" now runs slow tests, "0" disables the test env flags, and large-allocation tests skip on USMAllocationError. --- dpnp/tests/AGENTS.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/dpnp/tests/AGENTS.md b/dpnp/tests/AGENTS.md index bb78e2ee3a82..e61c9a0705d6 100644 --- a/dpnp/tests/AGENTS.md +++ b/dpnp/tests/AGENTS.md @@ -20,6 +20,7 @@ - Error cases run on both libraries: parametrize over `xp` in `[dpnp, numpy]` so both must raise. Gate NumPy-version-dependent behaviour with `numpy_version()` (or `testing.with_requires` in CuPy tests). - Compute-follows-data error tests expect `(ValueError, ExecutionPlacementError)`: dpnp raises either, depending on the helper. - A new function is added to the matching parametrized lists in `test_sycl_queue.py` and `test_usm_type.py` (`test_1in_1out`, `test_2in_1out`, ...); elementwise ones also to `test_strides.py`. +- A test that may exceed the device's maximum allocation size skips on `USMAllocationError` (from `dpnp.exceptions`), not `MemoryError`, which it does not derive from. - Markers are strict and only `slow` and `multi_gpu` are registered. Tests that need huge allocations are `@pytest.mark.slow`; they never run in CI by design. - Skip lists are for device- or backend-specific failures only; use the full parametrized node ID. - CI runs the installed package (`--pyargs dpnp.tests`), so a new test directory or data file needs a matching glob in `package_data` in `setup.py`, or CI silently skips it. @@ -35,7 +36,7 @@ python -m pytest -ra dpnp/tests/test_mathematical.py # one fi SKIP_TENSOR_TESTS=1 python -m pytest -n auto -ra --pyargs dpnp.tests # as CI; -n needs pytest-xdist python -m pytest -n auto -ra --pyargs dpnp.tests.tensor # tensor tests, a separate CI step python -m pytest -ra -m slow dpnp/tests/test_fft_gil.py # slow tests only +python -m pytest -ra -m "slow or not slow" dpnp/tests/test_fft_gil.py # all tests, slow included ``` -- `-m "slow or not slow"` still skips slow tests: `conftest.py` checks the `-m` expression for the substring `"not slow"`. -- Env vars in `config.py` are read with `bool(os.getenv(...))`, so any non-empty value, including `0`, turns them on; only `SKIP_TENSOR_TESTS` is parsed as an integer. +- Env flags in `config.py` are off when unset, empty, `0`, `false`, `no` or `off`; any other value turns them on. - `python -m dpctl -f` lists the available devices. From 0c194c34645acfad17a1dc5ae6b7a58514f77f9f Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Thu, 15 Oct 2026 12:33:03 +0200 Subject: [PATCH 3/4] Update guidance for the documentation cleanup in #3106 doc/docstring_template.py is current again, dpnp/tensor no longer mentions dpctl.tensor, and placeholder pages are marked :orphan:. --- doc/AGENTS.md | 1 + dpnp/AGENTS.md | 2 +- dpnp/tensor/AGENTS.md | 1 - 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/AGENTS.md b/doc/AGENTS.md index be6bb453f511..69153330118f 100644 --- a/doc/AGENTS.md +++ b/doc/AGENTS.md @@ -9,6 +9,7 @@ ## Rules for Changes - A new public function goes in the autosummary block of the matching `reference/.rst`; ufuncs also go in `reference/ufunc.rst`, `ndarray` members in `reference/ndarray.rst`. These pages already list many NumPy names that dpnp lacks, so check before adding a duplicate. - A new namespace needs its own `reference/.rst` and a toctree entry: in `reference/routines.rst` for NumPy namespaces (model: `reference/fft.rst`), in `reference/scipy.rst` for SciPy ones (model: `reference/scipy_linalg.rst`). +- A page kept out of every toctree on purpose, such as a placeholder, is marked `:orphan:`, or Sphinx warns about it. - Move a deprecated function from its reference page to `reference/_deprecated.rst` (an orphan page that keeps its stub) and remove cross-links to it. - A new wrapper base class (not a subclass of the existing `DPNP*Func` classes) must be added to `conf.py` and `_can_document_member`, or autodoc skips its instances. - Update `quick_start_guide.rst` (and `README.md`) when build options or supported Python versions change, and `reference/array_api.rst` when the Array API version or the conforming namespaces change. diff --git a/dpnp/AGENTS.md b/dpnp/AGENTS.md index 81338636ee14..635f25bb03c1 100644 --- a/dpnp/AGENTS.md +++ b/dpnp/AGENTS.md @@ -25,7 +25,7 @@ - `dpnp.tensor` replaces `dpctl.tensor`, which no longer exists here; `SyclQueue`, `SyclDevice`, `dpctl.memory` and `SequentialOrderManager` still come from `dpctl`. ## Docstrings -numpydoc, rendered by `doc/conf.py`. Model a new docstring on an existing function; `doc/docstring_template.py` is outdated. +numpydoc, rendered by `doc/conf.py`. `doc/docstring_template.py` shows the layout; model a new docstring on an existing function. - Sections in the order `doc/conf.py` renders them: summary, ``For full documentation refer to :obj:`numpy.`.``, Parameters, Returns, Raises, Warns, Warnings, Limitations, See Also, Notes, References, Examples. - Array parameters are typed `{dpnp.ndarray, usm_ndarray}`, optional ones `{None, dpnp.ndarray, usm_ndarray}, optional`; the return is `out : dpnp.ndarray`. - Every keyword with a default ends with a blank line and ``Default: ````.`` diff --git a/dpnp/tensor/AGENTS.md b/dpnp/tensor/AGENTS.md index b3cbc46c64ba..9ef039a17804 100644 --- a/dpnp/tensor/AGENTS.md +++ b/dpnp/tensor/AGENTS.md @@ -12,7 +12,6 @@ ## Rules for Changes - `dpnp.tensor` follows the Array API standard, with no NumPy extensions; `dpnp` builds NumPy compatibility on top of it and also stays Array API conformant. Put NumPy-only behaviour in the `dpnp` layer unless the kernel itself is wrong. - A libtensor change also changes `dpnp`: run `dpnp.tests.tensor` and the affected `dpnp/tests/test_*.py`. -- Many docstrings here still say `dpctl.tensor`; write `dpnp.tensor` in new code and don't copy the old name. - C++ rules are in `dpnp/backend/AGENTS.md`. libtensor files add a `/// \file` block after the license header. - New elementwise function: kernel in `libtensor/include/kernels/elementwise_functions/.hpp`; `populate__dispatch_vectors` and `init_` in `libtensor/source/elementwise_functions/.cpp`; register it in `libtensor/source/elementwise_functions/elementwise_common.cpp`; list the source in `dpnp/tensor/CMakeLists.txt`; add the Python object to `_elementwise_funcs.py` and export it in `__init__.py`. - Tests in `dpnp/tests/tensor/` get a queue with `get_queue_or_skip()` and skip unsupported dtypes with `skip_if_dtype_not_supported(dt, q)` (`dpnp/tests/tensor/helper/`). From a9f79ed8b06b2efbe49475f63e2f94a2af37bb61 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Thu, 15 Oct 2026 20:35:52 +0200 Subject: [PATCH 4/4] Update CMake guidance for the extension helper in #3111 New extensions call build_dpnp_pybind11_ext instead of copying an existing CMakeLists.txt, and request -fno-sycl-id-queries-fit-in-int with its LARGE_RANGES option. --- dpnp/backend/AGENTS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dpnp/backend/AGENTS.md b/dpnp/backend/AGENTS.md index 2297cc536078..2e18a86efe1a 100644 --- a/dpnp/backend/AGENTS.md +++ b/dpnp/backend/AGENTS.md @@ -14,7 +14,7 @@ 3. Register `init_(m)` in `extensions/ufunc/elementwise_functions/common.cpp` and add the source to `_elementwise_sources` in `extensions/ufunc/CMakeLists.txt`. 4. Wrap it in Python (see `dpnp/AGENTS.md`). -A new extension copies an existing extension's `CMakeLists.txt` (there is no helper function), keeps `add_dependencies( _usmarray)`, and is added with `add_subdirectory` in `dpnp/CMakeLists.txt`. +A new extension's `CMakeLists.txt` lists its sources and calls `build_dpnp_pybind11_ext( SOURCES ... [INCLUDE_DIRS ...] [LARGE_RANGES])` from `dpnp/CMakeLists.txt`, which sets the options all extensions share; only oneMKL/oneMath linking stays in the extension. Add it with `add_subdirectory` in `dpnp/CMakeLists.txt`, and put an option every extension needs into the helper, not into one extension. ## Rules for Changes These also apply to `dpnp/tensor/libtensor/`. @@ -26,7 +26,7 @@ These also apply to `dpnp/tensor/libtensor/`. - Guard double, complex and half with `dpnp::tensor::type_utils::validate_type_for_device(q)`; don't use double literals or casts in kernels that must run without fp64. - `blas`, `fft`, `lapack` and `sparse` build against either oneMKL or oneMath (`USE_ONEMATH`); `vm` is oneMKL-only and reports availability through `_is_available()`. - Release the GIL around blocking oneMKL calls (`py::gil_scoped_release`). -- A module that may launch more than `INT_MAX` work-items needs `-fno-sycl-id-queries-fit-in-int` in both `target_compile_options` and `target_link_options` (DPC++ 2026.2 and later check it at link time). +- An extension whose kernels may be launched over more than `INT_MAX` work-items passes `LARGE_RANGES` to `build_dpnp_pybind11_ext`, which adds `-fno-sycl-id-queries-fit-in-int` at compile and link time (DPC++ 2026.2 and later need it at link time). Outside the helper (`dpnp_backend_c`, `dpnp/tensor`) pass it to both `target_compile_options` and `target_link_options`. - clang-format 23 formats C++ (`.clang-format`: LLVM base, 4-space indent, 80 columns); gersemi formats CMake at 88 columns, except `cmake/Modules/`. ## Build