diff --git a/package-structure-code/python-package-structure.md b/package-structure-code/python-package-structure.md index 2ea629fb..c16e0994 100644 --- a/package-structure-code/python-package-structure.md +++ b/package-structure-code/python-package-structure.md @@ -133,6 +133,8 @@ Including the **tests/** directory in your **src/package** directory ensures tha Be sure to read the [pytest documentation for more about including tests in your package distribution](https://docs.pytest.org/en/7.2.x/explanation/goodpractices.html#choosing-a-test-layout-import-rules). +If you decide to ship tests with your package, see [shipping tests with your package](in-tree-tests) for how to lay them out and run them with `pytest --pyargs`. + :::{admonition} Challenges with including tests and data in a package wheel :class: tip diff --git a/tests/in-tree-tests.md b/tests/in-tree-tests.md new file mode 100644 index 00000000..93bf97ba --- /dev/null +++ b/tests/in-tree-tests.md @@ -0,0 +1,111 @@ +(in-tree-tests)= +# Shipping tests with your package (in-tree tests) + +Throughout this guide we recommend keeping your `tests/` directory **outside** +of your package directory (an "out-of-tree" layout). This keeps your wheel +small and, combined with the [src layout](src-layout-test), guarantees that +your tests run against the *installed* package rather than the files in your +working directory. + +Some packages, however, benefit from shipping their tests inside the package +("in-tree" tests). This is common in the scientific Python ecosystem: NumPy, +SciPy, and scikit-image all do it. + +## When in-tree tests make sense + +Consider in-tree tests if your package: + +* **Contains compiled code** (C, C++, Cython, Rust, or Fortran). A wheel built + on one platform may import fine but produce wrong results on another. + Letting users run the test suite against the wheel they installed + is the most reliable way to find out. +* **Is built by downstream packagers** such as conda-forge, Debian, or Spack. + Packagers routinely run a package's test suite as part of their build, and + they can only do so if the tests ship with the package. +* Has users who need to **verify an installation** on an HPC cluster or + another locked-down environment where they can't clone the repository. + +If your package is pure Python with a small test suite, out-of-tree tests are +simpler, and we recommend sticking with them. + +## Layout + +Move `tests/` inside your package directory and give it an `__init__.py` so +that it is discovered and installed like any other subpackage: + +```bash +src/ + mypackage/ + __init__.py + core.py + tests/ + __init__.py + test_core.py +docs/ +pyproject.toml +``` + +:::{admonition} Make sure the tests ship +:class: tip + +Check your build backend's package-discovery settings. If you list packages +explicitly, add `mypackage.tests`. After building, inspect the wheel +(`unzip -l dist/*.whl`) to confirm that the test files are included. +::: + +## Run tests against the installed package + +With tests inside the package, run them using pytest's `--pyargs` flag. This +tells pytest to import `mypackage` from the active environment, locate its +`tests` subpackage, and run it, regardless of your current working directory: + +```bash +pytest --pyargs mypackage +``` + +Many scientific packages also expose a convenience function so that users +don't need to remember the flag: + +```python +# src/mypackage/__init__.py +def test(*args): + """Run the mypackage test suite.""" + import pytest + + return pytest.main(["--pyargs", "mypackage", *args]) +``` + +```python +>>> import mypackage +>>> mypackage.test() +``` + +Because `pytest` is only imported inside the function, it stays an optional +dependency. Add it to a `test` extra in your `pyproject.toml`: + +```toml +[project.optional-dependencies] +test = ["pytest"] +``` + +## Pitfalls + +**Running tests on the source tree by accident.** This is the main reason we +recommend out-of-tree tests for beginners. If you run +`pytest src/mypackage/tests` from the repository root, Python may import your +*un-built* source files instead of the installed package. With compiled code, +this surfaces as confusing `ImportError`s about missing extension modules. +Always use `pytest --pyargs mypackage` (or a `nox` or `tox` session that +installs the package first) rather than pointing pytest at a path. + +**Test data.** Don't ship large datasets inside your wheel. See +[don't include test suite datasets in your package](src-layout-test) +for alternatives such as Pooch. + +**Wheel size.** Every test file is downloaded by every user. Keep the in-tree +suite focused. Very slow or data-heavy tests can stay out of tree and run only +in CI. + +For a real-world example, see how +[NumPy](https://github.com/numpy/numpy/tree/main/numpy) organizes +`numpy/tests` and `numpy//tests`, and exposes `numpy.test()`. diff --git a/tests/index.md b/tests/index.md index 63a04f17..497c3e8f 100644 --- a/tests/index.md +++ b/tests/index.md @@ -57,6 +57,15 @@ Set up continuous integration with GitHub Actions to run tests across Python versions and operating systems. :::: +::::{grid-item-card} ✨ Shipping tests with your package ✨ +:link: in-tree-tests +:link-type: doc +:class-card: left-aligned + +Learn when it makes sense to ship tests inside your package (for example, +if it has compiled code) and how users can run them with `pytest --pyargs`. +:::: + ::::{grid-item-card} ✨ Code coverage ✨ :link: code-cov :link-type: doc @@ -86,6 +95,7 @@ Test types Run tests locally Run tests with Nox Run tests online (using CI) +Shipping tests with your package Code coverage ```