Replace pytest with plain.test, Plain's own test runner

[davegaeddert]

Plain owns its test runner. plain-pytest is deleted, and every suite in the repo (29 suites, 4,386 tests) runs on plain.testing.

What an app sees

One dev package, plain.testing, provides everything a test file imports and the plain test command that runs them. Core ships no test code.

from plain.testing import Client, cases, raises


def test_signup():
    response = Client().post("/signup", form_data={"email": "a@example.com"})
    assert response.status_code == 302

• A test is a function, and a file is the group. There are no test classes.
• Nothing is passed to a test by name. A test gets what it needs in its body, by calling a helper or entering a with block, and takes values only from @cases.
• Decorators declare, bodies acquire. @cases, @skip and @tag attach static facts. Runtime state enters through override_settings, patch, raises, skip_test, and the captures.
• Protection is automatic, setup is not. Packages register a TestLifecycle (plain-postgres wraps each test in a rolled-back transaction; plain-email resets the outbox). A project declares its own in tests/lifecycle.py.
• Helpers live with their owner. plain.postgres.testing, plain.email.testing, plain.auth.testing, plain.sessions.testing.
• The runner knows nothing about pytest. It reports what is wrong in its own terms. The upgrade notes below are the only migration guide.

What a run prints

A run that passes prints two lines: the collected count and the summary. A failure prints every value inside the assert, a diff of two large values, the test's locals, what the test wrote, and the command that runs it again.

assert len(rows) == expected
  len(rows) = 2
    rows = [{'id': 1}, {'id': 2}]
  expected = 3

plain test --json prints the same run as one document.

The command

Every option has one name: --match, --tag, --exclude-tag, --fail-fast, --verbose, --show-output, --full-values, --json, --list-passed. A target is a directory, a file, a test (tests/test_x.py::test_name) or a line (tests/test_x.py:42). Exit codes run 0 to 4, and 130.

Where the time goes

The loop an agent runs most is one test, and in that loop the test itself is about 2 percent of the wall clock. The rest was plain.runtime.setup() and building a test database from scratch, and the runner reported none of it: "1 passed in 0.52s" for a run that took 1.15 s.

• The run reports its phases. Under --verbose, or whenever a run spends more than a second outside its tests, it prints a chronological block: interpreter start, imports, each setup hook and import over 50 ms by name, the test database, the tests, teardown. --json always carries phases. An app sees its own slow import instead of guessing.
• Test databases are clones of a template. The first run migrates and converges a template named by a digest of the migrations, the models, and the plain-postgres, psycopg and Postgres versions, and leaves it. Every later run clones it with CREATE DATABASE … TEMPLATE (about 25 ms) into a database of its own, so any number of runs can go at once in one checkout. A changed migration or model is a new template, and the old one is dropped on a later run by its own record, never by name. In example/, one test went from 1.05 s to 0.52 s wall clock; on three real apps of 58, 1,000 and 1,470 tests, from 0.44, 1.11 and 1.95 s to 0.35, 0.64 and 0.83 s, with the database phase at 0.10 s in all three. What is left is runtime setup, most of it the apps' own imports.

What core gains

• Request(body=...). A request could not be constructed with a body; the server and the client now build one the same way.
• plain.server.inprocess: InProcessServer().handle(request).send() handles a request on the calling thread and returns what would be sent. The client is built on it and reaches nothing private in core.
• plain docs <name> --api shows annotated attributes. Model fields had vanished from it.

plain request moves to plain-dev. It is the client pointed at the dev database, and it already refused to run without DEBUG.

Fixes that stand apart from the runner

Found on the way, and wrong on master too:

• A thread started by the code under test connected to the development database. The test had the test database through a context variable, and a new thread has none. For the whole of a run the configured database is now the test database.
• schema_lock() released its lock by closing its session, and Postgres frees it a moment later, so the next acquisition could wait out a five-second retry. It unlocks first.
• plain db clean could drop a database a live checkout uses, because it went by the checkout recorded in the database's metadata and not by which checkout is configured to use it. It no longer does, it always lists before it drops, and it has --dry-run and no --yes.
• patch restored what attribute lookup returned. That left an inherited attribute copied onto the subclass, and put a plain function back where a staticmethod had been.
• boom_on_500 in plain-templates' tests read self.template_name, which Template doesn't have, so it raised AttributeError on every template and the test passed for the wrong reason.
• plain agent install compared file times, so a fresh checkout (every file dated today) never received a package's older rule text and said "up to date" while differing from what the packages ship. It goes by content, and names each file it wrote or removed.
• A mistyped or uninstalled command started the dev services before the CLI knew the command didn't exist. They start only for a command that does.
• plain code check skipped any directory named test, which took every package's test/ helpers, and all of plain.test while it had that name, out of the annotation check. tests/ is on the type checker's search path now.

Not built

--lf, --pdb, @timeout, freeze_time, parallelism inside one run, shuffle and flake classification, --changed, browser testing, model factories, and the built-in suite.

The /plain-upgrade automation for this migration is not written. Section 19 of the upgrade notes says what it has to do.

Before release

• plain.testing pins plain>=0.165.0, which does not have the in-process server. The release has to bump core and raise the bound.
• plain-testing's changelog is its title only, for the release skill to write. What each package's notes must say is in section 18 of the upgrade notes.

Tried on five apps

Five apps were migrated against the branch, each in a worktree, with the upgrade notes and the runner's output as the only guide. Two more wait on browser testing, which is not built.

Suite	How	Result
4 tests	one agent, docs only	all pass
58 tests	one agent, docs only, no framework source read	all pass, about five minutes
223 tests	one agent	221 pass, 2 need a service the worktree lacks; the suite had been kept in app/, which the runner now reports instead of passing over
1,008 tests, nested conftest.py files	one agent, upgrade notes only, no script	all pass in 6 s, against 10 s under pytest, migrated in 21 minutes
1,470 tests, 274 fixtures, an 860-line conftest.py	one agent wrote helpers, tests/lifecycle.py and a migration script; five did the files in parallel	all pass in 104 s, against 166 s under pytest

Nothing in any suite was something the runner could not express. No migration script ships: at the larger sizes the agent wrote its own throwaway scripts for the repeated rewrites without being asked, and section 14 of the notes says which rewrites to script and which three counts catch every silent error.

Review shape

Read carefully:

• plain-testing/plain/testing/runner/ — collection, loading, the assertion rewriter, execution, output capture, failure description, reporting
• plain-testing/plain/testing/ — the vocabulary and the client
• plain/plain/server/inprocess.py, plain/plain/http/request.py, and the handler changes in plain/plain/internal/handlers/
• plain-postgres/plain/postgres/testing/ — the lifecycle, the per-run databases, and leftovers.py, which decides what a run may drop
• plain-dev/plain/dev/postgres/cleaning.py, which decides what plain db clean may drop
• plain-dev/plain/dev/request/

The converted test files can be sampled. Two conversions were checked by script against what they came from: pytest to plain.testing (every test master has is here, function by function), and classes to functions (the same bodies, decorators, assertions and comments).

Verification

• ./scripts/test: 4,386 passed, 0 failed, 9 skipped, 29 suites
• ./scripts/server-test: h1spec, Autobahn WebSocket and OpenAPI conformance (1301/1301 on every check) pass; h2spec and the load tests need tools not installed here and ran in CI
• ./scripts/type-validate: 26 of 26, no errors
• ./scripts/fix: clean
• The project's databases are the same before and after a full run

---

Upgrade notes

Upgrade notes

Moving an app from pytest to plain.testing

This is the only migration guide. The runner, its README and its rule file say how tests are written and nothing about how they used to be, so nothing plain test prints will name pytest, a fixture or a conftest.py. Everything about getting from there to here is in this file.

plain.testing replaces plain.pytest, and there is no release where both run. Assertions don't change. What changes is how a test gets what it needs: nothing is passed to a test by name, and everything it depends on is an import, a call or a with block in its own file.

The steps are in the order they are performed. Each one leaves fewer errors standing for the next.

Contents

1. Dependencies
2. Rule files
3. Commands, flags and exit codes
4. Before the first run: what conftest.py guarded against
5. The first run
6. tests/__init__.py
7. conftest.py
8. Helper imports
9. What a file imported from pytest
10. Fixtures
11. Classes
12. The client
13. Spans, metrics, logs and queries
14. Do the repeated parts with a script, and count
15. Things with no replacement
16. Behavior that changed without a name changing
17. The last run
18. What the release notes must say
19. What /plain-upgrade has to do

1. Dependencies

In pyproject.toml:

• Replace plain.pytest with plain.testing in the dev dependencies.
• Remove pytest and every pytest-* plugin.
• Delete [tool.pytest.ini_options], and pytest.ini if there is one.
• plain.test came with Plain, so a project could import it with nothing installed. It is gone. What it held is plain.testing, a package of its own: the package is plain.testing, the command is still plain test. If anything outside the tests imports plain.test, change the import to plain.testing and add plain.testing wherever that code's dependencies are declared.
• plain request comes from plain.dev now. Nothing to change if plain.dev is a dev dependency, which it is in a project made with plain-start.

Then uv sync.

Each package's test helpers are named after the package that uses them: plain.postgres.test is plain.postgres.testing, and the same for plain.email, plain.auth and plain.sessions. A from plain.auth.test import login_client becomes from plain.auth.testing import login_client. The runner reports the old name as an import error with its file and line.

Remove what set up the test environment by hand:

• PLAIN_CONNECT_EXPORT_ENABLED=false in .env.test, if it was there for the tests. plain.connect exports nothing during a test run.
• A suffix on PLAIN_POSTGRES_URL to keep runs apart. Each run makes a test database of its own and drops it.

2. Rule files

Run uv run plain agent install. It writes each rule and skill that isn't what the installed packages ship, and names the ones it wrote:

Agent: wrote 2, removed 1 in .claude/
  wrote rules/plain-dev.md
  wrote rules/plain-testing.md
  removed rules/plain-test.md

It goes by what the files hold, so it writes a rule your checkout has an older text of, however recently the file was written. Agent: up to date means nothing differed. Until it has run, .claude/rules/plain-test.md is the old one, and says to use pytest fixtures; the run removes it and writes plain-testing.md in its place.

3. Commands, flags and exit codes

plain test is the command, as before. What follows it changed. Update scripts, CI and any docs of your own.

Before	Now
-k TEXT	--match TEXT. The text as written: not a pattern, not an expression
-k "a and not b", -k "a or b"	no equivalent. Use targets, --tag and --exclude-tag
-x, --exitfirst, --maxfail=1	--fail-fast
-v	--verbose
-s, --capture=no	--show-output
-m slow, -m "not slow"	--tag slow, --exclude-tag slow
-q	nothing. A run that passes prints two lines
-n 4, -p no:randomly, --reuse-db	remove them
--pdb, --lf, --ff, --durations	no equivalent. breakpoint() in a test gets the terminal
-W error	PYTHONWARNINGS=error plain test, or python -W error -m plain.testing
tests/test_x.py::TestCart::test_add	tests/test_x.py::test_add, once the class is gone
a traceback's file and line	tests/test_x.py:42 runs the test that line is in
python -m pytest	python -m plain.testing
coverage run -m pytest	coverage run -m plain.testing

Every option has one name, and it is the long one. -k, -x, -v and -s are "No such option" and exit 2.

Exit codes:

Code	Meaning	pytest's
0	Every test that ran passed	0
1	A test failed, or a file couldn't be collected	1, 2
2	The command can't be used as given	4
3	Setting up failed and no test was run	3
4	No tests matched	5
130	Stopped with Ctrl-C	2

A script that treated 5 as "nothing to run" checks for 4.

What a run prints changed too. A test that passes prints nothing: a run that passes is Collected 164 tests and 164 passed in 3.21s. On a terminal a line says how far the run has got and is erased before the report. --verbose prints a line per test, with the outcome in lower case (passed tests/test_cart.py::test_empty_cart). Anything that read the old per-test lines reads --json, which is one document on stdout when the run is over.

4. Before the first run: what conftest.py guarded against

Do this before running anything, if the suite can reach a real service.

A conftest.py is not read any more, and nothing says so. An autouse fixture in one that kept every test away from the payment provider, the helpdesk or the mail server has stopped doing it, and the credentials in .env are still there. A test that forgot to stub a call now makes it.

Read every conftest.py, the one in tests/ and any below it, for autouse fixtures. Sort each by one question: does any test's assertion depend on it?

• No. It is the same for every test in the suite and no test asserts on it. It keeps tests from the outside world (no request reaches the payment provider, whatever the environment holds) or from each other (a rate limiter starts from zero, a cache is cleared). That is protection, and it goes in tests/lifecycle.py, now.
• Yes. It is setup: a setting pinned to the value some tests need, a user every test of a file reads. It goes in the body of each test that needs it, in step 10. That is so even when every test in a directory needs the same one. There is one lifecycle, for the whole suite, and none for a directory or a file.

## tests/lifecycle.py
from contextlib import contextmanager

from plain.testing import TestLifecycle, override_settings, patch

from app.accounts import throttles
from app.billing import payments


def refuse_to_send(*args, **kwargs):
    raise AssertionError("A test tried to reach the payment provider")


class AppTestLifecycle(TestLifecycle):
    @contextmanager
    def around_test(self, test):
        throttles.reset_all()
        with (
            override_settings(PAYMENTS_API_KEY=""),
            patch(payments, "send", refuse_to_send),
        ):
            yield

• The file is found by that exact path and holds exactly one TestLifecycle subclass, under any name. tests/lifecycles.py, or a lifecycle.py beside tests/, stops the run and says where it belongs.
• It is imported after the app is set up and before any test file, so it imports from app at the top, as a test file does. A conftest.py that put off an import until a fixture ran doesn't have to here.
• around_test(test) wraps every test. A patch a test makes on top of the lifecycle's wins inside the test, and the lifecycle's is back afterwards.
• A session-scoped autouse fixture is setup_worker() and teardown_worker(), which run once, before the first test and after the last.
• If a guard replaces something with a fake that raises, check that the code under test doesn't catch every exception. One that does swallows the guard. A fake that does nothing is the guard there.

Prove it guards before relying on it: with a made-up key in the environment, run one test that reaches the call with no stub of its own, and see that nothing was sent.

5. The first run

Run uv run plain test on the suite as it is. It runs what it can and reports the rest as collection errors, a file at a time. It says what is wrong in its own terms. This is what each thing a pytest suite holds comes out as:

The file has	The run prints	Step
import pytest	a traceback ending ModuleNotFoundError: No module named 'pytest'	9
tests that take fixtures	test_signup(db, client) takes parameters, and nothing passes them in., and each name's count	10
from tests.helpers import x	line 4: `from tests.helpers import x` should be `from helpers import x`	8
from .helpers import x	the same, with the path from tests/ of the module the dots reach	8
from helpers import x, for a module further down	line 4: `from helpers import x` should be `from billing.helpers import x`	8
a class with tests in it, or a TestCase	line 8: TestCart is a class with 3 tests in it.	11
a test with a yield in it	test_x() has a `yield` in it.	9
a test imported from another file	test_x is defined in billing, not in this file.	9
stacked parametrize, once converted	line 8: test_x has 2 @cases. A test takes one.	9
a suite kept in app/ (app/tests/, app/users/tests/)	one collection error app: how many test files are in app/, where, and that none was run. Exit 1	5
a call the client no longer takes	the test fails with Python's own TypeError or AttributeError	12

When more than one file has tests that take parameters, the run adds them up after the last file's error. That table is the list of fixtures the suite uses, and how much work each is:

parameters nothing passes in

  Tests in 61 files take these:

  db      605 tests in 58 files
  user    212 tests in 31 files
  client   96 tests in 12 files

A suite kept in app/

Tests live in tests/, beside app/. app/ is the application: it is imported as app, and it is what gets deployed. A test_*.py in it that has tests in it is not run, whether the run came to it by looking or was given it as a target, and the run reports all of them as one collection error and exits 1:

collection error app

  19 test files are in app/, and none of them was run:

    app/tests  19 files

  Tests live in tests/, beside app/. app/ is the application: it is
  imported as `app`, and it is what gets deployed. Move them to tests/. A
  test file is the same file there, and a helper module it imports is
  imported by its path from tests/.

Do this before anything else, because nothing else about those files is reported until they are moved:

• git mv app/tests tests, or for tests kept beside each package, git mv app/users/tests tests/users. Keep the directories: a file's id is its path.
• Helper modules that were in those directories move with them, and are imported by their path from tests/ (step 8). from app.tests.helpers import x becomes from helpers import x.
• Anything that named the old path names the new one: a CI step, a script, a testpaths setting that is going away anyway.
• A module of the application that is named test_*.py and defines no test (app/test_connection.py) is left alone and is not reported.

What the first run doesn't show

• A conftest.py. It is not read, so what it set up for every test is not set up, and nothing reports it. That is step 4, and step 7 for the rest of the file.
• A tests/__init__.py. Nothing imports it, so code in it doesn't run. Step 6.
• What is behind an error. A file is reported for what stops it first. One that imports pytest and has nothing else wrong that reading it shows gets the ModuleNotFoundError, and its tests' fixture parameters are reported only once the import is gone. One with an import the runner refuses gets that, and its import pytest fails on the run after. So the count of files with each problem goes up as well as down between runs, and the first run's totals are a lower bound.
• What a test would do if it ran. A file that can't be collected runs none of its tests, so a call the client no longer takes shows up only when its file is otherwise right.

Work through the steps below, then run it again.

6. tests/__init__.py

Delete tests/__init__.py, and every __init__.py under tests/. tests/ isn't a package: a test file is loaded on its own, a helper module is imported by its path from tests/, and a directory under tests/ is imported through with nothing in it.

If one had code in it, the code ran before every test. Sort it by the question in step 4: protection goes in tests/lifecycle.py, and setup goes in the tests that read it.

7. conftest.py

Move what each conftest.py holds, then delete the file. Nothing reads it. Its autouse fixtures that guard every test are in tests/lifecycle.py already, from step 4.

In conftest.py	Goes to
a fixture tests ask for (user, client)	a function in a helper module, such as tests/helpers.py, that a test calls
a fixture with teardown (yield)	a @contextmanager function in the helper module, entered with with
a fixture that returns a factory (make_user)	the factory itself, in the helper module
an autouse fixture that protects every test	tests/lifecycle.py: one TestLifecycle subclass, its around_test(test)
an autouse, session-scoped fixture	tests/lifecycle.py: setup_worker() and teardown_worker()
an autouse fixture that sets up what tests read	a helper each test that reads it calls or enters. Never the lifecycle
a plain function or a constant	the helper module
pytest_configure, pytest_collection_modifyitems, …	no equivalent. What it protected goes to the lifecycle, the rest is removed
pytest_plugins = [...]	removed
a conftest.py in a subdirectory	a helper module in that directory, named for what it holds. See below

The rule for the lifecycle is the question in step 4: if a test's assertions depend on it, it is setup and goes in the test.

A conftest.py in a subdirectory

tests/updates/conftest.py applied to the tests under tests/updates/ and no others. What it holds goes in a helper module in the same directory, tests/updates/update_helpers.py, which is imported by its path from tests/:

from updates.update_helpers import updates_enabled

An autouse fixture in it has no equivalent that applies to a directory. It was setup for that directory's tests, so it becomes a helper, and each test whose assertions depend on it enters it:

## tests/updates/update_helpers.py
from contextlib import contextmanager

from plain.testing import override_settings


@contextmanager
def updates_enabled():
    with override_settings(APP_UPDATES_ENABLED=True):
        yield

## tests/updates/test_jobs.py
from updates.update_helpers import updates_enabled


def test_the_sweep_queues_a_job():
    with updates_enabled():
        ...

Only the tests that reach code reading the setting need it. Find them by running the directory with the setting set the other way in the environment (PLAIN_APP_UPDATES_ENABLED=false uv run plain test tests/updates) and giving the helper to the ones that fail.

Name helper modules for what they hold. tests/ comes first on the import path, so a helper module or a directory under tests/ named like an installed package takes its place.

8. Helper imports

A helper module is imported by its path from tests/, from any test file in any directory, wherever the command runs from. tests/helpers.py is helpers, and tests/billing/refund_helpers.py is billing.refund_helpers. A directory under tests/ needs no __init__.py.

Before	Now
from tests.helpers import create_user	from helpers import create_user
from .helpers import create_user, in tests/	from helpers import create_user
from tests.conftest import make_user	from helpers import make_user
from conftest import make_user	from helpers import make_user
import tests.helpers	import helpers
from tests.billing.refund_helpers import create_refund	from billing.refund_helpers import create_refund
from .refund_helpers import create_refund, in tests/billing/	from billing.refund_helpers import create_refund
from ..helpers import create_user, in tests/billing/	from helpers import create_user
from refund_helpers import create_refund, in tests/billing/	from billing.refund_helpers import create_refund

Every spelling on the left is a collection error that prints the one on the right, with its line.

plain code check passes tests/ to the type checker as a search path, so these imports resolve there too.

9. What a file imported from pytest

Remove import pytest and replace what the file used. Everything in the right-hand column is imported from plain.testing unless it says otherwise.

pytest	Now
pytest.raises(E, match=...)	raises(E, match=...)
excinfo.value	caught.exception, readable after the block. Reading it inside the block raises
str(excinfo.value), excinfo.match(...)	str(caught.exception), or match= on raises
pytest.mark.parametrize("a,b", [...])	@cases((a, b), (a, b)). A single value is passed bare: @cases(1, 2)
pytest.param(..., id="x"), ids=[...]	case(..., id="x") inside @cases
the ids pytest made from the values, [5-500]	the same, when every value is a string, a number, a boolean, None or an enum member
ids of other values ([user0])	numbered: [0], [1]. Name them with case(..., id=)
stacked parametrize	one @cases(*[(a, b, c) for a in FIRST for b, c in SECOND]): each case one flat tuple
parametrize(..., indirect=True)	pass the value, and call the helper in the test
pytest.mark.skip(reason=...)	@skip("reason"). The reason is required
pytest.mark.skipif(condition, ...)	if condition: skip_test("reason") as the test's first line
pytest.skip("reason") in a test	skip_test("reason")
pytest.importorskip("x")	try: import x / except ImportError: skip_test(...) in the test
custom marks (pytest.mark.slow)	@tag("slow")
pytestmark = pytest.mark.slow	@tag("slow") on each test
pytest.mark.xfail	no equivalent. @skip it with the reason, or assert the failure with raises
pytest.mark.usefixtures("db")	removed. See step 10
pytest.mark.django_db, pytest.mark.asyncio	removed. The database is there, and async def test_* runs as written
pytest.approx(x, abs=...)	math.isclose(a, x, abs_tol=...) inside a bare assert
pytest.fail("why")	raise AssertionError("why")
pytest.warns(W)	with warnings.catch_warnings(record=True) as caught: and assert on caught
pytest.fixture	step 10
a test that yields values	@cases, or a @contextmanager helper for setup and cleanup
a test imported to be run here too	define it in the file that runs it, or import it under a name without test_
assert x == y, f"{x} != {y}"	assert x == y. A failure prints every value, a diff and the test's locals

10. Fixtures

Drop every fixture parameter, and get what it gave in the test's body. A test function takes no parameters except the values @cases passes.

## Before
def test_invite(db, client, user, settings):
    settings.INVITES_ENABLED = True
    client.force_login(user)
    ...

## Now
def test_invite():
    user = create_user()
    client = Client()
    login_client(client, user)
    with override_settings(INVITES_ENABLED=True):
        ...

Fixture	Now
db	nothing. Every test runs in a transaction that is rolled back
isolated_db	@isolated_db from plain.postgres.testing
transactional_db, a test that commits or runs DDL	@isolated_db
one of your own that a test reads	a function the test calls: user = create_user()
one of your own with teardown	with helper() as thing:
one that depended on other fixtures	a function that calls the functions those became
module- or session-scoped, for speed	a module-level functools.cache on the helper, if it is safe to share. Otherwise per test
client	Client() from plain.testing
settings	with override_settings(NAME=value):
monkeypatch.setattr(settings, "NAME", value)	with override_settings(NAME=value):. A setting is always this, never patch
monkeypatch.setattr(obj, "name", value)	with patch(obj, "name", value):. It takes the object, not a dotted string
monkeypatch.setattr("pkg.mod.name", value)	from pkg import mod, then with patch(mod, "name", value):
monkeypatch.setattr("pkg.mod.Class.name", value)	from pkg.mod import Class, then with patch(Class, "name", value):
a helper function that takes monkeypatch	a @contextmanager that yields what it returned. See below
monkeypatch.setenv("KEY", "value"), setitem	with patch(os.environ, "KEY", "value"):
monkeypatch.delenv, delattr, delitem	unittest.mock.patch.dict(os.environ) and delete inside it
monkeypatch.chdir(path)	with contextlib.chdir(path):
mocker (pytest-mock)	unittest.mock directly, or patch
otel_spans / otel_metrics	with capture_spans() as spans: / with capture_metrics() as metrics:. See step 13
caplog	with capture_logs() as logs:
capsys, capfd	with contextlib.redirect_stdout(io.StringIO()) as out:
tmp_path	with tempfile.TemporaryDirectory() as tmp: then Path(tmp)
tmp_path_factory	the same, where it is needed
request (request.param, request.node)	the value from @cases. There is no node
mailoutbox	from plain.email.testing import outbox. It is cleared around each test
testbrowser	no equivalent. See step 15
setup_method / teardown_method	a @contextmanager helper each test enters
setup_module, setup_class	a helper each test calls

monkeypatch is a restructuring, not a rename

monkeypatch.setattr lasted until the test ended. A with patch(...) lasts until its block ends. So everything in the test after the call moves inside the block, one level in:

## Before
def test_a_declined_card(monkeypatch):
    monkeypatch.setattr(settings, "PAYMENTS_LIVE", False)
    monkeypatch.setattr(payments, "send", decline)
    monkeypatch.setenv("PAYMENTS_KEY", "test")
    order = create_order()
    response = Client().post("/pay/", form_data={"order": order.id})
    assert response.status_code == 402


## Now
def test_a_declined_card():
    with (
        override_settings(PAYMENTS_LIVE=False),
        patch(payments, "send", decline),
        patch(os.environ, "PAYMENTS_KEY", "test"),
    ):
        order = create_order()
        response = Client().post("/pay/", form_data={"order": order.id})
        assert response.status_code == 402

• Consecutive patches are one with. Patches with nothing between them, or only setup that doesn't care about the order, merge into one parenthesized with, in the order they were made. A patch made after other statements opens a block of its own there, a level further in.
• Settings go together. Several monkeypatch.setattr(settings, ...) in a row are one override_settings(A=1, B=2).
• The assertions move in too. What is checked after the patched call usually depends on the patch, so the whole rest of the test goes inside. The exception is a capture, which is read after its own block ends (step 13).
• A patch inside an if, a for or a try lasted to the end of the test under pytest, and lasts to the end of that block now. Move it up, or use contextlib.ExitStack and stack.enter_context(patch(...)).
• What is inside triple-quoted strings doesn't move. A script that re-indents has to leave the lines of a multi-line string alone, or it changes the string.
• Where a dotted string ends. "app.billing.gateway.Gateway.charge" names an attribute, charge, of an object, Gateway, in a module, app.billing.gateway. The object is what is imported and passed to patch. The module ends at the last part that is a file or a directory; what follows it is reached by attribute.

A helper that takes monkeypatch

A function that a test called with its monkeypatch, to install a fake and hand back what the fake recorded, becomes a @contextmanager. It yields what it returned, and the caller enters it for the rest of the test:

## Before
def fake_gateway(monkeypatch, *, declines=False):
    charged = []

    def charge(self, amount):
        charged.append(amount)
        return not declines

    monkeypatch.setattr(Gateway, "charge", charge)
    return charged


def test_checkout_charges_once(monkeypatch):
    charged = fake_gateway(monkeypatch)
    checkout(create_order())
    assert charged == [500]


## Now
@contextmanager
def fake_gateway(*, declines=False):
    charged = []

    def charge(self, amount):
        charged.append(amount)
        return not declines

    with patch(Gateway, "charge", charge):
        yield charged


def test_checkout_charges_once():
    with fake_gateway() as charged:
        checkout(create_order())
    assert charged == [500]

A fixture that did the same is the same. How long the patch should last is a question the old code didn't ask: it lasts for the block, so put in the block everything that has to see the fake.

A parameter of the helper's that has the name of what the helper returns (def argv(argv)) needs one of the two renamed once there is no fixture between them.

11. Classes

A test is a function, and a file is the group. A class with test_* methods in it is a collection error, whatever it is named and whatever it inherits from.

## Before
class TestCheckout:
    def setup_method(self):
        self.user = create_user()

    def test_requires_login(self):
        assert Client().get("/checkout/").redirect_to == "/login/"

    def test_total(self):
        assert checkout_total(self.user) == 0


## Now
def test_checkout_requires_login():
    assert Client().get("/checkout/").redirect_to == "/login/"


def test_checkout_total():
    user = create_user()
    assert checkout_total(user) == 0

In a class	Now
def test_x(self):	def test_x(): at the top of the file
two classes with a test_create each	two names: test_invite_create, test_order_create. Or a file for each class
self.thing set in setup_method	a local: thing = make_thing()
a method the tests shared (self._post(...))	a function of the file
a class attribute the tests read	a module-level constant
@skip or @tag on the class	on each test
a base class of tests, run by each subclass	@cases, one case for what each subclass changed
a unittest.TestCase	functions. self.assertEqual(a, b) is assert a == b, assertRaises is raises
@staticmethod / @classmethod tests	functions
a target file.py::TestCart::test_add	file.py::test_add

A class that isn't a test and has a method named test_* (a fake backend with test_connection()) belongs in a helper module. A class that was imported into a test file is left alone.

12. The client

Import it from the same place: from plain.testing import Client. Everything after the path is keyword-only, and the keywords are the request's own names.

Before	Now
client.post(path, {...}), data={...}	form_data={...}
a file in data={...}	files={...}. form_data refuses one
data=json.dumps(x), content_type="application/json"	json_data=x: the value, not the string json.dumps made of it
data=x, content_type="application/json", json=x	json_data=x
raw bytes with a content type	body=b"...", content_type="..."
client.get(path, {...})	query_params={...}
follow=True	follow_redirects=True
response.content	response.body for bytes, response.text for a string
response.content.decode()	response.text
response.json(), json.loads(response.content)	response.json_data
response.url on a redirect	response.redirect_to
response.user	get_request_user(response.request) from plain.auth.requests
response.resolver_match	response.request.resolver_match
response.streaming	response.returned_response.streaming
any other attribute of the view's response	response.returned_response.<name>
RequestFactory().get("/x"), .post("/x", ...)	build_request("GET", "/x"), build_request("POST", "/x", ...)
RequestFactory().generic("PUT", "/x")	build_request("PUT", "/x")
RequestFactory().request(data=, query_string=)	the client's keywords: body=, form_data=, json_data=, query_params=
RequestFactory(headers=...), factory.cookies	headers= on each build_request(), with a Cookie header for cookies
RequestFactory(json_encoder=...)	gone. Encode it yourself and pass body=
secure=False	a full URL: client.get("http://testserver/x")
server_name=, server_port=	a full URL: client.get("https://example.com:8443/x")
Client(raise_request_exception=False)	Client(raise_exceptions=False)
client.trace(path)	client.request("TRACE", path), for any method
client.request(request) with a built Request	client.request(method, path, ...)
client.force_login(user)	login_client(client, user) from plain.auth.testing
client.logout()	logout_client(client) from plain.auth.testing
client.session	get_client_session(client) from plain.sessions.testing
client.cookies = SimpleCookie()	client.cookies.clear()
ws.response.request	ws.request

The client's response has a fixed set of names and passes nothing else through to the response the view returned: status_code, headers, cookies, body, text, json_data, redirect_to, redirect_chain, request, exception, and returned_response. WebSocketRejected.response is one of these too.

Only the test client's response changes. content on a Response your own code builds or a view returns is unchanged.

An old call is not explained by the client. A positional argument after the path, or a keyword it no longer takes, is Python's TypeError. A name the client or its response no longer has is an AttributeError, and the response's lists the names it has. The table above is where to look.

13. Spans, metrics, logs and queries

Each capture_* block hands back a read-only sequence, and it is read after the block ends.

Before	Now
otel_spans.get_finished_spans()	spans itself: len(spans), spans[0], for span in spans
[s for s in otel_spans.get_finished_spans() if s.name == "x"]	spans.filter(name="x"). It also takes kind=
otel_spans.get_finished_spans() == ()	list(spans) == []
otel_spans.clear() after the setup	open with capture_spans() as spans: after the setup
otel_metrics.get_metrics_data(), walked down to data points	metrics.number_points(name) or metrics.histogram_points(name)
filtering those points by an attribute	attributes={"key": "value"} on either
otel_metrics.collect()	nothing. Metrics are collected when the block ends
caplog.records, caplog.messages	logs itself, and logs.messages. It captures the plain and app loggers: name any other, capture_logs("stripe")
caplog.set_level(...), caplog.at_level(...)	capture_logs(level=logging.INFO). The loggers' levels are lowered for the block
reading any of them partway through the test	end the block first. Reading a capture inside its block raises
queries[0]["sql"] from capture_queries	queries[0].sql_with_params, or queries[0].sql for the statement with its %s in place
a hand-written execute_wrapper that records statements	queries.sql_statements() from capture_queries. It takes table=
a count of queries asserted by hand	with max_queries(n): from plain.postgres.testing, which fails on more than n. For an exact count, len(queries) from capture_queries
install_test_tracer(), install_test_meter()	gone. capture_spans() and capture_metrics() install what they need
from plain.test.otel import ...	from plain.testing import capture_spans, capture_metrics

14. Do the repeated parts with a script, and count

A suite of any size has the same rewrite hundreds of times. Done by hand, or by an agent editing file after file, the same thing comes out two ways in two files. The migrations this guide was written from stayed consistent because the repeated parts were scripted: in one, 661 db parameters, 97 .content reads, and the re-indentation of 191 tests under their new with blocks.

Write throwaway scripts, from the tables in this guide. Work on the syntax tree, not on text: a regular expression for .content misses .content == and rewrites a model field called content.

Script it	Read it, and decide
removing db and other parameters that become nothing	what each conftest.py fixture becomes, and whether it is setup or protection
a fixture parameter to the call or with it becomes, in order	which tests need a helper that a whole directory used to get
pytest.raises, parametrize with literal arguments, skip, marks	how long a helper's patch should last
monkeypatch calls to with blocks, and the re-indentation	a patch inside an if, a for, a try or a nested function
the client's keywords and its response's names	whether a .content or .json() is the test client's or something else's
force_login, RequestFactory, helper imports	base classes of tests, setup_method, anything session-scoped
classes to functions	names that collide once the class is gone

A script should leave alone what it has no rule for, and say so, a line for each thing it left. One that rewrites most of a file and leaves the rest looking finished is worse than none.

Build a fixture's dependencies the way pytest did: the fixtures it depends on first, each built once for the test and passed on. A test that asked for org and member, where member depends on org, builds org once.

Three counts, before and after, for every file

Every silent error in the migrations this guide was written from was caught by one of these, and by nothing else, since the tests passed.

1. Tests. The number of test functions in the file, and the number the run collects for it.
2. Assertions. Every assert and every raises block, compared as parsed code with the renames of this guide applied. The same number isn't enough: they should be the same assertions.
3. Comments and docstrings. Three scripted rewrites dropped comments with every test still passing: the comments on the values of a parametrize list folded into one @cases(...) line, and a fixture's docstring when the fixture was folded into a call. Count comment lines and docstrings, and compare their words.

Take the before counts of every file before editing any of them.

Where to keep the scripts, and these notes

plain check and plain code check lint, format-check and type-check every file in the project that git doesn't ignore, tracked or not. That includes a migration's own scripts, and the Python in the code blocks of a copy of this guide.

Keep both outside the project, or in a directory that .gitignore lists.

15. Things with no replacement

• pytest-xdist: tests run in one process. Several plain test runs can go at once in one checkout, each with a database of its own.
• pytest-randomly, pytest-rerunfailures: tests run in a fixed order, once.
• pytest-timeout: none.
• pytest-playwright and the testbrowser fixture: none. Browser tests can't be migrated yet. Keep them out of tests/, or under a name that isn't test_*.py, until they can.
• pytest-cov: coverage run -m plain.testing, then coverage report.
• freezegun, time-machine, hypothesis, responses, factory_boy: these are libraries, not plugins. They keep working, used as context managers or called in the test. Their decorators that pass an argument to the test work too, since the wrapper takes it.
• xfail, --pdb, --lf, --durations, -k expressions: none.
• Editor test explorers speak pytest's protocol and won't find these tests.
• Plugins of your own, and pytest_* hooks: none. There are no plugins. A package that has to protect every test registers a TestLifecycle under the plain.testing entry point group.

16. Behavior that changed without a name changing

Nothing reports these. A test that depended on one fails, or passes for another reason.

• What a test prints is held, at the file-descriptor level, and thrown away when it passes. A failing test's is printed with its failure.
• login_client writes the session once. A test that counted statements around a login counts fewer: a first login is two writes to the session table, where it was four.
• The tests that roll back share one database connection. Session-level state (a session advisory lock, a LISTEN) carries from one test to the next unless the test is @isolated_db.
• A redirect to another scheme is followed to that scheme's port. An http to https redirect used to be followed to https on port 80.
• patch puts back what the class itself held. An inherited attribute is inherited again, and a staticmethod is still one.
• Request(method="GET", path="/").body is b"". It used to raise AttributeError.
• A test with a yield in it used to be reported as passed with none of its body run. It is a collection error.
• DeprecationWarning and PendingDeprecationWarning are listed at the end of the run, once each.
• PYTEST_CURRENT_TEST is not set. PLAIN_TEST_RUNNING=1 is, for the whole run, and it is what Plain's own commands read to leave out color.
• .env.test is loaded by plain.dev, as before, and .env.local is left out of a test run.
• A failure report withholds what is known to be a secret: os.environ's values, a model's password or encrypted field. A secret held in a plain string is printed.

17. The last run

uv run plain test
uv run plain check

plain check runs the tests as one of its steps, and treats exit 4 (no tests) as nothing to do.

Then look for what is left over:

git grep -n -i -E "pytest|conftest" -- . ':!uv.lock'
git ls-files | grep -E "(^|/)(conftest|tests/.*__init__)\.py$"

18. What the release notes must say

Written by the release skill. This is what each package's notes have to carry, since plain-testing's own changelog starts empty and the runner explains none of it.

plain (core)

• plain.test is no longer part of Plain. What it held is plain.testing, a package of its own and a dev dependency: uv add plain.testing --dev. from plain.test import Client becomes from plain.testing import Client.
• plain request moved to plain.dev. The command, its flags and its output are the same, and it needs plain.dev installed.
• Request(body=...): a request can be constructed with a body. Request(...).body is b"" when none was given, where it raised.
• plain.server.inprocess: InProcessServer().handle(request).send() handles a request on the calling thread and returns what would be sent.
• plain check runs plain test and reads its exit codes: 4 is "no tests", where 5 was.
• plain docs <name> --api shows annotated attributes.
• plain agent install goes by what the files hold, not when they were written, and names each rule and skill it wrote or removed.
• Command output leaves out color when PLAIN_TEST_RUNNING is set, where it read PYTEST_CURRENT_TEST.
• plain.cli.runtime.set_running_command() / get_running_command(): code that runs during setup can ask which command is running.
• The rule file's examples use --fail-fast --match.
• Upgrade instructions: steps 1 to 3 above, and a pointer to the whole of this guide.

plain-testing (new, 0.1.0)

• Replaces plain.pytest. pytest is no longer supported, and there is no release where both run.
• Requires the core release that has plain.server.inprocess. The lower bound on plain has to be raised to it.
• Its upgrade instructions are this guide, steps 1 to 17.

plain-pytest

• Deleted. Its last release's notes, or the package's page, say plain.testing replaces it.

plain-dev

• plain request lives here now.
• With no database URL configured, each test run gets a database of its own, so any number of runs can go at once in one checkout. No suffix on PLAIN_POSTGRES_URL is needed to keep them apart.
• plain db clean --dry-run lists every database of the project with the reason it would be dropped or is left. plain db clean asks before it drops and has no flag that skips the question.
• A test run doesn't print which .env files were loaded.
• Dev services are started once the command is known to exist. A mistyped command, or plain test with no plain.testing installed, starts none.

plain-postgres

• The helpers moved from plain.postgres.test to plain.postgres.testing, to match the package they serve. plain.postgres.test.pytest is gone, and with it the db and isolated_db fixtures. Every test runs in a rolled-back transaction with nothing asked for, and @isolated_db is a decorator from plain.postgres.testing.
• capture_queries hands back a read-only sequence read after the block. Each query has .sql as sent and .sql_with_params, and queries.sql_statements(table=) lists statements. queries[0]["sql"] is queries[0].sql_with_params.
• max_queries and capture_queries nest.
• The tests that roll back share one connection.
• A field type says its value is a secret with value_is_secret = True. Encrypted fields do.
• A model instance and a queryset in a failure report are printed with their fields, and no query is run to do it.
• Test databases are made per run and dropped at the end, each a clone of a template (test_<database>_t<schema>) that the first run builds and leaves for the next. One template stays per checkout's database; a changed migration or model is a new one, and the old one goes on the next run. One left by a killed run, and a template of no use, is found by plain db clean.

plain-email

• The helpers moved from plain.email.test to plain.email.testing. plain.email.test.pytest and the mailoutbox fixture are gone. from plain.email.testing import outbox. During a test run the backend is the in-memory one and the outbox is cleared around each test, with nothing asked for.

plain-auth

• plain.auth.test is plain.auth.testing. login_client(client, user) and logout_client(client) there replace client.force_login() and client.logout().
• login_client writes the session once. logout_client on a client that has no session writes nothing.

plain-sessions

• plain.sessions.test is plain.sessions.testing. get_client_session(client) there replaces client.session.

plain-connect

• Nothing is exported during a test run, whatever the settings say. PLAIN_CONNECT_EXPORT_ENABLED=false for tests can be removed.

plain-passwords

• PasswordField has value_is_secret = True: a failed test's report prints a user without the hash.

plain-code

• plain code check passes tests/ to the type checker as a search path, so a helper imported by its path from tests/ resolves.
• The PT lint rules (pytest style) are no longer in the default set.

Every other package

• Dev dependency changed from plain.pytest to plain.testing. Nothing for a user to do.

19. What /plain-upgrade has to do

The skill is what carries an app across, since the runner doesn't. In order:

1. Make the edits of step 1 to pyproject.toml, and uv sync.
2. plain agent install, and read what it wrote.
3. Rewrite flags in scripts, CI files and docs from the table in step 3.
4. Read every conftest.py for what guarded every test, and write tests/lifecycle.py (step 4). Before anything is run.
5. Take the three counts of step 14 for every test file.
6. Run plain test once for the list of what is wrong, and the total of the parameters tests take (step 5).
7. Delete every __init__.py under tests/, moving code out of any that has some (step 6).
8. For each conftest.py, sort the rest of what it holds by the table in step 7, write the helper modules, and delete it.
9. Write the scripts of step 14 and run them. For each test file: helper imports (step 8), what it used from pytest (step 9), each fixture parameter as what it becomes in the body (step 10), classes as functions (step 11), client calls and response names (step 12), and captures (step 13).
10. Run plain test. Every collection error names its file and says what is wrong. Fix and repeat until there are none, then fix failures.
11. Take the three counts again and compare.
12. Run plain check (step 17).
13. Report what has no replacement (step 15) as left for the person: browser tests, xfail, plugins of their own.

[pullapprove5]

FAIL: Large-scale change: configuration required (no reviewers defined)

Large Scale Change

• Status: FAIL
• Points: 0/1

No review scopes matched.

View in PullApprove