Skip to content

test(testkit): run every testkit suite as one program, and let nextest run the prebuilt program #5765

Open

Description

@matthewgrossman

User Story

I maintain OpenShell's end-to-end tests and CI. #5755 made openshell-conformance the conformance test target, so openshell-conformance run, cargo test and cargo nextest run all run the same program, and added OPENSHELL_TEST_UNSUPPORTED. I need the same for every testkit suite, and nextest's scheduling for the prebuilt program in CI, so each feature of how tests run (timeouts, test groups, unsupported entries) is built once and behaves the same however the tests are started.

Description

Two steps remain after #5755.

1. Let nextest run the prebuilt program

  • Add openshell-conformance nextest-metadata <dir>. It writes a stub Cargo.toml and the two small JSON files nextest reads (--binaries-metadata, --cargo-metadata), naming the program itself as the only test binary.
  • With those, cargo nextest run --binaries-metadata ... --cargo-metadata ... runs the prebuilt program with no cargo, Rust toolchain or source checkout. A spike on 11-12 did this in about 100 lines.
  • Runner rows that use --program then run nextest on the program instead of openshell-conformance run. They get nextest's per-test timeouts, test groups and test(runner): apply per-family nextest config and a thread count to runner rows #5295's per-family settings, which today reach only nextest rows.
  • openshell-conformance run stays as the way to run tests with nothing but the binary: host lanes, outside drivers, local runs.
  • The runner already generates a nextest profile and maps nextest's exit codes for features rows.

2. Features and drivers suites become programs

Smaller follow-ups from #5755

  • CI reads entries from tests/suites/conformance/unsupported.yaml, and Branch Checks validates them per environment. But a program ignores entries for crates it doesn't link, so an entry with a misspelled crate (conformence::...) passes the check and is silently ignored. Reject entries whose crate isn't conformance in check-conformance-program.sh.
  • When features and drivers become programs, decide whether each family gets its own unsupported.yaml or the file moves up a level.

Context

Why one program. Before #5755, #[testkit::test] produced a registry entry, which openshell-conformance run used, plus a #[tokio::test] wrapper, which cargo test and nextest ran. Every run-time feature had to be built twice or worked on one path only:

Constraints for how tests run:

  • use nextest's parallel runs, timeouts and test groups;
  • stay compatible with cargo test and nextest;
  • outside users get one binary and never unpack anything;
  • a fairly minimal environment;
  • no bare nextest archive for conformance.

Costs:

  • Editor "Run test" buttons on each test go away, because they come from #[tokio::test]. cargo test <name> still works.
  • nextest binary IDs change, for example to conformance::bin/openshell-conformance.

Related:

Part of #5281.

Options considered, spike results and deferred ideas

Run-path options

Option What it is Verdict
1 Keep both paths; each feature picks one Where we were; not an end state
2 A program per family; drop nextest from CI Out: gives up nextest
3 nextest everywhere; the program only for outside users Out: CI wouldn't test what users get
4 Run-time features inside each test process A technique, used for OPENSHELL_TEST_UNSUPPORTED
5 The program is the test target; run it directly, or point nextest at it Chosen. #5755 plus this issue
6 Embed the nextest archive in the binary Out: unpacks 51 MB in 376 files, ~0.65 s, on every run
7 Embed cargo-nextest; the program runs nextest on itself Deferred (see below)

Also ruled out:

  • a bare nextest archive for outside users;
  • linking nextest's library, nextest-runner, which is an unstable 0.x crate.

Spike (11-12)

macOS arm64, nextest 0.9.143, a toy suite on the real testkit::program.

Check nextest in a checkout, program as test target nextest on the program from hand-made metadata run
Works Yes Yes, with no cargo or Rust tools installed Yes
A hung test killed at its timeout Yes Yes Yes
Parallel runs Yes Yes No (serial)
Skip shows SKIP Yes Yes Yes
Unknown name Error at listing Same Same

Not covered: Linux, passing signals through to nextest, and whether nextest version bumps change the metadata format.

On 11-18, harness = false on the real conformance [[bin]] listed all 49 tests under cargo nextest list and cargo test -- --list.

Deferred, not planned for now

  • Expected failure (OPENSHELL_TEST_EXPECTED_FAILURE): a failure passes, and an unexpected pass fails. Unsupported tests usually hang rather than failing fast, and a skip costs no run time and can't leave stuck sandboxes behind, so skips come first. Add this only for a known bug that fails quickly every time. A timeout would stay red, and nextest would show PASS, since it has no expected-failure label.
  • Embedded nextest (option 7): run would use a pinned cargo-nextest underneath, and testkit's own runner (~800 lines) could go. Costs: about 20 MB more binary (roughly 35 MB on Linux), a cache directory that allows running programs, and keeping nextest's version current. First run 0.9 s, later runs 0.07 s. Worth it only if direct runs need parallelism, or we want a single scheduler.
  • A run --unsupported shortcut for the environment variable. It would be safe, because libtest rejects flags it doesn't know, but one way to give the list is enough for now.
  • Flagging entries whose issue has closed.

Open questions

  • One program per family, or per package? Rows pick a package today. Crate-prefixed entries already keep packages from clashing.
  • Does conformance need parallel runs before milestone 1 (deleting e2e/rust) ends? That needs test(runner): apply per-family nextest config and a thread count to runner rows #5295's thread counts, a check that tests don't collide, and maybe larger VMs. The runner pins --test-threads 1 today because tests share one gateway and the VM has 4 GiB.
  • Do features and drivers need unsupported entries before step 2?

Definition of Done

  • A prebuilt testkit program can write the metadata nextest needs, and cargo nextest run runs it with no cargo or source checkout.
  • Runner rows with --program run nextest on the program, with nextest's per-test timeouts and the family's nextest settings.
  • Every features and drivers suite is a program that run, cargo test and nextest all run.
  • #[testkit::test] no longer generates #[tokio::test] wrappers.
  • An unsupported.yaml entry with an unknown crate fails Branch Checks.
  • The testkit README describes how to run a suite each way.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions