Skip to content

feat(runner): add the test runner, plans and tmachine Session API - #4775

Closed
matthewgrossman wants to merge 10 commits into
test/3954-migrate-suites-to-testkit/matthewgrossmanfrom
test/3954-test-runner-and-plans/matthewgrossman
Closed

feat(runner): add the test runner, plans and tmachine Session API#4775
matthewgrossman wants to merge 10 commits into
test/3954-migrate-suites-to-testkit/matthewgrossmanfrom
test/3954-test-runner-and-plans/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Oct 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Adds the test runner, plans, and tmachine Session API on top of testkit. This is step 5 of the test harness plan, stacked on the suite migration.

Most of the code comes from @elezar's draft #4629: the Session API, testkit-spec, and the runner, plans, environments, and waivers. It is adapted to testkit finding the gateway the way the CLI does, so the runner sets OPENSHELL_GATEWAY and OPENSHELL_BIN instead of writing a target file.

Stack

Part of a stack of five; each PR targets the one below it. The tmachine move (#4726) has merged, so the stack uses its tests/tmachine/ paths.

  1. test(testkit): add the testkit library for end-to-end suites #4724: the testkit library.
  2. test(sandbox): run a testkit suite in Branch E2E #4822: the first testkit suite, run in Branch E2E against a Docker gateway. The simplest route to testkit tests in CI.
  3. feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773: the test runner, plans, and tmachine Session API, with a plan running on CI.
  4. test(suites): port the universal conformance suites onto testkit and run them from the plan #4774: ports the old suites onto testkit and adds them to the plan.
  5. feat(runner): add the test runner, plans and tmachine Session API #4775: moves the release lanes onto plans and deletes the old suites.

Related Issue

Part of #3954

Changes

  • testkit: optional #[testkit::test(id = "t-…", requires = [...])], a testkit-spec crate that the macro and the runner share, and capabilities that reach a test through OPENSHELL_TEST_CAPABILITIES. Existing tests don't change. The macro rejects #[should_panic] and moves other attributes such as #[ignore] to the generated test, as test(testkit): add the testkit library for end-to-end suites #4724 now does; the requirement tests call testkit::run directly instead of using should_panic.
  • tests/runner (openshell-test): plan resolution, environments in tests/environments, waivers, JUnit result checking (a missing, skipped, or unselected test fails the lane), a strict CI mode, and --emit-github-matrix. The catalog leaves #[ignore] tests out of what a lane expects, and the JUnit check accepts nextest's skipped case for a test it never expected.
  • tests/plans/testkit.yaml plans the suites in the environments the conformance and feature lanes run today. tests/expected-matrix.json is checked in, so a change to what runs where shows up as a diff. tasks/scripts/check_plan_workflow_matrix.py fails when the release and Branch E2E workflow matrices drift from the plan, or when a testsuite's suite_expected_tests in tests/tmachine/config.nix differs from the number of tests the plan expects.
  • tmachine: the Session API; SSH, scp, and exec timeouts; down() never panics; QEMU is cleaned up on SIGTERM; the SSH port is TMACHINE_SSH_PORT.
  • fmt, clippy, and nextest for tests/runner and tests/tmachine in Branch Checks and the mise tasks.

persistent_workspace_survives_stop_start doesn't declare requires yet: the tmachine playbooks and mise run e2e:testkit don't pass OPENSHELL_TEST_CAPABILITIES, so a test with requires would fail there. Add it when the runner drives those lanes.

Not in this PR: generating the release workflow matrices from --emit-github-matrix, which needs a reusable workflow like the one in #4629. The drift check covers it meanwhile.

Testing

  • Checks appropriate to the affected code and behavior pass
    • fmt, clippy, and nextest pass for the runner (58 tests), testkit (23), and tmachine (8). The plan dry run, the matrix check, the drift check, and the lockfile and alignment checks pass. The sandbox suites still pass against a local Docker gateway (14/14), and the flake outputs evaluate.
    • Not run: nothing here booted a VM, the Nix build of the runner, or Branch Checks in CI.
  • Unit tests added/updated
  • E2E tests added/updated (n/a)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (TESTING.md, AGENTS.md, CONTRIBUTING.md)

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Oct 20, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@matthewgrossman
matthewgrossman added this pull request to stack #4776 October 20, 2026 23:38
@github-actions

github-actions Bot commented Oct 20, 2026

Copy link
Copy Markdown

🌿 Preview your docs: https://nvidia-preview-pr-4775.docs.buildwithfern.com/openshell

@copy-pr-bot

copy-pr-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@matthewgrossman
matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 56ebe92 to 6d64703 Compare October 21, 2026 20:38
@matthewgrossman
matthewgrossman force-pushed the test/3954-test-runner-and-plans/matthewgrossman branch from 2b28380 to 34caea4 Compare October 21, 2026 20:38
@matthewgrossman
matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 6d64703 to e3d9f01 Compare October 21, 2026 20:40
@matthewgrossman
matthewgrossman force-pushed the test/3954-test-runner-and-plans/matthewgrossman branch from 34caea4 to 03153d1 Compare October 21, 2026 20:40
@matthewgrossman
matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from e3d9f01 to 5e2cbc5 Compare October 21, 2026 20:49
@matthewgrossman
matthewgrossman force-pushed the test/3954-test-runner-and-plans/matthewgrossman branch from 03153d1 to fc07e55 Compare October 21, 2026 20:49
This was referenced Oct 21, 2026
Merged
Open
Draft
@matthewgrossman
matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 5e2cbc5 to cda205f Compare October 21, 2026 23:45
@matthewgrossman
matthewgrossman force-pushed the test/3954-test-runner-and-plans/matthewgrossman branch from fc07e55 to 8e6ac37 Compare October 21, 2026 23:45
matthewgrossman and others added 10 commits October 21, 2026 21:48
@matthewgrossman @elezar
build(cargo): add a check that aligned workspaces lock shared crates …
a7869ff 
…like the root

`cargo --locked` only checks a lockfile against its own manifest. A workspace
outside the root that builds root crates through path dependencies, such as the
testkit suites building `openshell-sdk`, has its own Cargo.lock, so the two
lockfiles can resolve a shared crate to different versions without any check
failing. That is how aws-lc-sys 0.46 was picked up in #4629, which does not link
against the AWS-LC that Nix provides.

check_cargo_lockfile_alignment.py fails when an aligned workspace locks a
registry crate that the root lockfile also contains at a version the root
lockfile does not have. With --fix it re-locks the workspace starting from the
root lockfile.

The script and its tests come from #4629 (a37ebe9). The aligned workspaces are
tests/testkit and tests/suites/sandbox instead of the old conformance suites.
The check is wired into the lockfile task and CI in the next commit.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman @elezar
test(sandbox): add the first testkit suite and run it in Branch E2E
38670c7 
Give new end-to-end tests a home and a CI lane. New tests go in testkit; the old
suites take fixes only.

- tests/suites/sandbox is a testkit suite workspace. It starts with the
  long_running_exec_delete test, adapted from the sandbox-lifecycle suite in
  #4629. Its README holds the guidance on what a test may assert.
- The suite builds into a Nix test archive with `--locked`, and a tmachine
  testsuite and playbook run it against the gateway the installer registered.
  testkit finds that gateway the way the CLI does, so no target file is needed.
  The playbook waits for the gateway first, so an unavailable gateway is one
  environment failure rather than N test failures. It runs nextest serially
  with a slow-timeout that terminates hung tests, and fails the lane when fewer
  tests run than expected.
- Branch E2E runs it in a `testkit-integration` job on Ubuntu with Docker and
  the Debian package. It isn't part of the Core E2E result yet.
- The alignment check runs in `rust:lockfiles:check` and the Cargo lockfiles CI
  job, with its tests in `mise run test`, and enrolls tests/testkit and the new
  suite. A PR that bumps a shared crate in the root Cargo.lock now needs
  `uv run tasks/scripts/check_cargo_lockfile_alignment.py --fix`.
- `mise run e2e:testkit` (and `e2e:testkit:podman`) run the suite against an
  ephemeral local gateway.
- fmt and clippy cover the new workspace in Branch Checks and the mise tasks.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
ci(tests): key the tmachine disk cache on the playbooks and keep JUni…
f44d1ed 
…t results

tmachine keys its disk layers on a hash of the Ansible tree, and actions/cache
does not save after an exact key hit, so the fixed cache key rebuilt the changed
layers on every run. Key the cache on the playbooks and fall back to the newest
entry for the same runner.

The sandbox-lifecycle playbook now has nextest write a JUnit report and fetches
it into artifacts/tmachine-diagnostics, which the integration runner already
uploads whether or not the lane passes.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
fix(tests): use the tmachine paths for the first testkit lane
7855b5d 
The tmachine move landed on main, so the sandbox-lifecycle playbook belongs
under tests/tmachine/ansible. Move it there, point the test bundle input and the
JUnit fetch at the new locations, and key the disk cache on the new playbook
path.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
build(cargo): realign the testkit and sandbox lockfiles with the root…
63f4141 
… lock

Main bumped shared crates in the root Cargo.lock after this branch was last
aligned. Re-lock tests/testkit and tests/suites/sandbox from it with
check_cargo_lockfile_alignment.py --fix.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman @elezar
test(suites): move the conformance, oci-image, provider-refresh and p…
dec35b8 
…odman suites to testkit

Cut the release lanes over to suites written on testkit and delete the old
conformance, features and drivers suites. See the PR description for the old
test to new test mapping.

The sandbox-lifecycle, file-transfer and policy-advisor ports start from #4629
by Evan Lezar.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
docs(tests): fix stale suite references, lint every testkit suite, an…
66f1b04 
…d keep JUnit results

- testkit README: describe `track`, `track_sandbox`, `adopt` and `delete_sandboxes` once.
- TESTING.md: the local suite command excludes oci-image, which needs a container
  engine, and names the sandbox suites instead of the deleted conformance ones.
- Run fmt and clippy for tests/suites/providers and tests/suites/podman in the
  mise tasks and Branch Checks.
- The generic suites playbook has nextest write a JUnit report and fetches it
  into artifacts/tmachine-diagnostics, which the integration runner uploads.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
build(cargo): realign the suite lockfiles with the root lock
7c75fbf 
Re-lock tests/suites/{sandbox,providers,podman} from the current root
Cargo.lock with check_cargo_lockfile_alignment.py --fix.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman @elezar
feat(runner): add the test runner, plans and tmachine Session API
ff22114 
Add the test runner (openshell-test), plans, environments and waivers, the
tmachine Session API, optional test IDs and `requires` on testkit with a shared
testkit-spec crate, and fmt, clippy and nextest for the new crates.

Most of the code comes from #4629 by Evan Lezar, adapted to testkit finding the
gateway the way the CLI does: the runner sets OPENSHELL_GATEWAY and OPENSHELL_BIN
instead of writing a target file.

testkit rejects `#[should_panic]` and moves other attributes such as `#[ignore]`
to the generated test. The runner's catalog leaves an ignored test out of what a
lane expects, and the JUnit check accepts nextest's skipped case for it.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
ci(tests): check lane test counts and Branch E2E against the plan
2353b2f 
The drift check now also covers Branch E2E, and compares each testsuite's
`suite_expected_tests` in tests/tmachine/config.nix with the number of tests the
plan expects, using tests/expected-matrix.json. Raising a count in one place
without the other fails the check.

Document why persistent_workspace_survives_stop_start doesn't declare
`requires`: the tmachine playbooks don't pass capabilities yet.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from cda205f to 7c75fbf Compare October 22, 2026 04:51
@matthewgrossman
matthewgrossman force-pushed the test/3954-test-runner-and-plans/matthewgrossman branch from 8e6ac37 to 2353b2f Compare October 22, 2026 04:51
@matthewgrossman
matthewgrossman removed this pull request from stack #4776 October 22, 2026 04:51
@matthewgrossman
matthewgrossman added this pull request to stack #4824 October 22, 2026 04:51
@matthewgrossman matthewgrossman mentioned this pull request Oct 22, 2026
Closed
5 tasks done
@matthewgrossman
matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 7c75fbf to bb39723 Compare October 22, 2026 05:29
@matthewgrossman

matthewgrossman commented Oct 22, 2026

Copy link
Copy Markdown
Member Author

Superseded by the reordered stack. The runner, plans, and tmachine Session API this PR added now live in #4773, below the migration in #4774. The release-lane cutover and old-suite deletion will come as a new PR on top of #4774 once the plan has run the ported suites green.

@matthewgrossman
matthewgrossman removed this pull request from stack #4824 October 22, 2026 05:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrunalp mrunalp Awaiting requested review from mrunalp mrunalp will be requested when the pull request is marked ready for review mrunalp is a code owner
@sjenning sjenning Awaiting requested review from sjenning sjenning will be requested when the pull request is marked ready for review sjenning is a code owner
@derekwaynecarr derekwaynecarr Awaiting requested review from derekwaynecarr derekwaynecarr will be requested when the pull request is marked ready for review derekwaynecarr is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@matthewgrossman