[matthewgrossman]

Summary

Ports the universal conformance suites (CLI, lifecycle, file-transfer, persistent-workspace) onto testkit and runs them from the cpu-tests plan on Docker, Podman (rootful and rootless) and k3s (single and HA). This is PR 2 of the plan in conformance-plan.md, stacked on the runner.

It adds the new suites without removing anything: the old suites, the hand-written release lanes (conformance-integration) and the old crates are untouched. Switching conformance-integration to the runner, and deleting the old CLI suite, is PR 3, after each environment has been shown green here a few times.

The lifecycle, persistent-workspace and file-transfer ports start from @elezar's draft #4629.

Stack

Part of a stack; each PR targets the one below it. The tmachine move (#4726) and the testkit library (#4724) have merged.

1. feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773: the test runner, plans, tmachine Session API and a smoke suite running on CI.
2. test(suites): port the universal conformance suites onto testkit and run them from the plan #4774: ports the universal conformance suites and runs them in the plan on five environments.

Still to come: (3) ship openshell-conformance as one program with testkit::main(), point the host lanes at it, delete the old CLI suite and switch conformance-integration to the runner; (4) migrate features and drivers (oci-image, Keycloak providers, Podman userns, e2e-podman) and delete crates/openshell-conformance.

Related Issue

Part of #3954

Changes

• tests/suites/conformance/sandbox: test files lifecycle, cli, persistent_workspace and file_transfer next to smoke. persistent_workspace_survives_stop_start stays in conformance: every conformant driver keeps the workspace across stop and start (MXC isn't conformant until it can restart a stopped sandbox).
• cx.track and cx.adopt in testkit, for sandboxes a test creates through the CLI.
• tests/environments: adds Podman (Fedora 44, rootful and rootless, RPM) and k3s and HA k3s (Ubuntu 24.04), amd64 only.
• tests/plans/cpu-tests.yaml: the whole conformance family on all five environments.
• Plan validation: a conformance entry must select the same suites in every environment, or the runner refuses the plan and names the environments that differ.
• The k3s installer now waits for the Agent Sandbox controller to reconcile (k3s-controller-ready.yaml, moved out of the old conformance playbook, which includes it too). Tests don't wait for it.
• e2e-suites.yml: the tmachine disk cache is keyed per environment. The k3s environments skip the cache, as integration-runner.yml does; its history doesn't say why, so that is still to find out.
• Git is installed in every tmachine test environment (test-tools.yaml), for the .gitignore filtering test.

Old test to new test

Old	New	Notes
smoke (scenario: status, create, get Ready, list, exec, delete, list empty)	sandbox/tests/cli.rs::cli_create_exec_list_delete	Same CLI steps. smoke.rs::create_exec_delete is the SDK variant (in #4773).
sandbox_lifecycle stop_start_preserves_workspace	persistent_workspace.rs::persistent_workspace_survives_stop_start (SDK) and cli.rs::cli_stop_start_delete (CLI)	Also lifecycle.rs::restart_stopped_sandbox.
sandbox_lifecycle stopped_can_be_deleted	lifecycle.rs::stopped_sandbox_can_be_deleted	Also at the end of cli_stop_start_delete.
file_transfer::round_trip	file_transfer.rs::upload_download_round_trip
file_transfer::git_filtering	file_transfer.rs::upload_respects_gitignore	Needs git where the tests run.
file_transfer::path_safety	file_transfer.rs::download_stays_inside_workspace

New with no old counterpart (from #4629): lifecycle.rs::long_running_exec_delete, run_to_completion_exit_delete, default_command_sandbox_is_listed_ready.

policy_advisor has no port: it is a filter (binary(policy_advisor)) over existing conformance tests that no workflow runs, so it goes with the old suite.

Not in this PR

• Switching conformance-integration: PR 3 per the plan, after each environment is shown green several times via workflow_dispatch (fallback: switch Docker first).
• oci-image, Keycloak providers, Podman userns, e2e-podman: PR 4, in features and drivers. Earlier drafts of this PR ported them; they were removed because the runner has no way yet for a VM to hand a suite extra inputs (the plan's decision 8).
• Find out why integration-runner.yml skips caching k3s before caching it here.
• arm64: not defined; needs an arm64 KVM runner and arm64 build inputs.
• The six open PRs that extend the old suite (test(conformance): move scenario files into Cargo tests #4120, test(conformance): organize stories into area modules #4536, test(conformance): migrate create-upload tests to file_transfer stories #4606, test(conformance): migrate workspace lifecycle tests to workspace stories #4607, test(conformance): migrate sandbox template tests to template stories #4608, test(conformance): migrate lifecycle main-result tests to lifecycle stories #4628): their tests will be ported later, after they land or are reworked.

Testing

• Checks appropriate to the affected code and behavior pass
  • Locally: fmt, clippy, the runner tests (48), testkit and lockfile alignment checks, and the plan dry run listing the suites in all five environments.
  • On CI: a Branch E2E workflow_dispatch on this head is running; results will be posted here. The ports passed on Docker on an earlier head (21 tests); Podman and k3s have not run through the runner yet.
• Unit tests added/updated
• E2E tests added/updated

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (TESTING.md, tests/plans/README.md)

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4774.docs.buildwithfern.com/openshell

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.