test(suites): port the universal conformance suites onto testkit and run them from the plan

.github/workflows/e2e-suites.yml

@@ -94,17 +94,20 @@ jobs:
 
       - uses: ./.github/actions/setup-nix
 
+      # integration-runner.yml skips the disk cache for ubuntu-k3s and doesn't say
+      # why, so the k3s environments skip it here too until that is understood.
       - name: Cache tmachine disks
+        if: ${{ !contains(matrix.environment, '.k3s.') }}
         uses: actions/cache@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
         with:
           path: ~/.cache/tmachine
           # tmachine keys its disk layers on a hash of the Ansible tree.
           # actions/cache doesn't save after an exact key hit, so a fixed key
           # would rebuild the changed layers on every run. Key on the playbooks
-          # and fall back to the newest entry for the same runner.
-          key: tmachine-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('tests/tmachine/ansible/**', 'tests/tmachine/images.nix') }}
+          # and fall back to the newest entry for the same environment and runner.
+          key: tmachine-${{ matrix.environment }}-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('tests/tmachine/ansible/**', 'tests/tmachine/images.nix') }}
           restore-keys: |
-            tmachine-${{ runner.os }}-${{ runner.arch }}-
+            tmachine-${{ matrix.environment }}-${{ runner.os }}-${{ runner.arch }}-
 
       - name: Download integration inputs
         uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1

TESTING.md

@@ -174,7 +174,9 @@ what runs where: CI reads it, so no workflow or playbook repeats it.
 
 - **Plans** (`tests/plans/*.yaml`) pair a family (a workspace under
   `tests/suites/`, such as `conformance`) and optionally some of its suites with
-  environment IDs. See [tests/plans/README.md](tests/plans/README.md).
+  environment IDs. See [tests/plans/README.md](tests/plans/README.md). A `conformance` entry must
+  select the same suites in every environment; the runner refuses a plan that
+  doesn't.
 - **Environments** (`tests/environments/*.yaml`) name the tmachine environment and
   installer that provision them and the gateway name the installer registers. The
   runner points the tests at that gateway and the installed CLI with

tests/environments/k3s.yaml

@@ -0,0 +1,21 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Kubernetes (k3s) environments, by Evan Lezar (NVIDIA/OpenShell#4629). See
+# docker.yaml for the fields and for adding arm64. The k3s installers register the gateway as
+# `tmachine`.
+environments:
+  - id: tmachine.ubuntu2404-amd64.k3s.k3s
+    backend: tmachine
+    machine: { os: ubuntu, version: "24.04", arch: amd64 }
+    runtime: k3s
+    installer: k3s
+    tmachine: { environment: ubuntu-k3s, installer: k3s }
+    gateway: tmachine
+  - id: tmachine.ubuntu2404-amd64.k3s.k3s-ha
+    backend: tmachine
+    machine: { os: ubuntu, version: "24.04", arch: amd64 }
+    runtime: k3s
+    installer: k3s-ha
+    tmachine: { environment: ubuntu-k3s, installer: k3s-ha }
+    gateway: tmachine

tests/environments/podman.yaml

@@ -0,0 +1,20 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Podman-backed environments, by Evan Lezar (NVIDIA/OpenShell#4629). See
+# docker.yaml for the fields and for adding arm64.
+environments:
+  - id: tmachine.fedora44-amd64.podman-rootful.rpm
+    backend: tmachine
+    machine: { os: fedora, version: "44", arch: amd64 }
+    runtime: podman-rootful
+    installer: rpm
+    tmachine: { environment: fedora-podman-rootful, installer: rpm }
+    gateway: openshell
+  - id: tmachine.fedora44-amd64.podman-rootless.rpm
+    backend: tmachine
+    machine: { os: fedora, version: "44", arch: amd64 }
+    runtime: podman-rootless
+    installer: rpm
+    tmachine: { environment: fedora-podman-rootless, installer: rpm }
+    gateway: openshell

tests/plans/README.md

@@ -12,14 +12,18 @@ matrix:
       - tmachine.ubuntu2404-amd64.docker-rootful.deb   # IDs from tests/environments/
 ```
 
+A `conformance` entry must select the same suites in every environment: the
+runner refuses a plan that doesn't, so conformance means the same thing
+everywhere. A test that can't pass everywhere belongs in another family.
+
 A family is a Cargo workspace directly under `tests/suites/` with at least one
 testkit suite. A suite is a test file in a crate's `tests/` directory containing
 `#[testkit::test]` functions. Its name is the file name without `.rs`, with `_`
 replaced by `-`: `tests/file_transfer.rs` is the suite `file-transfer`.
 
 | Plan | Runs |
 |---|---|
-| `cpu-tests.yaml` | The `conformance` family on Ubuntu 24.04 amd64 with rootful Docker and the Debian package |
+| `cpu-tests.yaml` | The `conformance` family on rootful Docker (Debian package), rootful and rootless Podman (RPM), and k3s and HA k3s, all amd64 |
 
 ## How a plan runs on CI
 

tests/plans/cpu-tests.yaml

@@ -16,3 +16,7 @@ matrix:
   - type: conformance
     environments:
       - tmachine.ubuntu2404-amd64.docker-rootful.deb
+      - tmachine.fedora44-amd64.podman-rootful.rpm
+      - tmachine.fedora44-amd64.podman-rootless.rpm
+      - tmachine.ubuntu2404-amd64.k3s.k3s
+      - tmachine.ubuntu2404-amd64.k3s.k3s-ha

tests/runner/src/jobs.rs

@@ -91,6 +91,8 @@ pub fn expand(
         }
     }
 
+    check_conformance_is_uniform(&selected)?;
+
     for id in &narrowing.environments {
         if !selected.contains_key(id) {
             bail!("--env {id:?} is not in the plan");
@@ -139,6 +141,44 @@ pub fn expand(
     Ok(jobs)
 }
 
+/// The family every driver and install must pass. A plan can't run part of it
+/// somewhere, or conformance would mean different things in different places.
+const CONFORMANCE: &str = "conformance";
+
+/// Every environment the plan runs conformance in must select the same suites.
+fn check_conformance_is_uniform(
+    selected: &BTreeMap<String, BTreeMap<String, Vec<String>>>,
+) -> Result<()> {
+    let mut by_suites: BTreeMap<BTreeSet<&str>, Vec<&str>> = BTreeMap::new();
+    for (id, families) in selected {
+        if let Some(suites) = families.get(CONFORMANCE) {
+            by_suites
+                .entry(suites.iter().map(String::as_str).collect())
+                .or_default()
+                .push(id);
+        }
+    }
+    if by_suites.len() > 1 {
+        let groups: Vec<String> = by_suites
+            .iter()
+            .map(|(suites, ids)| {
+                format!(
+                    "{} select {}",
+                    ids.join(", "),
+                    suites.iter().copied().collect::<Vec<_>>().join(", ")
+                )
+            })
+            .collect();
+        bail!(
+            "the plan selects different {CONFORMANCE} suites in different environments, so \
+             {CONFORMANCE} would not mean the same thing everywhere; use the same suites \
+             everywhere, or move the suite that can't run everywhere to another family:\n  {}",
+            groups.join("\n  ")
+        );
+    }
+    Ok(())
+}
+
 /// What strict mode (CI) refuses to run: nothing selected at all, an
 /// environment this host can't provision, and a family with no tests. Locally these are reported and skipped, because a
 /// laptop can't boot every environment; in CI each one means a lane that would
@@ -339,7 +379,7 @@ mod tests {
     fn groups_entries_by_environment() {
         let jobs = expand_plan(
             &format!(
-                "matrix:\n  - type: conformance\n    suites: [providers]\n    environments: [{AMD64}, {ARM64}]\n  - type: conformance\n    suites: [providers, sandbox-lifecycle]\n    environments: [{AMD64}]\n"
+                "matrix:\n  - type: conformance\n    suites: [providers]\n    environments: [{AMD64}, {ARM64}]\n  - type: conformance\n    suites: [sandbox-lifecycle]\n    environments: [{AMD64}, {ARM64}]\n"
             ),
             &Narrowing::default(),
         )
@@ -421,6 +461,35 @@ mod tests {
         );
     }
 
+    #[test]
+    fn conformance_must_select_the_same_suites_in_every_environment() {
+        let err = expand_plan(
+            &format!(
+                "matrix:\n  - type: conformance\n    environments: [{AMD64}]\n  - type: conformance\n    suites: [providers]\n    environments: [{ARM64}]\n"
+            ),
+            &Narrowing::default(),
+        )
+        .unwrap_err();
+        let message = err.to_string();
+        assert!(
+            message.contains("different conformance suites"),
+            "{message}"
+        );
+        assert!(
+            message.contains(AMD64) && message.contains(ARM64),
+            "{message}"
+        );
+
+        // The same suites listed differently are the same selection.
+        expand_plan(
+            &format!(
+                "matrix:\n  - type: conformance\n    suites: [providers, sandbox-lifecycle]\n    environments: [{AMD64}]\n  - type: conformance\n    environments: [{ARM64}]\n"
+            ),
+            &Narrowing::default(),
+        )
+        .unwrap();
+    }
+
     #[test]
     fn expected_tests_are_the_ones_in_the_suite() {
         let jobs = expand_plan(

tests/suites/conformance/sandbox/Cargo.toml

@@ -13,3 +13,6 @@ publish = false
 
 [dependencies]
 openshell-testkit = { path = "../../../testkit/testkit" }
+serde_json = "1"
+tempfile = "3"
+tokio = { version = "1", features = ["time"] }

tests/suites/conformance/sandbox/README.md

@@ -8,10 +8,17 @@ the file with `_` replaced by `-` (`file_transfer.rs` is `file-transfer`).
 | Suite | Covers |
 |---|---|
 | `smoke` | Create a sandbox, run `echo` in it, delete it |
+| `lifecycle` | SDK: long-running and run-to-completion sandboxes, the default command, restart of a stopped sandbox, deleting a stopped sandbox |
+| `cli` | The same lifecycle through the `openshell` CLI: status, create, get, list, exec, stop, start, delete |
+| `persistent-workspace` | Files in `/sandbox` survive stop and start |
+| `file-transfer` | `sandbox upload` and `download`: round trips, `.gitignore` filtering, workspace-escape and option-looking names |
 
-The suite is small on purpose. It proves that a plan can run a real test through
-the runner on CI. The older suites move to testkit by adding suites here (or a
-new family under `tests/suites/`) and listing them in a plan.
+`smoke` is the smallest end-to-end check that a plan can run a real test through
+the runner on CI. The other suites are ports of the old `conformance/cli`
+scenarios and of Evan Lezar's suites in NVIDIA/OpenShell#4629 (credited in each
+file). `file-transfer` runs `git` on the machine that runs the tests, so a test
+environment needs it installed (`tests/tmachine/ansible/playbooks/test-tools.yaml`
+does that for the tmachine environments).
 
 ## Running it
 
@@ -47,8 +54,9 @@ leave driver behavior alone.
   published docs.
 - Assert `phase` and `exit_code`. Don't assert condition reasons.
 - Check that something did not happen by reading state, not by sleeping.
-- Every environment must pass every test here. If a behavior is left to the
-  driver, the test belongs in a `features` or `drivers` suite instead.
+- Don't test behavior that only some drivers have. A test that can't pass on
+  every environment isn't conformance; put it in `tests/suites/features` or
+  `tests/suites/drivers`.
 - Use the CLI for upload, download, connect and forward. Rely on its exit code
   and `--output json`.
 - Add a testkit helper instead of calling raw gRPC from a test.