[matthewgrossman]

TL;DR

• Adds testkit, a small library for writing end-to-end tests against a running OpenShell gateway through the Rust SDK. A test is a documented async fn marked #[testkit::test] that runs with plain cargo nextest.
• Tests find the gateway the same way the openshell CLI does, so they run unchanged under e2e/with-docker-gateway.sh and e2e/with-podman-gateway.sh with no extra config.
• Each test gets unique sandbox names, and the sandboxes it creates are cleaned up whether it passes, fails, or panics.
• cx.cli(...) runs the CLI against the same gateway and workspace as the SDK.
• Branch Checks now formats, lints, and unit-tests testkit on all three platforms.
• No suites use it yet and existing e2e lanes are unchanged. This lands the foundation from test(conformance): run conformance suites from test plans in tmachine environments #4629 so the next PR can move suites onto it.

Stack

Part of a stack of three; each PR targets the one below it. The tmachine move (#4726) has merged, so the stack uses its tests/tmachine/ paths.

1. test(testkit): add the testkit library for end-to-end suites #4724 (this PR): the testkit library.
2. feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773: test selection (IDs, requirements, testkit-spec), the test runner, plans, and the tmachine Session API, with a plan running a smoke test on CI.
3. test(suites): port the universal conformance suites onto testkit and run them from the plan #4774: ports the old suites onto testkit and adds them to the plan.

Still to come, on top of #4774: moving the release lanes onto plans and deleting the old suites, once the plan has run the ported suites green.

Summary

Adds tests/testkit/, a support library for SDK-driven end-to-end suites, as its own Cargo workspace. This is a small first step to get the crate into main and checked in CI. Nothing uses it yet. The existing suites move onto it in one follow-up migration (#4774, stacked on this PR) that also deletes the old suites. Suites added after that arrive one at a time.

The code comes from @elezar's draft #4629 (head 673928804). This PR takes only the testkit portion, adapted to current main and cut down to what's needed today (see "Deferred and follow-ups" below).

Related Issue

Part of #3954

Changes

• tests/testkit/: the testkit and testkit-macros crates, plus the testkit-ready binary, taken from test(conformance): run conformance suites from test plans in tmachine environments #4629.
  • A test is an async fn marked #[testkit::test] with a required doc comment. Other attributes such as #[ignore] apply to the generated test. #[should_panic] is rejected, because a testkit test panics on any failure.
  • Tests find the gateway the same way the openshell CLI does: OPENSHELL_GATEWAY_ENDPOINT, then OPENSHELL_GATEWAY, then the active gateway. A registered gateway's endpoint and mTLS material come from its CLI config directory. Like the CLI, an endpoint takes its credentials from OPENSHELL_GATEWAY or the registered gateway with that endpoint. cx.cli runs OPENSHELL_BIN against the same gateway and workspace. Each test logs the gateway it used.
  • Cleanup deletes a test's sandboxes concurrently, each with its own timeout that covers the whole delete.
  • The readiness wait (testkit-ready) is bounded as a whole, including each connect and health call, so a gateway that accepts connections but never answers cannot hang it.
  • For an mTLS gateway, testkit reads the registered gateway's CA and client certificate and connects through the SDK's ClientConfig client identity, so TLS, connect timeouts and keep-alive stay in the SDK.
  • Cargo.lock is re-seeded from the root lockfile, because main's openshell-sdk has more dependencies than when test(conformance): run conformance suites from test plans in tmachine environments #4629 was cut.
• Branch Checks and the rust:lint, rust:format, rust:format:check, and test:rust mise tasks now run fmt, clippy, and nextest for the testkit workspace. This follows test(conformance): run conformance suites from test plans in tmachine environments #4629's cf72f1108, restricted to testkit.
• tests/testkit/README.md covers writing a test, what Context provides, and running against a gateway. TESTING.md, AGENTS.md, and CONTRIBUTING.md point to it.

Deferred and follow-ups

Dropped from #4629's testkit

Each can come back as an optional addition, without changing tests written in the meantime:

• Stable test IDs (id = "t-…"). Until a runner, waivers, or result history need them, the nextest name identifies a test.
• Capabilities and requires = [...]. These belong with the runner, which selects tests per environment. Until then, a suite excludes a test that an environment can't support with a nextest filter or #[ignore].
• The testkit-spec crate, for listing tests without compiling them. It returns with the runner's catalog.
• The target file (OPENSHELL_TEST_TARGET). It held a copy of the CLI's registered gateway plus capabilities, so testkit now reads the CLI's gateway config directly. As a result, step 3 no longer needs test(conformance): run conformance suites from test plans in tmachine environments #4629's openshell_client role change or the runner code that writes target.yaml. The runner only sets OPENSHELL_GATEWAY (or OPENSHELL_GATEWAY_ENDPOINT) and OPENSHELL_BIN.
• Guidance on what a conformance test may assert. It moves to the conformance suites.

Tests ported from #4629 must drop id and requires, because #[testkit::test] takes no arguments here. The runner PR (#4773) adds them back as optional arguments.

Rest of #4629, for later steps

• The runner, plans, environments, waivers, and matrix.
• The tmachine Session library, and the Ansible and config.nix changes.
• The new suites and e2e-suites.yml.
• The lockfile alignment script (test(conformance): run conformance suites from test plans in tmachine environments #4629's a37ebe917). It matters once testkit is built into the Nix test archives. Until then, check-cargo-lockfiles.sh already validates tests/testkit/Cargo.lock.
• Changes to existing e2e lanes.

Follow-ups found while landing this

• Share gateway resolution with the CLI. testkit's gateway.rs repeats the CLI's lookup: the endpoint-to-registered-gateway match, metadata.json auth modes, and the mtls/ files. The connection itself already goes through the SDK; only the lookup is copied, and the copies can drift apart. When step 3 lands or tmachine needs the same lookup, move it (gateway name or endpoint to endpoint, CA, and client identity) into openshell-bootstrap or an SDK helper so the CLI, testkit, and tmachine share one implementation.
• Other auth modes. testkit supports plaintext and mTLS gateways. A gateway registered with OIDC or edge-token auth is rejected with a clear error. Add support when a lane needs it.
• Small differences from the CLI. Both are documented or harmless for today's wrappers, and both go away if resolution is shared:
  • For an HTTPS URL that no registered gateway uses, testkit connects without a client certificate. The CLI guesses a certificate directory from the hostname instead.
  • testkit doesn't sanitize gateway names when building the mtls/ path, as the CLI does. Gateway names with only letters, digits, ., -, and _ are unaffected.
• Cleaning up more than sandboxes. Cleanup only deletes sandboxes created through cx.sandbox(...). Tests must delete anything else they create, such as providers. This is documented in the README. Extend the cleanup registry when suites create other resources often.
• CI build cache. testkit is its own Cargo workspace, so its build output in tests/testkit/target isn't covered by the Rust cache. This adds roughly 1–2 minutes to each Rust lint and test job. Caching it would add about 2 GB per platform cache. Revisit if suites make the build slower.
• Local runs on Docker Desktop. Sandboxes can't reach a gateway on the Mac's loopback unless Docker Desktop's host networking is on, or the gateway uses supervisor_network_mode = "bridge". e2e/with-docker-gateway.sh sets neither. Linux CI isn't affected. Consider setting bridge mode in the wrapper on macOS when the first testkit lane lands.

Raised in review and left as is

• CLI error messages include the full command line, so a credential passed as an argument would appear in test output. Arguments are already visible in process listings, and suites use throwaway credentials. Hiding values would mean guessing which arguments are secret.
• Some Sandbox helpers have no caller yet (for example config, draft_policy, and set_setting). They came from test(conformance): run conformance suites from test plans in tmachine environments #4629 for the suites in the next PR, so removing them now would only mean adding them back.

Testing

• cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check
• cargo clippy --locked --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings
• cargo nextest run --locked --config-file .config/nextest.toml --manifest-path tests/testkit/Cargo.toml (2 tests: the macro rejects misuse, and the readiness wait gives up on a gateway that never answers. Gateway resolution, the CLI wrapper, and cleanup are exercised against a real gateway below and by the suites that adopt testkit.)
• tasks/scripts/check-cargo-lockfiles.sh covers tests/testkit/Cargo.lock
• mise run license:check
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable). Not applicable: no suite uses testkit yet.

Against a real gateway

To check that testkit works end to end, I built a throwaway suite crate outside the repo. It depends on this branch's testkit and contains the six sandbox_lifecycle tests from #4629, plus a test that creates three sandboxes, checks that cx.cli(["sandbox", "list"]) shows them, and leaves them for cleanup. I ran it unchanged under e2e/with-docker-gateway.sh on macOS with Docker Desktop. The wrapper's own OPENSHELL_GATEWAY and OPENSHELL_BIN were the only configuration:

e2e/with-docker-gateway.sh sh -c 'testkit-ready 120 && cargo nextest run --manifest-path <suite>/Cargo.toml --test-threads 1'

testkit-ready resolved the wrapper's mTLS gateway, reported it healthy, and confirmed the CLI reaches it. All 7 tests passed in about 15s, and openshell sandbox list showed nothing left behind. Two #[ignore] tests, with the attribute above and below #[testkit::test], were skipped. With only OPENSHELL_GATEWAY_ENDPOINT set, as e2e/rust/e2e-vm.sh does, testkit found the registered gateway's mTLS material, and testkit-ready plus two tests passed.

On Docker Desktop, sandboxes need either Docker Desktop's host networking turned on (Settings → Resources → Network) or supervisor_network_mode = "bridge". Otherwise the supervisor cannot reach a gateway on the Mac's loopback. For this run I set bridge mode with a local, uncommitted change to the wrapper's gateway config.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4724.docs.buildwithfern.com/openshell

[drew]

Reviewed commit 40cc70740c0441fd1d862bf570b9c0610a46777a. Two findings:

1. Readiness and cleanup still lack enforced timeouts. wait_ready checks elapsed time only after health returns; cleanup awaits deletion before starting its polling timeout. An unresponsive gateway can therefore hang either operation. Reproduced again: testkit-ready 1 remained stuck after four seconds against an HTTP/2 peer that never answers health. Please wrap the complete operations in tokio::time::timeout. See gateway.rs:181 and context.rs:156.

2. The new catalog scanner silently misses nested tests. The macro works inside mod nested { ... }, and Cargo discovers those tests, but scan examines only top-level functions. I confirmed Cargo discovered three testkit tests while the scanner returned two. A runner using this catalog would miss nested tests and their requirements. Please traverse inline modules and preserve qualified names, or explicitly reject that layout. See testkit-spec/src/lib.rs:201.

The earlier attribute-forwarding issue is fixed: probes confirmed #[ignore] works above and below the macro. IDs and capability requirements are now supported through the shared parser.

All 23 existing tests passed locally. No real-gateway E2E run was performed.

[matthewgrossman]

Thanks, Drew. Status against your review of 40cc70740:

1. Timeouts: fixed in 6427db9. wait_ready now wraps the whole wait, including each connect and health call, in tokio::time::timeout, and each sandbox's cleanup wraps the delete plus the wait for deletion. A test covers a peer that accepts connections and never answers; it gives up after the timeout instead of hanging.
2. Nested tests: the scanner is part of the testkit-spec change, which moved to feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773 with the other test-selection code. There it now refuses #[testkit::test] functions inside inline modules, rather than missing them silently.
3. mTLS through the SDK: done (see the inline reply). It needed no SDK change because refactor(cli): use Rust SDK transport #4817 already added the client identity.
4. Scope: this PR is back to the testkit library only. IDs, requires/capabilities, and testkit-spec moved to feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773, the runner PR that consumes them, and the description lists them as deferred again.

The branch was rebased onto current main for #4817. No real-gateway run in CI yet; I ran the suite against a local mTLS Docker gateway.