Skip to content

feat(tests): let external compute drivers run the OpenShell conformance suite #4848

Open

Description

@matthewgrossman

User Story

As a maintainer, I field questions from teams building external compute drivers
against compute_driver.proto. They want to check, in their own CI, that their
driver behaves like the built-in ones.

Problem Statement

There is no supported way for an external compute driver to run OpenShell's
conformance tests against its own gateway. The only option today is running the
CLI conformance suite from a source checkout (see TESTING.md), which isn't
documented for external use and is being replaced by testkit suites.

Impact / Why This Matters

  • External driver authors can't tell whether their driver meets the same bar as
    the built-in drivers.
  • Without a shared suite, each driver writes its own checks and expectations
    drift.
  • We have no single answer to point people at.

Proposed Design

Goal: an external driver can point OpenShell's conformance tests at its own
gateway, from its own CI, without building OpenShell.

Direction, details to be worked out:

  • Conformance means behavior every driver must pass. Tests specific to a driver
    or a setup are kept separate.
  • The tests can be run against any existing gateway; they don't provision
    environments.
  • They are packaged so an external project can get them for a given OpenShell
    version.

Acceptance Criteria

  • An external driver's CI can run OpenShell conformance against its own gateway
    without checking out or building OpenShell.
  • Docs explain how to do it.

Alternatives Considered

  • Document the existing from-source path as the supported way. Works today, but
    ties external CI to building OpenShell and to a suite being replaced.
  • Run external drivers in OpenShell's CI. Doesn't scale.

Checklist

  • I've reviewed existing issues and the published docs
  • This is a design proposal, not a "please build this" request

Related: #3954 (testing strategy), #3460 (RFC 0016), #4773, #4774.

Activity

  1. jyotir07 commented on Oct 23, 2026

    @jyotir07

    Hey, I’d love to work on this if it’s still available! I’m planning to first understand the existing conformance test infrastructure, identify the driver-agnostic tests, and package them into a reusable runner that external compute-driver projects can run against their own gateways in CI, without needing to build OpenShell. I’ll also add a minimal CI example and documentation.
    Let me know if I can take this up!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions