Skip to content

test(testkit): let features and drivers suites read unsupported.yaml #5788

Closed

Description

@matthewgrossman

User Story

As a maintainer porting e2e/rust tests into features and drivers suites, I want those suites to read OPENSHELL_TEST_UNSUPPORTED, so a test that fails on one environment gets a tracked unsupported.yaml entry instead of being left off that CI row.

Description

#5755 added unsupported.yaml for conformance only. Features and drivers tests still run through the #[tokio::test] wrappers that #[testkit::test] generates under cfg(test). The standard test harness decides "ignored" at compile time, so it can't skip a test per environment.

testkit::main() already answers the libtest interface that cargo test and nextest use. A test target with harness = false builds without cfg(test), so the wrappers drop out and the target runs through testkit::main(), as conformance does. This is a narrower first part of #5765 step 2: no [[bin]] per suite and no file moves.

Changes

  1. Each test target under tests/suites/features/* and tests/suites/drivers/* (9 on main) gets a [[test]] entry with harness = false and a name that is unique across suites, plus fn main() -> ExitCode { testkit::main() }. A unique name is needed because the crate name is the first part of an entry: docker-updates and kubernetes-updates both have tests/update.rs.
  2. tests/suites/conformance/unsupported.yaml moves to tests/suites/unsupported.yaml, and the script that reads it loses its conformance- name.
  3. The CI check (tasks/scripts/check-conformance-program.sh):
  4. The "Marking a test unsupported in CI" docs move from tests/suites/conformance/README.md to one shared place.

Context

Definition of Done

  • Every features and drivers test target runs through testkit::main(); cargo nextest list shows the same tests as before.
  • A test listed in unsupported.yaml for an environment shows as skipped with its issue on that environment's row, in a features suite and in conformance.
  • The CI check covers every suite and rejects unknown crates, unknown environments and test targets without harness = false.
  • Docs describe one unsupported.yaml for all suites.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions