Testsuite for SCS standards

This test suite has two parts:

• one part for SCS-compatible IaaS, located under Tests/iaas,
• one part for SCS-compatible KaaS, located under Tests/kaas.

Preparation

Environment configuration (env.yaml)

Some tests can be adjusted to the environment by using variables. These variables can be specified in $HOME/.config/scs/env.yaml. Use the subject as a key for a dictionary for the variables. For instance, with a subject openstack and two variables dns_nameservers and external_gateway_net_id, you could declare:

openstack:
    dns_nameservers: ['1.2.3.4']
    external_gateway_net_id: abcdef12-3456-7890-abcd-ef1234567890

Similarly, for KaaS environments requiring specific test execution permissions (such as mounting an administrative kubeconfig into the Sonobuoy e2e test pod for subjects requiring system:masters group membership in Kubernetes 1.36), you can declare sonobuoy_fix:

syself-1.36:
    sonobuoy_fix: inject-kubeconfig

Python dependencies

Please ensure you have installed the testsuite dependencies from the respective requirements.txt file. We recommended using Python >= 3.12 and to install the requirements into a virtualenv as follows:

$ cd Tests/iaas  # NOTE for kaas do: cd Tests/kaas
$ python3 -m venv .venv && source .venv/bin/activate
$ pip install -r requirements.txt

Sonobuoy (SCS-compatible KaaS only)

Make sure that you have Sonobuoy installed and in your PATH environment variable, e.g.,

$ which sonobuoy
/home/username/.local/bin/sonobuoy

Running tests

SCS-compatible IaaS

With a cloud environment configured in your ~/.config/openstack/clouds.yaml and secure.yaml, run

cd Tests/iaas
./openstack_test.py run SUBJECT

where SUBJECT is name of your cloud environment in clouds.yaml.

SCS-compatible KaaS

Given a kubeconfig file path/to/kubeconfig.yaml, run

cd Tests/kaas
./k8s_test.py run --kubeconfig path/to/kubeconfig.yaml SUBJECT

Replace SUBJECT with an arbitrary, but meaningful subject name. Also, please note that the check will always use the current-context of the kubeconfig and will fail if it isn't set.

Note: Sonobuoy checks (such as CNCF k8s conformance) are known to run for multiple hours (during which you won't see feedback). In order to save wall-time, the sonobuoy checks will be run in parallel. You disable that by simply adding --execution_mode serial to the command-line.

You will find all files related to this test run under ~/.local/share/scs/sck/$SUBJECT (where sck is for scs-compatible-kaas); among them the files generated by Sonobuoy. Those include a JUnit XML file: plugins/e2e/results/global/junit_01.xml. You can render it to HTML with a tool like junit2html. This might give you hints as to why a test failed.

For environments requiring specific test execution workarounds (such as mounting the admin kubeconfig into the e2e test pod for subjects requiring system:masters group membership in Kubernetes 1.36), refer to the Environment configuration (env.yaml) section above to configure sonobuoy_fix in $HOME/.config/scs/env.yaml.

Submitting results to the compliance monitor

Prerequisites

The test scripts will automatically submit a report of the run to the compliance monitor, provided that the following files are present:

• $HOME/.config/scs/.secret/keyfile private key for signing the report,
• $HOME/.config/scs/.secret/tokenfile token for authenticating with the compliance monitor.

You can create these files using Tests/add_subject.py.

Note that you will also have to open a pull request to make the public counterparts of these files known to the compliance monitor. For more information, see the official SCS docs page.

Aggregating attestations

The command line for running the tests will only create attestations for individual testcases, such as scs-0123-service-network or cncf-k8s-conformance. In order to create attestations for the whole scope (SCS-compatible IaaS or KaaS), you need to run

# for iaas
cd Tests/iaas
./openstack_test.py aggregate SUBJECT

# for kaas
cd Tests/kaas
./k8s_test.py aggregate SUBJECT

Aggregating regions (SCS-compatible IaaS only)

If you want to have multiple regions certified, you would run tests for each region and then aggregate (like above) across all regions at once, like so:

cd Tests/iaas
# assuming acme is your product, with regions north and south
./openstack_test.py run acme-north acme-south
./openstack_test.py aggregate acme acme-north acme-south

If you use multiple subjects with openstack_test.py run like in this example, then these subjects will be tested concurrently. This means that the console output can become quite disorderly.

Provisioning and unprovisioning clusters (SCS-compatible KaaS only)

Usually the tests must be performed on freshly created clusters. The testsuite uses a plugin system to handle provisioning and unprovisioning of clusters across multiple cluster-lifecycle-management systems, such as Gardener or ClusterStacks.

For each of the test subjects (i.e., for each active k8s branch), create an entry in $HOME/.config/scs/clusters.yaml -- have a look at /playbooks/clusters.yaml.j2 in this repository to get an idea.

Once you have these entries in place, you can provision a cluster, run the testsuite on that cluster, and unprovision the cluster like so:

cd Tests/kaas
./k8s_test.py provision acme-1.35  # assuming acme is your product name
./k8s_test.py run acme-1.35   # no need to supply --kubeconfig!
./k8s_test.py unprovision acme-1.35

All these commands also accept multiple subjects, which will then be processed concurrently, trading wall time for an increased load on your resources.

Aggregating k8s branches (SCS-compatible KaaS only)

The certification in this case requires that all active k8s branches be tested. The attestations would need to be aggregated across all branches at once, like so:

cd Tests/kaas
# provision and unprovision omitted here for brevity
./k8s_test.py run acme-1.35 acme-1.36 acme-1.37
# (do three calls of run if you want it sequentially)
./k8s_test.py aggregate acme-kaas acme-1.35 acme-1.36 acme-1.37

Testing in docker containers

(this section is currently under construction)

Build a docker container

docker build --tag scs-compliance-check .

Run tests in a docker container

You'll have to bind mount your respective config(s), pass required parameters and the specification file.

For IaaS:

docker run -v ~/.config/openstack:/root/.config/openstack:ro scs-compliance-check -a os_cloud=CLOUDNAME -s CLOUDNAME scs-compatible-iaas.yaml

For KaaS:

docker run -v /path/to/kubeconfig.yaml:/root/kubeconfig.yaml:ro scs-compliance-check -a kubeconfig=/root/kubeconfig.yaml -a subject_root=. -s SUBJECT scs-compatible-kaas.yaml

If you want to test against a cluster running on localhost (e.g., kind cluster), replace docker run with docker run --net=host in the above invocation.

Debugging

docker run -it -v ~/.config/openstack:/root/.config/openstack:ro --entrypoint /bin/bash scs-compliance-check

Information for developers

Unit and regression tests

Some of the conformance tests scripts are themselves tested with unit tests.

To run them, first ensure that you have installed the unit test dependencies in addition to the main dependencies (inside your virtualenv as described above):

pip install -r test-requirements.txt

Now you can run the unit tests with pytest:

# Option A: let pytest discover and run all unit tests (**/*_test.py)
pytest

# Option B: run only a subset of the tests
pytest kaas/k8s-version-policy/k8s_version_policy_test.py

# Option C: produce a HTML code coverage report and open it
pytest --cov --cov-report=html
xdg-open htmldoc/index.html

You are encouraged to cover new conformance tests with unit tests! We run the tests on a regular basis in our GitHub workflows.

Maintaining the Python dependencies

We use pip-compile to pin and upgrade our dependencies in a controlled manner.

With pip-compile, you list your dependencies with as few version constraints as possible in a dedicated file, and then you have pip-compile generate the conventional requirements.txt with fully pinned dependencies.

The tool pip-compile can be installed via pip install pip-tools. It needs to be run in two cases:

1. You modified an *.in file: run pip-compile <INFILE>. For example:

   pip-compile test-requirements.in

2. You want to bump the pinned dependencies: add the --upgrade flag to the pip-compile invocation. For example:

   pip-compile --upgrade requirements.in

We use a layered approach to allow for selective installation:

• the basic layer is requirements.in,
• above that we have test-requirements.in.

Whenever you change or recompile one of these layers, all layers above that layer have to be recompiled as well.

Note: The Python version used for running pip-compile should be consistent. The currently used version is documented in the header of the requirements.txt. It should match the version used in the Docker image (see Dockerfile) and in our GitHub workflows (lint-python.yml and test-python.yml in .github/workflows).