Understanding the integration testing infrastructure

Contributors are encouraged to write tests for their operators. For small bug fixes and enhancements, unit tests are preferred. Larger developments should have integration tests. Integration tests are declarative in form of manifest files that are executed by KuTTL on a Kubernetes cluster. Each test case is comprised of multiple steps, each with a setup and an assertion file. Furthermore, to support a high dimensional matrix of configurations, the manifest files are partly templated as Jinja2 files. This allows for example to test the behaviour of multiple product versions with and without various features enabled. Having the manifest files as Jinja2 templates avoids duplication and makes the tests more maintainable.

Requirements

To run the integration tests from your local machine, you need to have the follwing tools installed:

Python to run scripts.
KuTTL to run the tests.
beku our custom test expander.
stackablectl to install operators.

A Kubernetes cluster is needed to run the tests. You can use kind to create one on your local machine.

Running the integration tests

To run integration tests, use the scripts/run-tests Python script. This script assumes you have access to a Kubernetes cluster and the requirements above are installed.

To run the tests just call scripts/run-tests in the root of the operator repository.

To run just one test case pass the --test option like this:

$ ./scripts/run-tests --test smoke

To skip operator installation (assume the operator is already installed in the cluster):

$ ./scripts/run-tests --skip-release

There are many more arguments available. Use the --help argument to find out more about the tool.

Tests structure

The integration tests are located in the test directory of the operator repository.

The structure of the tests directory is as follows:

tests
├── test-definition.yaml
├── release.yaml
├── kuttl-test.yaml.jinja2
├── README-templating.md
├── templates
│   └── kuttl
│       ├── resources
│       └── smoke
└── _work
    ├── kuttl-test.yaml
    └── tests
        └── smoke

The tests directory contains the test manifest files.
- tests/test-definition.yaml specifies the variables to use for each test case.
- tests/release.yaml specifies operators to test. SDP operators usually work together to achieve a single task, so they are tested together. You can override the versions in the file by passing the --operator command line argument to run-tests.
- tests/templates contains the test cases organized in subdirectories. In the example above, there two test cases: smoke and resources.
- tests/_work is a temporary (non-versioned) folder that is generated by expanding the test matrix according to the call of the run-tests script. This folder is overwritten on every call of the script and should not be added to version control.

How to run tests in the CI (Jenkins)

Stackable operates a Jenkins instance where the tests are ran every night. You can also run the tests manually there, for a particular pull request, if you have an account (only internal contributors).

Adding a new test

To add a new test, add a new directory in tests/templates/kuttl and also a corresponding in the test-defintion.yaml file. Have a look at other tests to familiarize yourself with the general pattern of steps in a test defintion.