Testing

JVM upgrade tests can be run precisely in the same way as any other JVM distributed tests. However, running them requires some preparation - for example, if a test verifies the upgrade from Cassandra 3.0 and Cassandra 3.11 to the current version (say Cassandra 4.0), you need to have prepared dtest uber JARs for all involved versions. To do this:

If you are on Linux, you need to install Docker using the system package manager.

If you are on MacOS, you can use either Docker Desktop or some other approach.

The Docker image used on the official Cassandra CI can be found in this repository. You can use either docker/testing/ubuntu2004_j11.docker or docker/testing/ubuntu2004_j11_w_dependencies.docker The second choice has prefetched dependencies for building each main Cassandra branch. Those images can be either built locally (as per instructions in the GitHub repo) or pulled from the Docker Hub - see here.

First, pull the image from Docker Hub (it will either fetch or update the image you previously fetched):

To log into the container, use the following docker exec command:

The tests are implemented in Python, so a Python virtual environment (see here for details) with all the required dependencies is good to be set up. If you are familiar with the Python ecosystem, you know what it is all about. Otherwise, follow the instructions; it should be enough to run the tests.

For Python distributed tests do:

For CQLSH tests, replace some paths:

The above commands are also helpful for importing those test projects into your IDE. In that case, you need to run them on your host system rather than in Docker container. For example, when you open the project in IntelliJ, the Python plugin may ask you to select the runtime environment. In this case, choose the existing virtualenv based environment and point to bin/python under the created dtest-venv directory (or cqlsh-venv, or whichever name you have chosen).

Whether you want to play with Python distributed tests or CQLSH tests, you need to select the right virtual environment. Remember to switch to the one you want:

or

You may run all test tests from the selected file by passing that file as an argument:

To run a specific test case, you need to specify the module, class name, and the test name, for example:

Note that you need to set up and activate the virtualenv for DTests (see Setup Python environment section for details). Tests are implemented with the PyTest framework, so you use the pytest command to run them. Let’s run some tests:

That command runs the test_clustering_order test case from TestSchemaMetadata class, located in the schema_metadata_test.py file. You may also provide the file and class to run all test cases from that class:

or just the file name to run all test cases from all classes defined in that file.

You may also specify more individual targets:

If you run pytest without specifying any test, it considers running all the tests it can find. More on the test selection here You probably noticed that --cassandra-dir=/home/cassandra/cassandra is constantly added to the command line. It is one of the cassandra-dtest custom arguments - the mandatory one - unless it is defined, you cannot run any Cassandra dtest.

All the possible options can be listed by invoking pytest --help. You see tons of possible parameters - some of them are native PyTest options, and some come from Cassandra DTest. When you look carefully at the help note, you notice that some commonly used options, usually fixed for all the invocations, can be put into the pytest.ini file. In particular, it is quite practical to define the following:

so that you do not have to provide --cassandra-dir param each time you run a test. The other two options set up console logging - remove them if you want logs stored only in log files.

There are a couple of options to enforce exact test configuration (their names are quite self-explanatory):

Note that the list can grow in the future as new predefined configurations can be added to dtests. It is also possible to pass extra Java properties to each Cassandra node started by the tests - define those options in the JVM_EXTRA_OPTS environment variable before running the test.

You can do a dry run, so that the tests are only listed and not invoked. To do that, add --collect-only to the pytest command. That additional -q option will print the results in the same format as you would pass the test name to the pytest command:

lists all the tests pytest would run if no particular test is specified. Similarly, to list test cases in some class, do:

You can copy/paste the selected test case to the pytest command to run it.

Upgrade tests always involve more than one product version. There are two kinds of upgrade tests regarding the product versions they span - let’s call them fixed and generated.

In case of fixed tests, the origin and target versions are hardcoded. They look pretty usual, for example:

prints:

When you look into the code, you will see the fixed upgrade path:

The generated upgrade tests are listed several times - the first occurrence of the test case is a generic test definition, and then it is repeated many times in generated test classes. For example:

prints:

In this example, the test case name is just test_set, and the class name is TestCQL - the suffix of the class name is automatically generated from the provided specification. The first component is the cluster specification - there are two variants: Nodes2RF1 and Nodes3RF3 - they denote that the upgrade is tested on 2 nodes cluster with a keyspace using replication factor = 1. Analogously the second variant uses 3 nodes cluster with RF = 3.

Then, there is the upgrade specification - for example, Upgrade_indev_3_11_x_To_indev_4_0_x - which means that this test upgrades from the development version of Cassandra 3.11 to the development version of Cassandra 4.0 - the meaning of indev/current and where they are defined is explained later.

When you look into the implementation, you notice that such upgrade test classes inherit from UpgradeTester class, and they have the specifications defined at the end of the file. In this particular case, it is something like:

As you can see, there is a list of the cluster specifications and the cross product is calculated with upgrade paths returned by the build_upgrade_pairs() function. That list of specifications is used to dynamically generate upgrade tests.

Suppose you need to test something specifically for your scenario. In that case, you can add more cluster specifications, like a test with 1 node or a test with 5 nodes with some different replication factor or consistency level. The build_upgrade_pairs() returns the list of upgrade paths (actually just the origin and target version). That list is generated according to the upgrade manifest.

The upgrade test can use your local Cassandra distribution, the one specified by the cassandra_dir property, as the target version if the following preconditions are satisfied:

For example, your local distribution is branched off from the cassandra-4.0 branch, likely matching indev_4_0_x. It means that the upgrade path with target version indev_4_0_x uses your local distribution. There is a handy command line option which will filter out all the upgrade tests which do not match the local distribution: --upgrade-target-version-only. Given you are on cassandra-4.0 branch, when applied to the previous example, it will be something similar to:

prints:

You can see that the upgrade tests were limited to the ones whose target version is indev and family matches 4.0.

A couple of common PyTest arguments control what is logged to the file and the console from the Python test code. Those arguments which start from --log-xxx are pretty well described in the help message (pytest --help) and in PyTest documentation, so it will not be discussed further. However, most of the tests start with the cluster of Cassandra nodes, and each node generates its own logging information and has its own data directories.

By default the logs from the nodes are copied to the unique directory created under logs subdirectory under root of dtest project. For example:

The last item is a symbolic link to the directory containing the logs from the last executed test. Each such directory includes logs from each started node - system, debug, GC as well as standard streams registered upon each time the node was started:

Those log files are not collected if --delete-logs command-line option is added to PyTest. The nodes also produce data files which may be sometimes useful to examine to resolve some failures. Those files are usually deleted when the test is completed, but there are some options to control that behavior:

Now, how to find where is that directory for the certain test - you need to grab that information from the test logs - for example, you may add -s option to the command line and then look for "dtest_setup INFO" messages. For example:

says that the cluster work directory is /tmp/dtest-0onwvgkr, and all node directories can be found under the test subdirectory: