Compaction overview

Data in Cassandra is created in memtables. Once a memory threshold is reached, to free up memory again, the data is written to an SSTable, an immutable file residing on disk.

Because SSTables are immutable, when data is updated or deleted, the old data is not overwritten with inserts or updates, or removed from the SSTable. Instead, a new SSTable is created with the updated data with a new timestamp, and the old SSTable is marked for deletion. The piece of deleted data is known as a tombstone.

Over time, Cassandra may write many versions of a row in different SSTables. Each version may have a unique set of columns stored with a different timestamp. As SSTables accumulate, the distribution of data can require accessing more and more SSTables to retrieve a complete row.

To keep the database healthy, Cassandra periodically merges SSTables and discards old data. This process is called compaction.

Since SSTables are consulted during read operations, it is important to keep the number of SSTables small. Write operations will cause the number of SSTables to grow, so compaction is necessary. Besides the issue of tombstones, data is deleted for other reasons, too, such as Time-To-Live (TTL) expiration of some data. Deleting, updating, or expiring data are all valid triggers for compaction.

Two important factors accomplished by compaction are performance improvement and disk space reclamation. If SSTables have duplicate data that must be read, read operations are slower. Once tombstones and duplicates are removed, read operations are faster. SSTables use disk space, and reducing the size of SSTables through compaction frees up disk space.

Compaction works on a collection of SSTables. From these SSTables, compaction collects all versions of each unique row and assembles one complete row, using the most up-to-date version (by timestamp) of each of the row’s columns. The merge process is performant, because rows are sorted by partition key within each SSTable, and the merge process does not use random I/O. The new versions of each row is written to a new SSTable. The old versions, along with any rows that are ready for deletion, are left in the old SSTables, and are deleted as soon as pending reads are completed.

The concept of compaction is used for different kinds of operations in Cassandra, the common thing about these operations is that it takes one or more SSTables, merges, and outputs new SSTables. The types of compactions are:

Different compaction strategies are available to optimize for different workloads. Picking the right compaction strategy for your workload will ensure the best performance for both querying and for compaction itself.

Data in Cassandra can have an additional property called time to live - this is used to automatically drop data that has expired once the time is reached. Once the TTL has expired the data is converted to a tombstone which stays around for at least gc_grace_seconds. Note that if you mix data with TTL and data without TTL (or just different length of the TTL) Cassandra will have a hard time dropping the tombstones created since the partition might span many SSTables and not all are compacted at once.

If an SSTable contains only tombstones and it is guaranteed that SSTable is not shadowing data in any other SSTable, then the compaction can drop that SSTable. If you see SSTables with only tombstones (note that TTL-ed data is considered tombstones once the time-to-live has expired), but it is not being dropped by compaction, it is likely that other SSTables contain older data. There is a tool called sstableexpiredblockers that will list which SSTables are droppable and which are blocking them from being dropped. With TimeWindowCompactionStrategy it is possible to remove the guarantee (not check for shadowing data) by enabling unsafe_aggressive_sstable_expiration.

With incremental repairs Cassandra must keep track of what data is repaired and what data is unrepaired. With anticompaction repaired data is split out into repaired and unrepaired SSTables. To avoid mixing up the data again separate compaction strategy instances are run on the two sets of data, each instance only knowing about either the repaired or the unrepaired SSTables. This means that if you only run incremental repair once and then never again, you might have very old data in the repaired SSTables that block compaction from dropping tombstones in the unrepaired (probably newer) SSTables.

Since tombstones and data can live in different SSTables it is important to realize that losing an SSTable might lead to data becoming live again - the most common way of losing SSTables is to have a hard drive break down. To avoid making data live tombstones and actual data are always in the same data directory. This way, if a disk is lost, all versions of a partition are lost and no data can get undeleted. To achieve this a compaction strategy instance per data directory is run in addition to the compaction strategy instances containing repaired/unrepaired data, this means that if you have 4 data directories there will be 8 compaction strategy instances running. This has a few more benefits than just avoiding data getting undeleted:

When an SSTable is written a histogram with the tombstone expiry times is created and this is used to try to find SSTables with very many tombstones and run single SSTable compaction on that SSTable in hope of being able to drop tombstones in that SSTable. Before starting this it is also checked how likely it is that any tombstones will actually will be able to be dropped how much this SSTable overlaps with other SSTables. To avoid most of these checks the compaction option unchecked_tombstone_compaction can be enabled.

There is a number of common options for all the compaction strategies;

Further, see the section on each strategy for specific additional options.

The nodetool <nodetool> utility provides a number of commands related to compaction:

It is possible to switch compaction strategies and its options on just a single node using JMX, this is a great way to experiment with settings without affecting the whole cluster. The mbean is:

and the attribute to change is CompactionParameters or CompactionParametersJson if you use jconsole or jmc. For example, the syntax for the json version is the same as you would use in an ALTER TABLE <alter-table-statement> statement:

The setting is kept until someone executes an ALTER TABLE <alter-table-statement> that touches the compaction settings or restarts the node.

Enable with the compaction option log_all and a more detailed compaction log file will be produced in your log directory.