5 things to set when configuring RocksDB and MyRocks

The 5 options to set for RocksDB and MyRocks are:

1. block cache size
2. number of background threads
3. compaction priority
4. dynamic leveled compaction
5. bloom filters

I have always wanted to do a "10 things" posts but prefer to keep this list small. It is unlikely that RocksDB can provide a great default for the block cache size and number of background threads because they depend on the amount of RAM and number of CPU cores in a server. But I hope RocksDB or MyRocks are changed to get better defaults for the other three which would shrink this list from 5 to 2.

Options

My advice on setting the size of the RocksDB block cache has not changed assuming it is configured to use buffered IO (the default). With MyRocks this option is rocksdb_block_cache_size and with RocksDB you will write a few lines of code to setup the LRU.

The number of background threads for flushing memtables and doing compaction is set by the option rocksdb_max_background_jobs in MyRocks and max_background_jobs in RocksDB. There used to be two options for this. While RocksDB can use async read-ahead and write-behind during compaction, it still uses synchronous reads and a potentially slow fsync/fdatasync call. Using more than 1 background job helps to overlap CPU and IO. A common configuration for me is number-of-CPU-cores / 4. With too few threads there will be more stalls from throttling. With too many threads there the threads handling user queries might suffer.

There are several strategies for choosing the next data to compact with leveled compaction in RocksDB. The strategy is selected via the compaction_pri option in RocksDB. This is harder to set for MyRocks -- see compaction_pri in rocksdb_default_cf_options. The default value is kByCompensatedSize but the better choice is kMinOverlappingRatio. With MyRocks the default is 0 and the better value is 3 (3 == kMinOverlappingRatio). I first wrote about compaction_pri prior to the arrival of kMinOverlappingRatio. Throughput is better and write amplification is reduced with kMinOverlappingRatio. An awesome paper by Hyeontaek Lim et al explains this.

Leveled compaction in RocksDB limits the amount of data per level of the LSM tree. A great description of this is here. There is a target size per level and this is enforced top down (smaller to larger levels) or bottom up (larger to smaller levels). With the bottom up approach the largest level has ~10X (or whatever the fanout is set to) more data than the next to last level. With the top down approach the largest level frequently has less data than the next to last level. I strongly prefer the bottom up approach to reduce space amplification. This is enabled via the level_compaction_dynamic_level_bytes option in RocksDB. It is harder to set for MyRocks -- see rocksdb_default_cf_options.

Bloom filters are disabled by default for MyRocks and RocksDB. I prefer to use a bloom filer on all but the largest level. This is set via rocksdb_default_cf_options with MyRocks. The reason for not using it with the max level is to consume less memory (reduce cache amplification). The bloom filter is skipped for the largest level in MyRocks via the optimize_filter_for_hits option. The example at the end of this post has more information on enabling bloom filters. All of this is set via rocksdb_default_cf_options.

Examples

A previous post recently explained how to set rocksdb_default_cf_options for compression with MyRocks. Below I share an example my.cnf for MyRocks to set the 5 options I listed above. I set transaction isolation because read committed is a better choice for MyRocks today. Repatable read will be a great choice after gap locks are added to match InnoDB semantics. In rocksdb_default_cf_options block_based_table_factory is used to enable the bloom filter, level_compaction_dynamic_level_bytes enables bottom up management of level sizes, optimize_filters_for_hits disables the bloom filter for the largest level of the LSM tree and compaction_pri sets the compaction priority.

transaction-isolation=READ-COMMITTED

default-storage-engine=rocksdb

rocksdb

rocksdb_default_cf_options=block_based_table_factory={filter_policy=bloomfilter:10:false};level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true;compaction_pri=kMinOverlappingRatio

rocksdb_block_cache_size=2g

rocksdb_max_background_jobs=4