Configuring the RocksDB block cache

I learn by making mistakes and the rate at which RocksDB improves makes it easier for me to make mistakes. My mistake here was reading that RocksDB has high and low priority classes for the block cache and assuming there was a medium priority. There is a third priority but its name is bottom and it is lower than low priority.

RocksDB has had an option to do key-value separation for many years. That was recently rewritten so it could be feature complete with the RocksDB API. The official name for the feature is Integrated BlobDB but I just call it BlobDB.

I am excited about BlobDB and have begun testing it after adding support for it to tools/benchmark.sh. But it brings a new challenge with respect to the RocksDB block cache that stores uncompressed data, index and filter blocks in memory. 

Thanks to a great intern project, with BlobDB there is now another thing that can be stored in the block cache -- blob values. The usage of the db_bench option named use_shared_block_and_blob_cache option shows how that is enabled. There are 3 priority classes that can be used by the block cache: high, low, bottom.

The options that control this are:

• high_pri_pool_ratio
• The fraction of the cache devoted to filter and index blocks.
• low_pri_pool_ratio 
• When zero then data block and blob values have the same priority. Otherwise, this is the fraction of the cache for data blocks and blob values get the remainder (1 - high_pri_pool_ratio - low_pri_pool_ratio). The sum of high_pri_pool_ratio and low_pri_pool_ratio should be <= 1.

The priority for storing things in the cache is one of the following depending on how you configure it:

• everything has the same priority
• when high_pri_pool ratio=0 and low_pri_pool_ratio=0
• index & filter blocks have high priority, data blocks and blob values have low priority
• when high_pri_pool ratio > 0 and low_pri_pool_ratio=0
• index & filter blocks have high-pri, data blocks have low-pri, blob values have bottom-pri
• when high_pri_pool ratio > 0 and low_pri_pool_ratio > 0
• data, index & filter blocks have low priority, blob values have bottom priority
• when high_pri_pool_ratio=0 and low_pri_pool_ratio > 0