Checking MyRocks 5.6 for regressions with sysbench and a medium server, part 3

I used sysbench to check for performance regressions over many releases of MyRocks 5.6. I previously shared results using the Insert Benchmark. Here I have results for builds from 2021/04/07 through 2023/05/29.

tl;dr

• QPS for modern MyRocks on read-only tests is mostly better than older MyRocks
• QPS for modern MyRocks on write-only tests is ~10% less than older MyRocks
• Modern MyRocks does much better than older MyRocks for read-heavy tests that run after the database has been subject to random writes.
• Modern MyRocks does much better on full scans

Builds

The builds tested are described in the previous post. All builds used the y9c5_gcp_c2s30 my.cnf file.

Benchmark

I used sysbench and my usage is explained here. RocksDB was configured to cached all tables.

The benchmark used a c2-standard-30 server from GCP with Ubuntu 22.04, 15 cores, hyperthreads disabled, 120G of RAM and 1.5T of storage from RAID 0 over 4 local NVMe devices with XFS.

The benchmark is run with 8 clients and 4 tables with 20M rows per-table. The read-only tests ran for 600 seconds each and the other tests ran for 1200 seconds each.

Results

All of the results are here and each chart below is there too (and easier to read there). For the charts below I split the results into two groups of point query tests, two groups of range query tests and one group for read-write/write-only tests. 

The charts use relative QPS which is: (QPS for a given version / QPS for the base case) and the base case is the 20210407 build that has a much longer name, fbmy5635_rel_202104072149, in my previous post. The names used below are the date for which builds were done, YYYMMDD, and then for the last three the three digits at the end (832, 843, 850) are the RocksDB version when I used a more recent version of RocksDB than the build would otherwise use.

Note 

• the y-axis for all charts starts at 0.8 rather than 0.0 to improve readability
• the test names under the x-axis are truncated in the images here but readable here

Point query, group 1

The points-covered-si_range=100 test is an outlier. The test fetches 100 rows via point lookups on a covered secondary index. The same query is done for points-covered-si_pre_range=100 and the improvement there isn't as large. The difference between these two is that pre_range is run before the database is subject to random writes.

Point query, group 2

Here, modern MyRocks does better on the tests that run after the database gets random writes than it does on the pre_range tests that run prior.

Range query, group 1

Four tests show a significant improvement in modern MyRocks. All of them run after the database is subject to random writes. The benchmark steps that don't show the speedup run prior to the random writes.

• range-covered-pk_range=100
• range-covered-si_range=100
• range-notcovered-pk_range=100
• range-notcovered-si_range=100

Range query, group 2

Performance for full scans is much better in modern MyRocks. Relative performance is better for tests that run after the database has been subject to random writes.

Read-write and write-only

QPS on write-only tests in modern MyRocks is ~10% less than older MyRocks. The cases below where it modern is faster than older MyRocks is from tests that are read-write.

Summary statistics

Summary statistics for the relative QPS results are in the spreadsheet here in the tab named qps.rel.56cached. Summary stats for the most recent build (20230529_850) are also below.

Point-1	20230529_850
Average	1.27
Median	1.14
Min	0.97
Max	2.14
Stddev	0.372
Point-2	20230529_850
Average	1.10
Median	1.08
Min	0.99
Max	1.24
Stddev	0.118
Range-1	20230529_850
Average	1.14
Median	1.11
Min	0.91
Max	1.45
Stddev	0.223
Range-2	20230529_850
Average	1.23
Median	1.09
Min	0.92
Max	2.17
Stddev	0.445
Writes	20230529_850
Average	0.94
Median	0.91
Min	0.87
Max	1.11
Stddev	0.080