Searching for regressions in RocksDB with db_bench: part 2

In a recent post I shared results for RocksDB performance tests using versions from 6.0 through 9.0 and 3 different types of servers (small, medium, big). While there were few regressions over time, there is one regression that arrived in version 8.6, bug 12038, and the workarounds are one of:

  • use O_DIRECT for compaction reads
  • set compaction_readahead_size to be <= max_sectors_kb for the database storage device. When SW RAID is used I don't know whether the value that matters is from the underlying storage devices or the SW RAID device.
In this post I have more results from tests done with compaction_readahead_size set to a value <= max_sectors_kb.

tl;dr
  • Setting compaction_readahead_size to be <= max_sectors_kb was good for performance on the small and big servers. One effect of this is the average read request size is large (tens of KB) when the value is correctly sized and ~4K (single-block reads) when it is not.
  • If you don't want to worry about this then use O_DIRECT for compaction reads
Read the Builds and Benchmark sections from my recent post for more context.

Hardware

I tested on three servers:
  • Small server
    • SER4 - Beelink SER 4700u (see here) with 8 cores and a Ryzen 7 4700u CPU, ext4 with data=writeback and 1 NVMe device. The storage device has 128 for max_hw_sectors_kb and max_sectors_kb.
    • I set compaction_readahead_size to 96K
  • Medium server
    • C2D - a c2d-highcpu-32 instance type on GCP (c2d high-CPU) with 32 vCPU and 16 cores, XFS with data=writeback, SW RAID 0 and 4 NVMe devices. The RAID device has 512 for max_hw_sectors_kb and max_sectors_kb while the storage devices have max_hw_sectors_kb =2048 and max_sectors_kb =1280.
    • I set compaction_readahead_size to 512K
  • Big server
    • BIG - Intel CPU, 1-socket, 48 cores with HT enabled, enough RAM (vague on purpose), xfs with SW RAID 0 and 3 devices. The RAID device has 128 for max_hw_sectors_kb and max_sectors_kb while the storage devices have max_hw_sectors_kb =2048 and max_sectors_kb =1280.
    • I set compaction_readahead_size to 96K

Workloads

There are two workloads:

  • byrx - the database is cached by RocksDB
  • iobuf - the database is larger than memory and RocksDB uses buffered IO
Results: byrx (cached)

For each server there are links to two sets of results.

The first set of results has 3 lines per test. The first line is from RocksDB 8.5.4, the second from 8.7.3 using the default (=2M) for compaction_readahead_size and the third from 8.7.3 with compaction_readahead_size =96K. An example is here.

The second set of results is similar to the first, except the second and third lines are from RocksDB 9.3.1 instead of 8.7.3.

Below I use CRS to mean compaction_readahead_size and compare the QPS from the overwriteandwait microbenchmark.

The results:
  • SER4 (small server)
    • Results for 8.5.4 vs 8.7.3 and then for 8.5.4 vs 9.3.1
    • Results for overwriteandwait are here for 8.7.3 and for 9.3.1
      • 8.7.3 and 9.3.1 with CRS =2M get ~12% less QPS than 8.5.4
      • 8.7.3 and 9.3.1 with CRS =96K get ~6% less QPS than 8.5.4
      • Setting CRS to be <= max_sectors_kb is good for perf
  • C2D (medium server)
    • Results for 8.5.4 vs 8.7.3 and then for 8.5.4 vs 9.3.1
    • Results for overwriteandwait are here for 8.7.3 and for 9.3.1
      • 8.7.3 and 9.3.1 with CRS =2M get ~2% more QPS than 8.5.4
      • 8.7.3 and 9.3.1 with CRS =96K get ~26% more QPS than 8.5.4
      • Setting CRS to be <= max_sectors_kb is good for perf
  • BIG (big server)
    • Results for 8.5.4 vs 8.7.3 and then for 8.5.4 vs 9.3.1
    • Results for overwriteandwait are here for 8.7.3 and for 9.3.1
      • 8.7.3 gets the same QPS as 8.5.4 with CRS set to either =2M or =96K
      • 9.3.1 gets ~4% less QPS than 8.5.4 with CRS set to either =2M or =96K
Summary
  • Setting compaction_readahead_size to be <= max_sectors_kb helps performance on the small and medium server but not on the big server. Note there are large differences on the big server between the value of max_sectors_kb for the RAID device and for the underlying storage devices -- it is much larger for the storage devices.
  • In the cases where reducing the value of compaction_readahead_size helped, QPS from overwriteandwait in RocksDB 8.5.4 is still better than the versions that follow
Results: iobuf (IO-bound with buffered IO)

For each server there are links to two sets of results.

The first set of results has 4 lines per test. The first line is from RocksDB 8.5.4, the second from 8.7.3 using the default (=2M) for compaction_readahead_size and the third from 8.7.3 with compaction_readahead_size =96K. O_DIRECT was not used for the first three lines. The fourth line is from 8.7.3 using O_DIRECT. An example is here.

The second set of results is similar to the first, except the second, third and fourth lines are from RocksDB 9.3.1 instead of 8.7.3.

Below I use CRS to mean compaction_readahead_size and compare the QPS from the overwriteandwait microbenchmark.

The results:
  • SER4 (small server)
    • Results for 8.5.4 vs 8.7.3 and then for 8.5.4 vs 9.3.1
    • Results for overwriteandwait are here for 8.7.3 and for 9.3.1
      • 8.7.3 and 9.3.1 with CRS =2M get 30% to 40% less QPS than 8.5.4
      • 8.7.3 and 9.3.1 with CRS =96K get ~10% less QPS than 8.5.4
      • 8.7.3 and 9.3.1 with O_DIRECT get ~2% more QPS than 8.5.4
      • Setting CRS to be <= max_sectors_kb is good for perf but O_DIRECT is better
      • Average ead request size per iostat (see rareqsz here) is much larger with CRS =96K than =2M (84.5 vs 4.6)
  • C2D (medium server)
    • Results for 8.5.4 vs 8.7.3 and then for 8.5.4 vs 9.3.1
    • Results for overwriteandwait are here for 8.7.3 and for 9.3.1
      • 8.7.3 and 9.3.1 with CRS =2M get 4% to 7% less QPS than 8.5.4
      • 8.7.3 and 9.3.1 with CRS =512K get ~20% more QPS than 8.5.4
      • 8.7.3 and 9.3.1 with O_DIRECT get ~11% more QPS than 8.5.4
      • Setting CRS to be <= max_sectors_kb is good for perf and better than O_DIRECT
      • Average read request size per iostat (see rareqsz here) is similar with CRS =512K and 2M (185.9 vs 194.1)
  • BIG (big server)
    • Results for 8.5.4 vs 8.7.3 and then for 8.5.4 vs 9.3.1
    • Results for overwriteandwait are here for 8.7.3 and for 9.3.1
      • 8.7.3 and 9.3.1 with CRS =2M get ~28% less QPS than 8.5.4
      • 8.7.3 and 9.3.1 with CRS =96K get 4% to 7% less QPS than 8.5.4
      • 8.7.3 and 9.3.1 with O_DIRECT get 7% to 10% more QPS than 8.5.4
      • Setting CRS to be <= max_sectors_kb is good for perf but O_DIRECT is better
      • Average ead request size per iostat (see rareqsz here) is much larger with CRS =96K than =2M (61.2 vs 5.1)
Summary
  • Setting compaction_readahead_size to be <= max_sectors_kb helps on all servers
  • On the small and big server, performance with O_DIRECT was better than without.













Comments

Post a Comment

Popular posts from this blog

Fixing bug 109595 makes MySQL almost 4X faster on the Insert Benchmark

Image
MySQL 8.0.35 includes a fix for bug 109595  and with that fix the QPS is almost 4X larger on the read+write benchmark steps compared to MySQL 8.0.34. Thank you to MySQL for fixing this quickly. I reported the bug in January of 2023. I have been aware of the performance problem for years, but didn't spend time debugging it until this year. I assume this problem was limited to InnoDB because I did not see problems with MyRocks. The chart below shows relative QPS which is (QPS for MySQL 8.0.35) / (QPS for MySQL 8.0.34). When the value is larger than 1.0 then MySQL 8.0.35 is faster than 8.0.34. And below the relative QPS is almost 4.0 on the read+write benchmark steps: q100, q500, q1000. The Insert Benchmark is explained here .
Read more

Postgres versions 11, 12, 13, 14, 15, and 16 vs sysbench with a medium server

Image
This provides additional results for Postgres versions 11 through 16 vs Sysbench on a medium server. My previous post is here . The goal is to document how performance changes over time with a focus on CPU overheads. tl;dr Postgres avoids performance regressions from old to new versions. This is excellent. Postgres 16.0 is mostly faster than 11.21 on read-only benchmark steps Postgres 16.0 and 11.21 have similar throughput on read-write and write-only benchmark steps Builds I compiled Postgres 11.21, 12.16, 13.12, 14.9, 15.2, 15.3, 15.4 and 16.0. The builds are named  o3_native_lto  which is shorthand for using: -O3 -march=native -mtune=native -flto. Benchmark I used sysbench and my usage is  explained here . Postgres was configured to cache all tables. This benchmark used a c2-standard-30 server from GCP with 15 cores, hyperthreads disabled, 120G of RAM, Ubuntu 22.04 and 1.5TB of NVMe SSD with XFS (SW RAID 0 over 4 local devices). The benchmark is run with 8 clients and 4 table with 2
Read more

Postgres vs MySQL: the impact of CPU overhead on performance

Image
This post documents how MySQL and Postgres performance has changed over time using sysbench. It compares QPS for Postgres 11.21 vs MySQL 5.6.21 and then Postgres 16.0 vs MySQL 8.0.34. The focus here is on CPU overhead and for that a workload that is low-concurrency, CPU-bound and cached is sufficient. However a full comparison would also include results for workloads with more concurrency and workloads that are IO-bound. The benchmark was explained in a series of posts - see  part 1 ,  part 2 ,  part 3 ,  part 4  and part 5  and the results here are from my small server. Usually I avoid direct comparisons between Postgres and MySQL but I am doing this because I care about the future of MySQL and the CPU regressions in it over time have become a problem. tl;dr Postgres 11.21 and MySQL 5.6.21 have similar QPS on most sysbench tests Postgres 16.0 gets much more QPS than MySQL 8.0.34 on most sysbench tests The problem for MySQL is new CPU overhead Results The spreadsheet with all of the nu
Read more