This post has results for RocksDB performance using db_bench on 8-core and 48-core servers. I previously shared results for RocksDB performance using gcc and clang and then for RocksDB on a small Arm server.
tl;dr
- RocksDB is boring, there are few performance regressions.
- There was a regression in write-heavy workloads with RocksDB 10.6.2. See bug 13996 for details. That has been fixed.
- I will repeat tests in a few weeks
Software
I used RocksDB versions 9.8 through 10.0.
I compiled each version clang version 18.3.1 with link-time optimization enabled (LTO). The build command line was:
flags=( DISABLE_WARNING_AS_ERROR=1 DEBUG_LEVEL=0 V=1 VERBOSE=1 )# for clang+LTOAR=llvm-ar-18 RANLIB=llvm-ranlib-18 CC=clang CXX=clang++ \make "${flags[@]}" static_lib db_bench
I used servers with 8 and 48 cores, both run Ubuntu 22.04:
- 8-core
- Ryzen 7 (AMD) CPU with 8 cores and 32G of RAM.
- storage is one NVMe SSD with discard enabled and ext-4
- benchmarks are run with 1 client, 20M KV pairs for byrx and 400M KV pairs for iobuf and iodir
- 48-core
- an ax162s from Hetzner with an AMD EPYC 9454P 48-Core Processor with SMT disabled, 128G of RAM
- storage is 2 SSDs with RAID 1 (3.8T each) and ext-4.
- benchmarks are run with 36 clients, 200M KV pairs for byrx and 2B KV pairs for iobuf and iodir
Benchmark
Overviews on how I use db_bench are here and here.
Most benchmark steps were run for 1800 seconds and all used the LRU block cache. I try to use Hyperclock on large servers but forgot that this time.
Tests were run for three workloads:
- byrx - database cached by RocksDB
- iobuf - database is larger than RAM and RocksDB used buffered IO
- iodir - database is larger than RAM and RocksDB used O_DIRECT
- fillseq
- load RocksDB in key order with 1 thread
- revrangeww, fwdrangeww
- do reverse or forward range queries with a rate-limited writer. Report performance for the range queries
- readww
- do point queries with a rate-limited writer. Report performance for the point queries.
- overwrite
- overwrite (via Put) random keys and wait for compaction to stop at test end
Relative QPS
Many of the tables below (inlined and via URL) show the relative QPS which is:
(QPS for my version / QPS for RocksDB 9.8)
The base version varies and is listed below. When the relative QPS is > 1.0 then my version is faster than RocksDB 9.8. When it is < 1.0 then there might be a performance regression or there might just be noise.
The spreadsheet with numbers and charts is here. Performance summaries are here.
Results: cached database (byrx)
From 1 client on the 8-core server
- Results are stable except for the overwrite test where there might be a regression, but I think that is noise after repeating this test 2 more times and the cause is that the base case (result from 9.8) was an outlier. I will revisit this.
From 36 clients on the 48-core server
- Results are stable
Results: IO-bound with buffered IO (iobuf)
From 1 client on the 8-core server
- Results are stable except for the overwrite test where there might be a large improvement. But I wonder if this is from noise in the result for the base case from RocksDB 9.8, just as there might be noice in the cached (byrx) results.
- The regression in fillseq with 10.6.2 is from bug 13996
- Results are stable except for the overwrite test where there might be a large improvement. But I wonder if this is from noise in the result for the base case from RocksDB 9.8, just as there might be noice in the cached (byrx) results.
- The regression in fillseq with 10.6.2 is from bug 13996
From 1 client on the 8-core server
- Results are stable
- The regression in fillseq with 10.6.2 is from bug 13996
From 36 clients on the 48-core server
- Results are stable
- The regression in fillseq with 10.6.2 is from bug 13996
No comments:
Post a Comment