This has results for sysbench with MyRocks, MySQL/InnoDB and Postgres using public cloud servers. The goal is to document how relative performance changes from single-threaded to high-concurrency workloads.
By relative performance I mean that I present performance as a ratio: throughput on an Arm server / throughput on an x86 server. By throughput I mean QPS for all of the workloads except for scan, which does a full scan, and throughput for scan is millions of rows read/second.
I compare c7g vs c6i instance types from AWS and then t2a vs c2 server types from GCP. I am comparing absolute performance rather than price performance. For AWS the c7g list price is a bit lower than c6i while for GCP the t2a and c2 list prices are about the same.
Regardless, my goal is only to understand how relative throughput changes with concurrency. Of course, I also look for perf bugs that might be fixable.
tl;dr:
- Results for MyRocks have the most variance and for Postgres have the least variance
- x86 usually gets more throughput than Arm although the advantage is smaller to non-existent when the CPU is fully/over-subscribed. A common throughput ratio for Arm was between 0.7 and 0.9 meaning that Arm got between 70% and 90% of the throughput (QPS) compared to an x86 in the same price range. Alas, there is much detail and nuance. In some cases the Arm server got the same or better throughput. Relatively, the Arm does much better when the CPU is saturated (number of concurrent requests and background threads >= number of Arm cores) because it has more cores but the per-core perf on the x86 is better.
- InnoDB does the worst on Arm relative to x86 with AWS (c7g vs c6i) as the workload & server size are scaled up for write-heavy workloads. MyRocks is similar but not as bad in this respect.
- The advantage for x86 vs Arm is smaller on the write heavy benchmark steps, especially with the workloads where the CPU is fully/over-subscribed
- The advantage for x86 vs Arm is larger when comparing t2a vs c2 (GCP) than c7g vs c6i (AWS)
Note that I disable hyperthreading on the x86 servers because from past experience the use of hyperthreading hurts DBMS performance. I don't reevaluate this (hyperthreading enabled vs disabled) for every benchmark because that takes more time but I will revisit the topic. The impact of disabling hyperthreading is that the Arm server has 2X more cores than the comparable x86. For example, the 16xlarge instances on AWS (c7g is Arm, c6i is x86) have 64 vcpu and there are 64 cores on Arm, 64 cores on x86 with hyperthread enabled and 32 cores on x86 with hyperthread disabled.
Benchmark
I use upstream sysbench with a few extra Lua scripts from here driven by shell scripts from here. Read the Lua scripts to see what SQL statements are used. For example, the scan query does a full scan with a WHERE clause that filters all rows on the DBMS and returns 0 rows (and read this fun bug to learn what the MongoDB planner does for such queries).
There are 42 Lua scripts for which I provide results. Each of them represent a (micro)benchmark step that was run for 10 minutes. I place them into three groups -- point, range, write -- based on the common operation done for each where point does point queries, range does range queries and write does insert/update/delete.
The benchmark is invoked by this script (r.sh) which then invoked cmp_pg.sh, cmp_in.sh and cmp_rx.sh. All scripts end up calling all_small.sh to run sysbench for a sequence of Lua scripts. Each Lua script was run for 10 minutes using 8 tables with 32 clients (threads, connections). Note that the scripts linked above (r.sh, cmp_??.sh) are similar to the scripts I used but I had to edit them for different levels of concurrency and server size.
In all cases there were 20M rows/table but the number of tables increased with the server size. The benchmark was run twice -- once with most queries done against a primary key index and then again with a secondary index used for most queries (sysbench --secondary).
I used Postgres 15.1, MySQL 8.0.31 for InnoDB and FB MySQL 8.0.28 for MyRocks.
The test database always fit in the database buffer pool (or RocksDB block cache).
For each server size the benchmark is repeated first with a number of clients so that the CPU is under-subscribed (WRT to number of x86 cores) and second with a number of clients so that the CPU is full subscribed for queries and possibly over-subscribed for writes. For example on the 2xlarge AWS instances both have 8 vcpu, the c6i has 4 cores (hyperthreading disabled) and the c7g has 8 cores. The test was run with 1 and 4 clients. With 4 cliens the x86 CPU is fully subscribed for queries and over-subscribed for writes because background threads must also run for b-tree checkpoint or LSM compaction.
The impact of under-subscribed vs fully/over-subscribed is visible on the graphs. The difference between x86 and Arm is larger on the read heavy microbenchmarks because the x86 instances have better per-core performance than Arm. On the write-heavy tests the difference is smaller.
Hardware
Tests were repeated on AWS and GCP servers. For AWS I used c7g (Arm) and c6i (x86). For GCP I used t2a (Arm) and c2 (x86). The c7g and c6i servers are closer in performance than the t2a and c2. Benchmarks were repeated for several instance/server sizes.
All servers used Ubuntu 22.04. I did not enable huge pages for Postgres or InnoDB.
All servers used Ubuntu 22.04. I did not enable huge pages for Postgres or InnoDB.
AWS
- 2xlarge
- Has 8 vcpu and 16G RAM. There are 8 cores on c7g and 4 on c6i because I disabled hyperthreading. Both servers used EBS for the database (io2, 256G, 10K IOPs).
- Sysbench was run with 1 table and 20M rows for 1 and 4 clients.
- The database config files are here for MyRocks, for Postgres and for InnoDB.
- 8xlarge
- Has 32 vcpu and 64G RAM. There are 32 cores on c7g and 16 on c6i because I disabled hyperthreading. Both servers used EBS for the database (io2, 1TB, 49K IOPs).
- Sysbench was run with 4 tables and 20M rows/table for 8 and 16 clients.
- The database config files are here for MyRocks, for Postgres and for InnoDB.
- 16xlarge
- Has 64 vcpu and 128G RAM. There are 64 cores on c7g and 32 on c6i because I disabled hyperthreading. Both servers used EBS for the database (io2, 2TB, 49K IOPs).
- Sysbench was run with 8 tables and 20M rows/table for 16 and 32 clients.
- The database config files are here for MyRocks, for Postgres and for InnoDB.
GCP
- t2a-standard-8, c2-standard-8
- Has 8 vcpu and 16G RAM (t2-standard-8, c2-standard-8). There are 8 cores on t2a and 4 on c2 because I disabled hyperthreading. Both have 32G RAM. The database was on SSD Persistent disk (400G).
- Sysbench was run with 1 table and 20M rows for 1 and 4 clients.
- The database config files are here for MyRocks, for Postgres and for InnoDB.
- t2a-standard-32, c2-standard-30
- The t2a instance has 128G RAM, 32 vcpu and 32 cores. The c2 instance has 120G RAM, 30 vcpu and 15 cores because I disabled hyperthreading. The database was on SSD Persistent disk (2TB).
- Sysbench was run with 4 tables and 20M rows/table for 8 and 15 clients.
- The database config files are here for MyRocks, for Postgres and for InnoDB.
Results
Each benchmark consists of 42 (micro)benchmark steps. Each benchmark step is a Lua script and most scripts just test one operation (point query, range query, insert, update, etc) thus it is best to think of them as microbenchmarks. The read-only and read-write benchmark steps are the traditional sysbench workload and consist of different SELECT statements that use range scans and sometimes do aggregation.
Below there are charts for each benchmark that show the relative performance for each benchmark step as: Arm throughput / x86 throughput. The Arm server is faster when this ratio is larger than 1. Alas, the ratios are almost always less than 1 because the x86 server was usually faster.
The x-axis starts at 0.4 or 0.5 rather than 0.0 to make it easier to see the differences. The spreadsheet is here for AWS and for GCP. It is easier to see detail on the spreadsheet than on the inline images below.
For each benchmark step there is a pair of results named pk0 and pk1. By pk1 I mean that most queries used the primary key index. By pk0 I mean that most queries used the secondary index (sysbench --secondary).
Result sections
Each graph below has 42 pairs of lines -- each pair has a result for pk0 and pk1. So I scanned all of them and attempted to draw some conclusions using eyeball stats rather than the real thing. The last section of this blog post has summary statistics (min, max, average, median, standard deviation) for each of the microbenchmark groups (point, range, writes).
On each graph the 42 microbenchmarks are in three groups from left to right (Point, Range, Write). The Point group is for point queries, the Range group is for range queries and the Write group is for writes.
In the sections that follow where I describe whether x86 does better than, similar to or worse than Arm the judgement is mostly based on the average values for the throughput ratio with help from the median values. Generally, if the ratio is between 0.95 and 1.05 I claim the throughput is similar.
Results: 8 vcpu, 1 client, AWS
The CPU was under-subscribed.
- MyRocks has the most variance
- For queries
- x86 is better than Arm for MyRocks and Postgres
- x86 is similar to Arm for InnoDB
- For writes
- x86 is better than Arm for MyRocks and Postgres
- x86 is worse than Arm for InnoDB
The graphs are also on the spreadsheet.
.png)
.png)
.png)
Results: 8 vcpu, 4 clients, AWS
The CPU is fully/over-subscribed during the write-heavy benchmark steps.
- MyRocks has the most variance
- For queries
- x86 is better than Arm
- For writes
- x86 is worse than Arm for Postgres and InnoDB
- x86 is similar to Arm for MyRocks
- For writes the advantage shifts towards Arm given the CPU is fully/over-subscribed
The graphs are also on the spreadsheet.
.png)
.png)
.png)
Results: 32 vcpu, 8 clients, AWS
The CPU was under-subscribed.
- MyRocks has the most variance
- For queries
- x86 is better than Arm except for MyRocks+pk1 where they are similar
- For writes
- x86 is better than Arm for MyRocks and InnoDB
- x86 is similar to Arm for Postgres
The graphs are also on the spreadsheet.
.png)
.png)
.png)
Results: 32 vcpu, 16 clients, AWS
The CPU is fully/over-subscribed during the write-heavy benchmark steps.
- MyRocks has the most variance
- Compared to the under-subscribed case above the advantage shifts towards Arm here
- For point queries
- x86 is better than Arm for Postgres and InnoDB
- x86 is worse than Arm for MyRocks with pk0 and similar with pk1
- For range queries
- x86 is better than Arm for Postgres and InnoDB
- x86 is similar to Arm for MyRocks with pk0 and better with pk1
- For writes
- x86 is worse than Arm for Postgres and InnoDB
- x86 is similar to Arm for MyRocks with pk0 and worse than Arm for pk1
The graphs are also on the spreadsheet.
.png)
.png)
.png)
Results: 64 vcpu, 16 clients, AWS
The CPU was under-subscribed.
- MyRocks has the most variance
- For point queries
- x86 is better than Arm except for MyRocks+pk0 where they are similar
- For range queries
- x86 is better than Arm
- For writes
- x86 is better than Arm for MyRocks and InnoDB.
- For InnoDB the x86 advantage increases as the instance size increases from 2xl to 8xl to 16xl here. This is true for both the under-subscribed and fully/over-subscribed workloads although the change appears to be worse in the under-subscribed CPU case.
- x86 is similar to Arm for Postgres
The graphs are also on the spreadsheet.
.png)
.png)
.png)
Results: 64 vcpu, 32 clients, AWS
The CPU is fully/over-subscribed during the write-heavy benchmark steps.
- MyRocks has the most variance
- For point queries
- x86 is better than Arm for Postgres and InnoDB
- x86 is similar to Arm for MyRocks with pk0 and worse with pk1
- For range queries
- x86 is better than Arm for Postgres and InnoDB
- x86 is similar to Arm for MyRocks with pk0 and worse with pk1
- For writes
- x86 is better than Arm for MyRocks and InnoDB
- x86 is worse than Arm for Postgres
The graphs are also on the spreadsheet.
.png)
.png)
.png)
Results: 8 vcpu, 1 client, GCP
The CPU was under-subscribed.
- For queries
- x86 is better than Arm
- For writes
- x86 is better than Arm
.png)
.png)
.png)
Results: 8 vcpu, 4 clients, GCP
The CPU is fully/over-subscribed during the write-heavy benchmark steps.
- For queries
- x86 is better than Arm
- For writes
- x86 is better than Arm
.png)
.png)
.png)
Results: 30/32 vcpu, 8 clients, GCP
The CPU was under-subscribed.
- For queries
- x86 is better than Arm
- For writes
- x86 is better than Arm
The graphs are also on the spreadsheet.
.png)
.png)
.png)
Results: 30/32 vcpu, 15 clients, GCP
The CPU is fully/over-subscribed during the write-heavy benchmark steps.
- For queries
- x86 is better than Arm
- For writes
- x86 is better than Arm
.png)
.png)
.png)
The summary statistics group the micro benchmarks into 3 groups:
- Point - benchmarks that do point queries
- Range - benchmarks that do range queries, including one that does a full range scan
- Write - benchmarks that do writes
The statistics are computed for the throughput ratio which is: (throughput for Arm / throughput for x86) and throughput is QPS for all tests except scan. For scan the throughput is millions of rows read/second. The throughput for Arm is better than x86 when the ratio is greater than 1.
For each table there are results for pk0 and pk1. The pk1 results are the normal way to run sysbench -- most queries use the PK index. The pk0 results are from sysbench --secondary so that most queries use a secondary index (which is created on the same columns as the PK index had been created).
Summary statistics: 8 vcpu, 1 client, AWS
For MyRocks:
For Postgres:
For InnoDB:
Summary statistics: 8 vcpu, 4 clients, AWS
For MyRocks:
For Postgres:
For InnoDB:
Summary statistics: 32 vcpu, 8 clients, AWS
For MyRocks:
For Postgres:
For InnoDB:
Summary statistics: 32 vcpu, 16 clients, AWS
For MyRocks:
For Postgres:
For InnoDB:
Summary statistics: 64 vcpu, 16 clients, AWS
For MyRocks:
For Postgres:
For InnoDB:
Summary statistics: 64 vcpu, 32 clients, AWS
For MyRocks:
For Postgres:
For InnoDB:
Summary statistics: 8 vcpu, 1 clients, GCP
For MyRocks:
For Postgres:
For InnoDB:
Summary statistics: 8 vcpu, 4 clients, GCP
For MyRocks:
For Postgres:
For InnoDB:
Summary statistics: 30/32 vcpu, 8 clients, GCP
For MyRocks:
For Postgres:
For InnoDB:
Summary statistics: 30/32 vcpu, 15 clients, GCP
For MyRocks:
For Postgres:
For InnoDB:
No comments:
Post a Comment