Monday, July 13, 2020

Updates for the insert benchmark

I continue to run and improve the insert benchmark. Eventually I will even share results. I also have scripts that generate charts, graphs and tables all combined into an HTML-formatted report. This is an update to the overview of the insert benchmark and details on how I share performance results is here.

Per interval results

The first change is that the benchmark client displays per-second performance results including IPS, QPS and max response time for inserts and queries. This makes it easier to understand stalls.

This is output from a load and the legend for the columns is:
  • i_sec - number of seconds for this interval
  • t_sec - cumulative number of seconds
  • i_ips, i_qps - average insert and query rate for that interval
  • t_ips, t_qps - average insert and query rate from test start until now
  • max_i, max_q - max insert and query response time for that interval, in microseconds
  • t_ins, t_query - total inserts and queries from test start until now

i_sec   t_sec   i_ips   t_ips   i_qps   t_qps   max_i   max_q   t_ins   t_query
1.0     1.0     61833   61833   0       0       2919    0       61900   0
1.0     2.0     63625   62729   0       0       1983    0       125600  0
1.0     3.0     63829   63095   0       0       2080    0       189500  0

Test steps

The second change is in how I run the benchmark. As described in the overview, it used to be run as: create tables/collections with secondary indexes, load X million rows/documents, do a full scan of each secondary index, do read+write tests with rate-limited writers.

I have changed it since then. The new pattern is:
  • create tables/collections without secondary indexes
  • load some data (l.i0)
  • create secondary indexes (l.x)
  • load some more data (l.i1)
  • do read+write tests (q100.1, q100.2, q200.1, q200.2, ...)
The read+write tests use rate-limited writers and the tests are run for varying limits. The test starts with 100 insert/s per client (q100) and then 200, 400, 600, 800 and 1000. In some cases the DBMS is unable to sustain the target insert rates. Also, for each limit the test is run twice so the results might use the names q100.1, q100.2, q200.1, q200.2, ..., q1000.1, q1000.2.

No comments:

Post a Comment