I have been reporting on low-concurrency performance regressions in MySQL for a few years and recently published many reports to compare MySQL 5.6.35, 5.7.17 and 8.0.1 using Intel NUC servers. This is tracked by bug 86215. My summary of the recent tests is:
- The problem is not that there is a CPU regression from MySQL 5.6 to 5.7 to 8 as that is expected. The problem is that the regression is too large. In the worst case, I have MySQL 5.6 gets up to 2X more QPS than 5.7 and 8 using sysbench. In a typical case MySQL 5.6 gets 1.2X to 1.3X more QPS than 5.7 and 8.
- Most of the slowdown is from MySQL 5.6 to 5.7 and less of the problem is from 5.7 to 8. I think this is good news. This is based on my results from the i5 NUC.
- My team is committed to making this better. I hope that upstream is too. One day the big Percona Live presentation from upstream will include benchmark results for MySQL at low concurrency in addition to the results we always get for extremely high concurrency.
- If you publish results from tests for N configurations, you will always get a request for testing one more configuration. So keep N small and save energy for the followup. But my hope is that we learn something from the tests that I did, rather than ask for more tests. A more clever person would run tests for N configurations, initially share results for N-2 and then when asked for a few more configs wait a day and share the final two. Alas, I am not clever.
Background reading
My recent performance reports are here:
My recent performance reports are here:
- May 2017 - Sysbench, in-memory & Intel NUC
- May 2017 - Sysbench, IO-bound & Intel NUC for MySQL 5.6, 5.7 & 8
- June 2017 - Insert benchmark, in-memory & Intel NUC for MySQL 5.6, 5.7 and 8
- June 2017 - Insert benchmark, IO-bound & Intel NUC for MySQL 5.6, 5.7 & 8
- June 2017 - Linkbench, in-memory & Intel NUC for MySQL 5.6, 5.7 & 8
- June 2017 - Linkbench, IO-bound & Intel NUC for MySQL 5.6, 5.7 & 8
- June 2017 - Sysbench for MySQL 5.0, 5.1, 5.5, 5.6, 5.7 and 8
- June 2017 - Impact of perf schema on sysbench at low concurrency
- June 2017 - Linux perf and the CPU regression in MySQL 5.7
- June 2017 - MyISAM, small servers and sysbench at low concurrency
- June 2017 - One more time with sysbench, a small server & MySQL 5.7, 5.7 and 8.0
- July 2017 - I overstated the CPU regression in MySQL 5.7
I don't know the QA process for upstream -- for correctness and performance. The source is open, the development process is not. An in-memory sysbench workload would be sufficient for catching the CPU regressions that I have been reporting. With modern sysbench you can write new tests in Lua and we have been using that to spot regressions that we add by mistake to MyRocks.
ReplyDelete