How to win at IO-bound benchmarks

I spend time evaluating database performance on IO-bound workloads and have learned a few things that can help you to get much better results on benchmarks than will occur in production. But this is really about benchmarketing.

The tips are:

• Load the database in key order to avoid a fragmented B-Tree index or a randomized LSM tree. If there is a secondary index then create it after the load to avoid fragmentation there. This makes the database smaller and can make searches faster.
• Only use 10% of the storage device. This reduces the average seek latency for disks. This also reduces the overhead from flash GC because there is less live data to copy out during erase block cleaning. This also makes it less likely that erase block cleaning will be needed during the test. Erase-block cleaning can also be avoided by starting with a freshly setup or long-idle SSD.
• Don't run tests for too long. This reduces the chance that a B-Tree index will become fragmented, that an LSM will get slower from running compaction and that erase block cleaning will be started for an SSD. 
• Use a fixed number of user threads. When there is a stall there will be at most N(threads) operations that are stalled. In the real world the request arrival rate is usually steady and a stall will create a huge convoy of requests. If you are measuring p99 latency then a fixed number of threads will allow you to understate the impact from stalls. This is called coordinated omission and YCSB has been updated to support a steady arrival rate. Don't use that feature. I think fio will also be updated. I still need to fix innosim.
• Use a small number of threads to measure response time and a large number of threads to measure throughput. Make sure that writes and erase block cleaning are not in progress when measure read performance.

For more tips specific to LevelDB and RocksDB see an old post by me.

One other tip. If you are using a benchmark with more than 2B rows/keys/documents make sure that the random number generator supports that. It might not in LevelDB.