Small Datum: Evaluating vector indexes in MariaDB and pgvector: part 2

This post has results from the ann-benchmarks with the fashion-mnist-784-euclidean dataset for MariaDB and Postgres (pgvector) with concurrent queries (--batch). My previous post has results when not using concurrent queries. This work was done by Small Datum LLC and sponsored by the MariaDB Corporation.

I compare MariaDB with pgvector because I respect the work that the Postgres community has done to support vector search. And I am happy to report that MariaDB has also done a great job on this. While I don't know the full story of the development effort, this feature came from the MariaDB Foundation and the community and it is wonderful to see that collaboration.

Performance for MariaDB is excellent

peak QPS for a given recall target is much better than pgvector
time to create the index to achieve a given recall target is almost always much better than pgvector

Benchmark

Part 1 has all of the details.

I am using the ann-benchmarks via my fork of a fork of a fork at this commit. I added --batch to the command lines listed in part 1 to repeat the benchmark using concurrent queries. Note that parallel index create was disabled for Postgres and isn't (yet) supported by MariaDB.

With --batch there is one concurrent query per CPU core and my server has 8 cores.

Files related to these tests are archived here.

Results: QPS vs recall graphs

The recall vs QPS graph is created by: python3 plot.py --dataset fashion-mnist-784-euclidean --batch. This chart is very similar to the chart in part 1. One difference is that the peak QPS for MariaDB and Postgres are ~6000/s and ~4000/s there vs ~22000/s and ~15000/s here.

The results below for MariaDB are excellent. It gets more QPS than pgvector at a given recall target.

Results: create index

I am still trying to figure out how to present this data. All of the numbers are here for the time to create an index and the size of the index. The database configs for Postgres and MariaDB are shared above, and parallel index create is disabled by the config for Postgres (and not supported yet by MariaDB). The summary is:

Indexes have a similar size with MariaDB and Postgres with halfvec. The Postgres index without halfvec is about 2X larger.
Time to create the index for Postgres is similar with and without halfvec
Time to create the index for MariaDB is less than for pgvector. Perhaps the best way to compare this is the time to create the index for a similar point on the QPS/recall graph (see the last section of this blog post)

Results: best QPS for a given recall

Many benchmark results are marketed via peak performance (max throughput or min response time) but these are usually constrained optimization problems -- determine peak performance that satisfies some SLA. And the SLA might be response time or efficiency (cost).

With ann-benchmarks the constraint might be recall where you seek the best QPS that satisfies a recall target. Below I share the best QPS for a given recall target along with the configuration parameters (M, ef_construction, ef_search) at which that occurs for each of the algorithms (MariaDB, pgvector with float32, pgvector with float16).

For all cases below except the first (recall = 1.000) the time to create the index is less with MariaDB.

For all cases below the best QPS at a given recall target is much better, sometimes 3X better, with MariaDB than with pgvector.

Legend

* recall & QPS - results from the benchmark

* isecs - number of seconds to create the index for M and ef_cons (ef_cons)

Best QPS with recall = 1.000

recall QPS isecs algorithm

1.000 4727 115.9 PGVector(m=16, ef_cons=256, ef_search=120)

1.000 5479 98.6 PGVector_halfvec(m=16, ef_cons=192, ef_search=120)

1.000 13828 108.5 MariaDB(m=32, ef_search=10)

Best QPS with recall >= 0.99

recall QPS isecs algorithm

0.990 10704 71.2 PGVector(m=16, ef_cons=96, ef_search=20)

0.991 11377 90.1 PGVector_halfvec(m=16, ef_cons=256, ef_search=20)