Open VDBBench: a fair and open benchmark for vector databases
If you want to know which vector database is fastest, you probably end up at VectorDBBench. Most engineers check it, and it’s the benchmark AI assistants cite when you ask them. Zilliz, the company behind Milvus and Zilliz Cloud, maintains the harness and runs the leaderboard at zilliz.com/vdbbench-leaderboard. Zilliz products hold the top three places on it.
When we submitted a client for Infino last month, we went back through the repo history to see how results make it onto that board, and found serious integrity problems. Most of the merged clients have never had a result published. Zilliz’s own products get re-run on each new release, while competitors sit on versions more than a year old, some with their compression switched off. And the dates and versions that would show you this were removed from the leaderboard data. Read the full critique for the PRs and configs behind each point.
We wanted a board where every engine runs its current version with compression on, and every number says when it was measured. So we forked the harness and ran the engines ourselves. The results are live at infino.ai/open-vdbbench.
How Open VDBBench runs
Open VDBBench uses the same VectorDBBench cases, Cohere 1M and Cohere 10M at 768 dimensions, with our fork of the harness. Each engine gets its own Azure Standard_D16ads_v7 (16 vCPU, 64 GB, local NVMe), with the client on the same VM. We sweep each engine’s search parameter and rank it by its highest QPS at recall ≥ 0.90. If you care about higher recall, the board can re-pick every engine at 0.95 or 0.99.
Every row carries the date it was measured and the version the runner read from the engine itself. If an engine’s client supports a quantized index, we use it. Anything that runs gets published, whatever it scores, and every row links to the result file it came from. The full rules are in METHODOLOGY.md.
Here’s how three engines ran on each board:
| zilliz board | open board | |
|---|---|---|
| Elasticsearch | 8.17, HNSW float32, rescore off | 9.5.1, int8_hnsw, rescore, oversample 2.0 |
| OpenSearch | 2.17, use_quant: false | 3.8.0, HNSW with fp16 scalar quantization |
| Milvus | 2.6.14, SQ4U+FP16 and SQ8 | 2.6.14, HNSW_SQ, SQ4U with FP16 refinement |
The first pass covers 19 engines at 1M and 13 at 10M. Infino runs under the same rules as everyone else, and it isn’t first at 1M. OceanBase is.
Add your database
If your database is missing or on an old version, send a pull request to infino-ai/open-vdbbench. It needs an entry in harness/matrix.json pinned to an exact image tag, plus bring-up scripts if the database is new. A version bump only changes the tag.
We’ll merge every new database within a month of its pull request. Every month, we run each new and updated database at 1M and 10M on the reference machine and put the results on the board.