Zilliz Adds Cost-Aware Benchmarking to VDBBench, the Open-Source Vector Database Benchmark

Zilliz, a leading AI data infrastructure company and the creator of Milvus, recently announced a major update to VectorDBBench (VDBBench), an open-source, vendor-neutral benchmark for vector databases, adding cost as a first-class dimension alongside performance under production conditions — not just how fast a system performs in an idealized test.

Most benchmarks optimize for a single headline number: peak queries per second (QPS) on static, fully indexed data. But teams rarely choose a vector database on speed alone. They need to know what it costs to hit a target QPS, when newly written data actually becomes searchable, how filters and payload size change the query surface, how a system holds up across thousands of tenants, and how it responds on the first query after sitting idle. VDBBench now measures those dimensions directly.

“Most benchmarks answer one question — how fast can it go on ideal data,” said James Luan, VP of Engineering at Zilliz. “But teams buy on total cost and real production behavior. We built VDBBench as an open, reproducible benchmark so any team can see what a vector database will actually cost and how it will behave on their own workload — and decide for themselves.”

What’s New in VDBBench

Building on VDBBench 1.0, which moved benchmarking closer to production by testing under streaming ingestion, filtering, recall, and concurrency, the latest release adds cost as a first-class dimension through four new cloud-oriented test cases:

• Insert readiness and write cost — separating when a write is accepted, when it becomes searchable, and when it is fully indexed, alongside the cost of loading data.
• Payload-aware search — how response shape (IDs, metadata, or full vectors) and filters change QPS, latency, and recall.
• Multitenant search — sustained throughput across many tenants and namespaces, as in SaaS workloads.
• Cold-start latency — the first query against a collection that has gone idle, versus the warmed path.

The results feed a new Cost Leaderboard with a cost-performance (Pareto) view that models operating cost at target QPS levels, showing where usage-metered serverless pricing is most efficient at low, spiky volume and where provisioned, flat-rate capacity becomes more cost-effective for the sustained traffic typical of production workloads.

Built to Benchmark Any Vector Database

VDBBench is open-source and supports more than 30 vector databases and search systems. To show the new cases in action, the Cost Leaderboard launches with a sample evaluation of several widely used managed vector databases — Pinecone, Turbopuffer, and Zilliz Cloud — chosen to illustrate how differently products can behave on freshness, filtering, multitenancy, and cold starts. The comparison is a demonstration, not a ranking: because the benchmark is open, teams can reproduce every case, swap in any other candidate, and run it against their own production-like data.

In the sample run, for instance, Zilliz Cloud returned newly inserted data as searchable immediately at a single-digit-dollar bulk-load cost, held a near-flat cold-start latency profile, and grew more cost-effective as sustained query volume rose — an illustration of the performance-and-cost balance the new cases are built to reveal. A product tuned for a different workload could show different strengths, which is exactly why every case is open to reproduce.

“This isn’t about naming a winner,” Luan added. “Different databases are built for different workloads, and the right choice depends on freshness, filtering, tenancy, budget, and many other factors. The results aren’t ours to declare — they’re yours to reproduce.”

Write to us [⁠wasim.a@demandmediaagency.com] to learn more about our exclusive editorial packages and programmes.