Zilliz, the creators of Milvus, have significantly upgraded their VectorDBBench tool, adding a crucial new dimension to how vector databases are measured. Previously, benchmarks mainly evaluated speed, such as the maximum number of queries a system could process per second. However, in real-world scenarios, engineers need more than just performance metrics. The updated tool now allows users to gauge the financial implications of achieving specific performance targets in production settings.
Historically, many benchmarks focused on showcasing peak performance, often under ideal conditions. The latest iteration of VectorDBBench introduces four cloud-centric test cases to address the missing cost component. These tests examine how quickly new data is available for search, the impact of data filters and payload size on performance, and the system's ability to manage multiple users and long periods of inactivity without a drop in efficiency.
The new cost-based evaluations are grouped into four key areas. First, they analyze the time it takes for newly added data to become searchable and the associated loading costs. Second, they assess how varying data size and filters affect search performance, including QPS, latency, and recall. Third, they evaluate the system's capacity to handle simultaneous user activity across many tenants, akin to SaaS environments. Lastly, they measure the delay when a query is run after an extended period of system inactivity.
The newly introduced Cost Leaderboard offers detailed insights into these evaluations. It breaks down operating costs at various query levels. For workloads with low and unpredictable query volumes, serverless pricing models can be more efficient. However, for sustained and high-volume traffic typical of production, flat-rate pricing may prove more cost-effective. The leaderboard is designed to be interactive, allowing users to apply their own data and scenarios to gain tailored results.
During initial testing, Zilliz Cloud demonstrated strong performance. It allowed newly inserted data to be searchable immediately at a minimal bulk-load cost. The system maintained consistent low latency even after periods of inactivity and became increasingly cost-effective as query volume increased. Pinecone and Turbopuffer, two other well-known platforms, showed different performance traits under similar conditions.
The primary aim of VectorDBBench is not to rank systems but to empower users to make informed decisions based on comprehensive data. The open-source nature of the tool enables anyone to replicate the tests, ensuring transparency and customization. The tool is compatible with over 30 vector databases and search systems. Users can also share their findings via GitHub or the Milvus Discord community, fostering a collaborative approach to benchmarking and evaluation.
VectorDBBench is freely available on GitHub, where users can re-run the tests or benchmark their own systems. The VDBBench blog provides additional details and updates for those interested in learning more about the benchmark and its features.
Zilliz is a leading provider of AI data infrastructure solutions and the creator of Milvus, the most widely adopted open-source vector database. With over 45,000 GitHub stars and more than 100 million Docker pulls, Milvus has earned a strong following among developers and organizations.
Milvus is a lake-native vector database specifically designed for handling 100-billion-scale vector search workloads. It offers a balance of high throughput and low latency, making it ideal for large-scale applications.
For further information about Zilliz and its products, visit Zilliz.com. Contact details for reaching Molly Chen can be found on the website.