Scaling Vector Search: How Milvus Handles Billions+

Introduction to Vector DB and Vector Search and how Milvus does it at Billions+ Scale

1. 1 | © Copyright 8/16/23 Zilliz1 | © Copyright 8/16/23 Zilliz
Stephen Batifol | Zilliz

Unstructured Data Meetup, July 16th
Scaling Vector Search: How
Milvus Handles Billions+

2. 2 | © Copyright 8/16/23 Zilliz2 | © Copyright 8/16/23 Zilliz
Stephen Batifol
Developer Advocate, Zilliz/ Milvus
[email protected]
linkedin.com/in/stephen-batifol/
@stephenbtl
Speaker

3. 3 | © Copyright 8/16/23 Zilliz3 | © Copyright 8/16/23 Zilliz
27K+
GitHub
Stars
25M+
Downloads
250+
Contributors
2,600
+Forks
Milvus is an open-source vector database for GenAI projects. pip install on your
laptop, plug into popular AI dev tools, and push to production with a single line of
code.
Easy Setup

pip install
pymilvus to start
coding in a notebook
within seconds.
Reusable Code

Write once, and
deploy with one line
of code into the
production
environment
Integration

Plug into OpenAI,
Langchain,
LlamaIndex, and
many more
Feature-rich

Dense & sparse
embeddings,
filtering, reranking
and beyond

4. 4 | © Copyright 8/16/23 Zilliz4 | © Copyright 8/16/23 Zilliz
Seamless integration with all popular AI toolkits

5. 5 | © Copyright 8/16/23 Zilliz5 | © Copyright 8/16/23 Zilliz
Well-connected in LLM infrastructure to enable RAG
use cases
Framework
Hardware
Infrastructure
Embedding Models LLMs
Software Infrastructure
Vector Database

6. 6 | © Copyright 8/16/23 Zilliz6 | © Copyright 8/16/23 Zilliz
Retrieval Augmented
Generation (RAG)
Expand LLMs' knowledge by
incorporating external data sources
into LLMs and your AI applications.
Match user behavior or content
features with other similar ones to
make effective recommendations.
Recommender System
Search for semantically similar
texts across vast amounts of
natural language documents.
Text/ Semantic Search
Image Similarity Search
Identify and search for visually
similar images or objects from a
vast collection of image libraries.
Video Similarity Search
Search for similar videos, scenes,
or objects from extensive
collections of video libraries.
Audio Similarity Search
Find similar audios in large datasets
for tasks like genre classification or
speech recognition
Molecular Similarity Search
Search for similar substructures,
superstructures, and other
structures for a specific molecule.
Anomaly Detection
Detect data points, events, and
observations that deviate
significantly from the usual pattern
Multimodal Similarity Search
Search over multiple types of data
simultaneously, e.g. text and
images
Common AI Use Cases

7. 7 | © Copyright 8/16/23 Zilliz7 | © Copyright 8/16/23 Zilliz 7| © Copyright 8/16/23 Zilliz7| © Copyright 8/16/23 Zilliz
01
Introduction to Vector DB
and Vector Search

8. 8 | © Copyright 8/16/23 Zilliz8 | © Copyright 8/16/23 Zilliz
Traditional database was built upon exact search

9. 9 | © Copyright 8/16/23 Zilliz9 | © Copyright 8/16/23 Zilliz
…which misses context, semantic meaning, and user intent





VS.
Apple





VS.
Rising dough





VS.
Change car tire
Rising Dough
Proofing Bread
✔
❌

10. 10 | © Copyright 8/16/23 Zilliz10 | © Copyright 8/16/23 Zilliz
…and cannot process increasingly growing unstructured data
*Data Source: The Digitization of the World by IDC
20%
Other
newly generated data in 2025
will be unstructured data80%

11. 11 | © Copyright 8/16/23 Zilliz11 | © Copyright 8/16/23 Zilliz
Vector
Databases
Where do Vectors Come From?

12. 12 | © Copyright 8/16/23 Zilliz12 | © Copyright 8/16/23 Zilliz
Embeddings Models

13. 13 | © Copyright 8/16/23 Zilliz13 | © Copyright 8/16/23 Zilliz
Vector Embedding

14. 14 | © Copyright 8/16/23 Zilliz14 | © Copyright 8/16/23 Zilliz
Vector Space

15. 15 | © Copyright 8/16/23 Zilliz15 | © Copyright 8/16/23 Zilliz 15| © Copyright 8/16/23 Zilliz15| © Copyright 8/16/23 Zilliz
02
How do Vector Databases
Work?

16. 16 | © Copyright 8/16/23 Zilliz16 | © Copyright 8/16/23 Zilliz
How Similarity Search Works
V
n, 1
…
…
…
1
2
34
5
Transform into
Vectors
Unstructured Data
Images
User Generated
Content
Video
Documents
Audio
Vector Embeddings
Perform Approximate
Nearest Neighbor
Similarity Search
Perform Query
Get Results
Store in Vector Database

17. 17 | © Copyright 8/16/23 Zilliz17 | © Copyright 8/16/23 Zilliz 17| © Copyright 8/16/23 Zilliz17| © Copyright 8/16/23 Zilliz
02
How does Milvus do it at
Billions+ Scale?

18. 18 | © Copyright 8/16/23 Zilliz18 | © Copyright 8/16/23 Zilliz
Design Principles
•Disaggregate storage and computation
•Fully depends on mature storage
systems
•Micro Service - scale by functionality
•Separate Streaming and historical data
•Pluggable engine, storage and index
•Log As data

19. 19 | © Copyright 8/16/23 Zilliz19 | © Copyright 8/16/23 Zilliz
Meta Storage
Root Query Data Index
Coordinator Service
Proxy
Proxy
etcd
Log Broker
SDK
Load Balancer
DDL/DCL
DML
NOTIFICATION
CONTROL SIGNAL
Object Storage
Minio / S3 / AzureBlob
Log Snapshot Delta File Index File
Worker Node
QUERY DATA DATA
Message Storage
VECTOR
DATABASE
Access Layer
Query Node Data Node Index Node
Fully Distributed Architecture

20. 20 | © Copyright 8/16/23 Zilliz20 | © Copyright 8/16/23 Zilliz
Each shard is managed by a supervisor
(shard leader). This supervisor is
responsible for:
•Adding new information to the shard.
•Regularly storing the data in a safe
place (object storage).
•Serving the latest information for
search requests.
•Forwarding historical data requests to
other cabinets (query nodes) if
needed.

Milvus Data Layout - Shard

21. 21 | © Copyright 8/16/23 Zilliz21 | © Copyright 8/16/23 Zilliz
Growing Segment:
•In-memory segment replaying data
from the Log Broker.
•Uses a FLAT index to ensure data is
fresh and appendable.
Sealed Segment:
•Immutable segment using
alternative indexing methods for
efficiency.
Milvus Data Layout - Segments

22. 22 | © Copyright 8/16/23 Zilliz22 | © Copyright 8/16/23 Zilliz
Behind the Scenes: How Data Gets Added and
Accessed
•Sharding: Large datasets are
divided into smaller,
manageable sections called
shards. Each shard is handled
by a dedicated datanode.
•Write-Ahead Log (WAL):
When you add new data, a
proxy service writes it to a
temporary log called a WAL
(e.g., Kafka, Pulsar). Think of it
as a to-do list for the
datanodes.

23. 23 | © Copyright 8/16/23 Zilliz23 | © Copyright 8/16/23 Zilliz
Behind the Scenes: How Data Gets Added and
Accessed
•Datanodes subscribe to the WAL
and:
•Add new data to their assigned shard.
•Remove outdated data (if needed)
•Flush accumulated data to permanent
storage.
•Query Nodes also subscribe to the
WAL but focus on:
•Creating and managing Segments
within each shard for fast searching.
•Ensuring searches access the latest
information.

24. 24 | © Copyright 8/16/23 Zilliz24 | © Copyright 8/16/23 Zilliz
Index Building
To avoid frequent index building
for data updates.

A collection in Milvus is divided
further into segments, each with
its own index.

25. 25 | © Copyright 8/16/23 Zilliz25 | © Copyright 8/16/23 Zilliz
Data query refers to:
•Searching a specified
collection for k number of
vectors nearest to a target
vector or for all vectors within
a specified distance range to
the vector.
Data query

26. 26 | © Copyright 8/16/23 Zilliz26 | © Copyright 8/16/23 Zilliz
Some customers with Millions/ Billions Scale

27. 27 | © Copyright 8/16/23 Zilliz27 | © Copyright 8/16/23 Zilliz | © Copyright 8/16/23 Zilliz 27
Demo!

28. 28 | © Copyright 8/16/23 Zilliz28 | © Copyright 8/16/23 Zilliz 28| © Copyright 8/16/23 Zilliz28| © Copyright 8/16/23 Zilliz
09
RAG in action with Milvus
Lite

29. 29 | © Copyright 8/16/23 Zilliz29 | © Copyright 8/16/23 Zilliz
milvus.io
github.com/milvus-io/
@milvusio
@stephenbtl


/in/stephen-batifol
Thank you

30. 30 | © Copyright 8/16/23 Zilliz30 | © Copyright 8/16/23 Zilliz