09-26-2024 Conf 42 Kube Native: Unleashing the Potential of Cloud Native Open Source Vector Databases

09-26-2024 Conf 42 Kube Native: Unleashing the Potential of Cloud Native Open Source Vector Databases

https://www.conf42.com/kubenative2024

The quick and easy way to run a true cloud native open source vector database for RAG, Semantic Search, Image Search and many more use cases the cloud native way.

https://www.conf42.com/Kube_Native_2024_Tim_Spann_cloud_native_open_source_vector_databases

milvus
vector database
LLM
RAG
unstructured data

1. 1 | © Copyright 2024 Zilliz1
Unleashing the Potential of Cloud Native
Open Source Vector Databases
Tim Spann @ Zilliz

2. 2 | © Copyright 2024 Zilliz2 2| © Copyright 10/22/23 Zilliz 2| © Copyright 2024 Zilliz
Tim Spann
Principal Developer
Advocate, Zilliz
[email protected]
https://www.linkedin.com/in/timothyspann/
https://x.com/PaaSDev

3. 3 | © Copyright 2024 Zilliz3
https://milvus.io/milvus-demos/reverse-image-search
Show Me

4. 4 | © Copyright 2024 Zilliz4
https://zilliz-semantic-search-example.vercel.app/
Show Me Another Demo

5. 5 | © Copyright 2024 Zilliz5
Extracting Value from Unstructured Data
Example
•A company has 100,000s+ pages of
proprietary documentation to
enable their staff to service
customers.
Problem
•Searching can be slow, inefficient,
or lack context.
Solution
•Create internal chatbot with
ChatGPT and a vector database
enriched with company
documentation to provide direction
and support to employees and
customers.
https://osschat.io/chat

6. 6 | © Copyright 2024 Zilliz6
Unstructured Data is Everywhere
Unstructured data is any data that does not conform to a predefined data model.

By 2025, IDC estimates there will be 175 zettabytes of data globally (that's 175
with 21 zeros), with 80% of that data being unstructured. Currently, 90% of
unstructured data is never analyzed.

Images Videos and more!Text

7. 7 | © Copyright Zilliz7
…and cannot process increasingly growing unstructured data
Data Source: The Digitization of the World by IDC
20%
Other
newly generated data in 2025
will be unstructured data80%

8. The challenge of unstructured data
●Problem: Unstructured data comes in lots of forms, no easy
way to interact with it all
●Solution: Vector embeddings
●How: Neural networks e.g. embedding models

Vector
Databases

9. 9 | © Copyright Zilliz9
02
Overview of Vector Databases

10. Why a Vector Database?
•Vector database
•Advanced filtering (filtered vector search, chained
filters)
•Hybrid search (e.g. full text + dense vector)
•Durability (any write in a db is durable, a library
typically only supports snapshotting)
•Replication / High Availability
•Sharding
•Aggregations or faceted search
•Backups
•Lifecycle management (CRUD, Batch delete,
dropping whole indexes, reindexing)
•Multi-tenancy
•Vector search library
•High-performance vector search

•How do I support different applications?
•High query load
•High insertion/deletion
•Full precision/recall
•Accelerator support (GPU, FPGA)
•Billion-scale storage

Purpose-built to store, index and query vector embeddings from unstructured
data.

11. V
n, 1
…
…
…
1
2
3
4
5
Transform into
Vectors
Unstructured Data
Images
User Generated
Content
Video
Documents
Audio
Vector Embeddings
Perform
Approximate
Nearest Neighbor
Similarity Search
Perform Query
Get Results
Store in Vector Database
How Similarity Search Works

12. 1
2
2024
A vector database stores embedding vectors and allows for semantic
retrieval of various types of unstructured data.
Vector Database: Making Sense of Unstructured Data

13. 13 | © Copyright Zilliz13
Do you really need a Vector Database?
•50M100M vectors
•PostgreSQL, ElasticSearch, Big
Query, MongoDB, etc with
ANNS plug-ins
Existing Solutions Vector Databases
•Purpose-built for vectors top
support the requirements and
lifecycle of vectors
•Billion+ scale
•CRUD, real-time search,
top-k/range/hybrid search,
multi-modal, mulit-vector query,
distributed
•Semantic Search is core to your
business
ANN Libraries
•FAISS, ANNOY, HNSW
•Supports 1M vectors
•Good for prototyping
Vector Databases are purpose-built to handle
indexing, storing, and querying vector data.

14. 14 | © Copyright Zilliz14
03
A Quick Introduction to Milvus

15. 15 | © Copyright Zilliz15
About Milvus
Milvus is an open-source vector database for
GenAI projects. pip install on your laptop, plug into
popular AI dev tools, and push to production with
a single line of code.
29K
GitHub Stars
25M
Downloads
250
Contributors
2,600
Forks
Easy Setup
Pip-install to start coding in a notebook within seconds
Integration
Plug into OpenAI, Langchain, LlmaIndex, and many more
Reusable Code
Write once, and deploy with one line of code into the production
environment
Feature-rich
Dense & sparse embeddings, filtering, reranking and beyond

16. 16 | © Copyright Zilliz16
Milvus Features
Multi-Tenancy

Hardware-
Accelerated
Compute Support
Python, Java,
Golang, NodeJS

Milvus Lite, K8,
Zilliz Cloud, Docker

Scalable and Elastic
Architecture

Diverse Index
Support

Versatile Search
Capabilities

Tunable
Consistency

17. 17 | © Copyright Zilliz17
Technologies for various types of Use
cases
Compute Types


Designed for various
compute powers, such as
AVX512, Neon for SIMD,
quantization cache-aware
optimization and GPU


Leverage strengths of each
hardware type, ensuring
high-speed processing and
cost-effective scalability for
different application needs


Search Types


Support multiple types such
as top-K ANN, Range ANN,
sparse & dense,
multi-vector, grouping,
and metadata filtering

Enable query flexibility and
accuracy, allowing
developers to tailor their
information retrieval needs
Multi-tenancy


Enable multi-tenancy
through collection and
partition management



Allow for efficient resource
utilization and customizable
data segregation, ensuring
secure and isolated data
handling for each tenant
Index Types


Offer a wide range of 15
indexes support, including
popular ones like
Hierarchical Navigable
Small Worlds HNSW, PQ,
Binary, Sparse, DiskANN
and GPU index

Empower developers with
tailored search
optimizations, catering to
performance, accuracy and
cost needs

18. 18 | © Copyright Zilliz18
What is Milvus/Zilliz ideal for?
•Advanced filtering
•Hybrid search
•Multi-vector Search
•Durability and backups
•Replications/High Availability
•Sharding
•Aggregations
•Lifecycle management
•Multi-tenancy
•High query load
•High insertion/deletion
•Full precision/recall
•Accelerator support GPU,
FPGA
•Billion-scale storage

Purpose-built to store, index and query vector embeddings from unstructured data at scale.

19. 19 | © Copyright Zilliz19
Milvus: From Dev to Prod
AI Powered Search made easy
Milvus is an Open-Source Vector
Database to store, index, manage, and
use the massive number of embedding
vectors generated by deep neural
networks and LLMs.
contributors
285
stars
29K
downloads
50M
forks
2.8K

20. 202024
Higher Scalability

10B vectors
of 1536 dimensions
in a single Milvus/Zilliz Cloud
instance

100B vectors
in one of the largest deployment

21. Milvus: decoupling computation and storage

22. 222024
Indexes
Most of the vector index types supported by Milvus use approximate nearest neighbors search ANNS,
●HNSW: HNSW is a graph-based index and is best suited for scenarios that have a high demand for
search efficiency. There is also a GPU version GPU_CAGRA, thanks to Nvidiaʼs contribution.
●FLAT: FLAT is best suited for scenarios that seek perfectly accurate and exact search results on a small,
million-scale dataset. There is also a GPU version GPU_BRUTE_FORCE .
●IVF_FLAT: IVF_FLAT is a quantization-based index and is best suited for scenarios that seek an ideal
balance between accuracy and query speed. There is also a GPU version GPU_IVF_FLAT.
●IVF_SQ8: IVF_SQ8 is a quantization-based index and is best suited for scenarios that seek a significant
reduction on disk, CPU, and GPU memory consumption as these resources are very limited.
●IVF_PQ: IVF_PQ is a quantization-based index and is best suited for scenarios that seek high query
speed even at the cost of accuracy. There is also a GPU version GPU_IVF_PQ.

23. 232024
Indexes Continued.
●SCANN: SCANN is similar to IVF_PQ in terms of vector clustering and product quantization. What makes
them different lies in the implementation details of product quantization and the use of SIMD
Single-Instruction / Multi-data) for efficient calculation.
●DiskANN: Based on Vamana graphs, DiskANN powers efficient searches within large datasets.

24. 24 | © Copyright Zilliz24
Meta Storage
Root Query Data Index
Coordinator Service
Proxy
Proxy
etcd
Log Broker
SDK
Load Balancer
DDL/DCL
DML
NOTIFICATION
CONTROL SIGNAL
Object Storage
Minio / S3 / AzureBlob
Log Snapshot Delta File Index File
Worker Node
QUERY DATA DATA
Message
Storage
Access Layer
Query Node Data Node Index Node
Milvusʼ fully distributed architecture is
designed scalability and performance

25. 25 | © Copyright Zilliz25
High-level Overview of Milvusʼ Architecture

26. Milvus: decoupling computation and storage

27. 27 | © Copyright Zilliz27
pip install pymilvus
Milvus Lite

28. 28 | © Copyright Zilliz28
05
Building a local RAG application

29. 29 | © Copyright Zilliz29
Vector embeddings are something
computers can understand

30. 3
0
Retrieval-Augmented Generation (RAG)
2024
A technique that combines the
strength of retrieval-based and
generative models:
●Improve accuracy and relevance
●Eliminate hallucination
●Provide domain-specific
knowledge

31. 3
1
RAG : an economic perspective
2024
A business model that bridges public
data and private data
●Data sovereignty
●You can't and shouldn't give your
private data to others

32. 32 | © Copyright Zilliz32
Open Source
Deploy fully managed or “Bring Your
Own Cloudˮ BYOC
Commercial Offerings

Zilliz Cloud
Optimized Milvus with essential data and
security tools for a high-performing vector
search platform
VECTOR SEARCH
ENGINE
VECTORDB
BENCHMARK TOOL
VECTOR DATABASE
SEMANTIC CACHE
FOR LLM QUERIES
GPTCache
Product Portfolio
GUI for Milvus

33. 33 | © Copyright Zilliz33
Embeddings Models

34. 34 | © Copyright Zilliz34 | © Copyright Zilliz34
RESOURCES

35. 35 | © Copyright Zilliz35
Vector Database Resources
Give Milvus a Star!




Chat with me on Discord!
https://github.com/milvus-io/milvus

36. 36
Unstructured Data Meetup


https://www.meetup.com/unstructured-data-meetup-new-york/

This meetup is for people working in unstructured data. Speakers will come present about related topics
such as vector databases, LLMs, and managing data at scale. The intended audience of this group
includes roles like machine learning engineers, data scientists, data engineers, software engineers, and
PMs.
This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.

37. 37 | © Copyright Zilliz37
https://zilliz.com/learn/generative-ai

39. https://medium.com/@tspann/unstructured-street-data-in-new-york-8d3cde0a1e5b

40. https://medium.com/@tspann/not-every-field-is-just-text-numbers-or-vectors-976231e90e4d

41. https://medium.com/@tspann/shining-some-light-on-the-new-milvus-lite-5a0565eb5dd9

42. https://medium.com/@tspann/unstructured-data-processing-with-a-raspberry-pi-ai-kit-c959dd7fff47
Raspberry Pi AI Kit Hailo
Edge AI

43. 43 | © Copyright 2024 Zilliz43
43
This week in Milvus, Towhee, Attu, GPT
Cache, Gen AI, LLM, Apache NiFi, Apache
Flink, Apache Kafka, ML, AI, Apache Spark,
Apache Iceberg, Python, Java, Vector DB
and Open Source friends.
https://bit.ly/32dAJft
https://github.com/milvus-io/milvus

AIM Weekly by Tim Spann

44. 44 | © Copyright 2024 Zilliz44
milvus.io
github.com/milvus-io/
@milvusio
@paasDev


/in/timothyspann
Connect with me! Thank you!

45. 45 | © Copyright 2024 Zilliz45

46. 46 | © Copyright 2024 Zilliz46

47. 47 | © Copyright 2024 Zilliz47

48. 48 | © Copyright 2024 Zilliz48
Join us at our next meetup!
meetup.com/unstructured-data-meetup-
new-york/

49. 49 | © Copyright Zilliz49
T H A N K Y O U

50. 50 | © Copyright Zilliz50
05
What is Similarity Search?

51. 51 | © Copyright Zilliz51
Image from Nvidia
Vector Search Overview

52. 52 | © Copyright Zilliz52
Vector Similarity Measures: L2 Euclidean)
Queen = [0.3, 0.9]
King = [0.5, 0.7]
d(Queen, King) = √(0.3-0.5)
2
+ (0.9-0.7)
2

= √(0.2)
2
+ (0.2)
2

= √0.04 + 0.04
= √0.08 ≅ 0.28

53. 53 | © Copyright Zilliz53
Vector Similarity Measures: Inner Product IP
Queen = [0.3, 0.9]
King = [0.5, 0.7]
Queen · King = (0.3*0.5) + (0.9*0.7)
= 0.15 + 0.63 = 0.78

54. 54 | © Copyright Zilliz54
Queen = [0.3, 0.9]
King = [0.5, 0.7]
Vector Similarity Measures: Cosine
??????
cos(Queen, King) = (0.3*0.5)+(0.9*0.7)
√0.3
2
+0.9
2
* √0.5
2
+0.7
2

= 0.15+0.63 _
√0.9 * √0.74
= 0.78 _
√0.666
≅ 0.03

55. 55 | © Copyright Zilliz55 osschat.io

56. 56 | © Copyright Zilliz56