Exploring Multimodal Embeddings with Milvus

Explore how multimodal embeddings work with Milvus. We will see how you can explore a popular multimodal model - CLIP - on a popular dataset - CIFAR 10. You use CLIP to create the embeddings of the input data, Milvus to store the embeddings of the multimodal data (sometimes termed “multimodal embeddings”), and we will then explore the embeddings.

1. 1 | © Copyright 10/22/23 Zilliz1 | © Copyright 10/22/23 Zilliz
Stephen Batifol, Zilliz
Multimodal Embeddings with
Milvus

2. 2 | © Copyright 10/22/23 Zilliz2 | © Copyright 10/22/23 Zilliz
Stephen Batifol
Developer Advocate, Zilliz
[email protected]
https://www.linkedin.com/in/stephen-batifol/
https://twitter.com/stephenbtl
Speaker

3. 3 | © Copyright 10/22/23 Zilliz3 | © Copyright 10/22/23 Zilliz
Open Source
Deploy fully managed or “Bring Your
Own Cloud” (BYOC)
Commercial Offerings

Zilliz Cloud
Optimized Milvus with essential data and
security tools for a high-performing vector
search platform
VECTOR SEARCH
ENGINE
VECTORDB
BENCHMARK TOOL
VECTOR DATABASE
SEMANTIC CACHE
FOR LLM QUERIES
GPT-Cache
Product Portfolio
GUI for Milvus

4. 4 | © Copyright 10/22/23 Zilliz4 | © Copyright 10/22/23 Zilliz
Couple of Customers

5. 5 | © Copyright 10/22/23 Zilliz5 | © Copyright 10/22/23 Zilliz
Cloud
Service
Provider
Data Platform
GenAI Tooling
Chip
Manufacturer
Partner with Industry Leaders

6. 6 | © Copyright 10/22/23 Zilliz6 | © Copyright 10/22/23 Zilliz
01Where do Vectors Come From?
CONTENTS
03
04Demo!
02OpenAI CLIP
What is Multimodal?

7. 7 | © Copyright 2023 Zilliz7
Where do Vectors Come From?01

8. 8 | © Copyright 2023 Zilliz8
Vector
Databases
Unstructured Data + ML = Vector Magic

9. 9 | © Copyright 2023 Zilliz9
Embeddings Models

10. 10 | © Copyright 2023 Zilliz10
“A building with a small window”
Vector Embeddings
Embedding
Model
[-0.099,-0.028,-0.047,0.012,...,-0.011,...]

11. 11 | © Copyright 2023 Zilliz11
Vector Embeddings

12. 12 | © Copyright 2023 Zilliz12 12| © Copyright 9/25/23 Zilliz12| © Copyright 9/25/23 Zilliz
“The iPhone is a line of smartphones produced by
Apple”
Vector Embedding
Embedding
Model
[-0.099,-0.028,-0.047,0.012,...,-0.011]

13. 13 | © Copyright 2023 Zilliz13 | © Copyright 9/25/23 Zilliz 13
Milvus can handle
different data and not
only text

14. 14 | © Copyright 2023 Zilliz14
OpenAI CLIP02

15. 15 | © Copyright 2023 Zilliz15
CLIP

16. 16 | © Copyright 2023 Zilliz16
CLIP

17. 17 | © Copyright 2023 Zilliz17
CLIP Embeddings
CLIP

18. 18 | © Copyright 2023 Zilliz18
What is Multimodal?03

19. 19 | © Copyright 2023 Zilliz19
Multimodal Embeddings
Image Encoder
[-0.099,-0.028,-0.047,0
.012,...,-0.011]
Text Encoder
[-0.096,-0.026,-0.044,0
.012,...,-0.011]
Audio Encoder
[-0.093,-0.021,-0.047,0.
012,...,-0.010]
A dog smiling

20. 20 | © Copyright 2023 Zilliz20
Multimodal Embeddings

21. 21 | © Copyright 2023 Zilliz21
Search over everything
A dog smiling
Lorem
Ipsum

22. 22 | © Copyright 2023 Zilliz22 | © Copyright 9/25/23 Zilliz 22
Demo!
MultiModal RAG

23. 23 | © Copyright 8/16/23 Zilliz23 | © Copyright 8/16/23 Zilliz
Basic RAG Architecture

24. 24 | © Copyright 8/16/23 Zilliz24 | © Copyright 8/16/23 Zilliz
Give Milvus a Star! Chat with me on Discord!
Questions?
Github