Confluent Current 2024 - Multimodal Embeddings

Frank's talk on multimodal embeddings

1. Multimodal Embeddings
Frank Liu | Head of AI & ML, Zilliz

2. Frank Liu
Head of AI & ML | Zilliz

3. A Quick Refresher
Beyond Text and Image Embeddings
Multimodal Embeddings
Demo Time!
Agenda

01
02
03
04

4. About Milvus
Milvus is an open-source vector database
for GenAI projects. pip install on your laptop,
plug into popular AI dev tools, and push to
production with a single line of code.
29K
GitHub Stars
25M
Downloads
250
Contributors
2,600
Forks
Easy Setup
Pip-install to start coding in a notebook within seconds
Integration
Plug into OpenAI, Langchain, LlmaIndex, and many more
Reusable Code
Write once, and deploy with one line of code into the production
environment
Feature-rich
Dense & sparse embeddings, filtering, reranking and beyond

5. A Quick
Refresher

6. Vectors Unlock Unstructured Data
Knowledge Base
Documents)
Embedding Models Vectors Vector Databases

7. Embeddings models workhorses of AI apps

8. Beyond Text and Image
Embeddings

9. Back in the day…

10. …feature vectors were handcrafted


SIFT
TFIDF
Harris Corner Detector

11. Circa 2012, convnets became immensely popular




Source: CS230 notes

12. And many people discovered the power of vectors

13. RNNs were the OG language model







Source: CS230 notes

14. Vectors are for more than just text and images

15. Vectors are for more than just text and images

16. Vectors are for more than just text and images

17. Vectors are for more than just text and images

18. Vectors are for more than just text and images

19. Multimodal
Embeddings

20. Visual + language embeddings (CLIP-like)













Source: CLIP blog

21. One embedding space, six modalities (ImageBind)





Source: Girdhar, et al.

22. LLMs are becoming natively multimodal…







Source: Llama Team, AI @ Meta

23. … and the best embedding models will too









Original: Fuyu-8B blog

24. Demo Time

25. Vanilla RAG is no longer enough…











Your Documents
Embedding Model
Milvus
Search
Gen AI Model
Reliable Answers
What is the default
AUTOINDEX distance
metric in Milvus Client?
The default
AUTOINDEX distance
metric in Milvus Client is
L2.
Question
Question + Context

26. … we need multimodal RAG











Multimodal Model
Milvus
Question
Question + Context
Search
Gen AI Model
Reliable Answers
What kind of music did
they play in the
pre-show?
The musician played
improvised electronic
music.

27. Connect with me









LinkedIn Twitter Demo Notebook