The History of Embeddings & Multimodal Embeddings

Frank Liu will walk through the history of embeddings and how we got to the cool embedding models used today. He'll end with a demo on how multimodal RAG is used.

1. Frank Liu

Multimodal Embeddings

2. 2 | © Copyright Zilliz2
Speaker
Frank Liu
Head of AI & ML
[email protected]
https://www.linkedin.com/in/fzliu
https://www.twitter.com/frankzliu

3. 3 | © Copyright Zilliz3
A Quick
Refresher

4. 4 | © Copyright Zilliz4
Vectors unlock unstructured data
Knowledge Base
(Documents)
Embedding Models Vectors Vector Databases

5. 5 | © Copyright Zilliz5
Embeddings models workhorses of AI apps

6. 6 | © Copyright Zilliz6
History of
Embeddings

7. 7 | © Copyright Zilliz7
Back in the day…

8. 8 | © Copyright Zilliz8
…feature vectors were handcrafted
SIFT
TF-IDF
Harris Corner Detector

9. 9 | © Copyright Zilliz9
Circa 2012, convnets became immensely popular
Source: CS230 notes

10. 10 | © Copyright Zilliz10
And many people discovered the power of vectors

11. 11 | © Copyright Zilliz11
Recurrent neural networks
Source: CS230 notes

12. 12 | © Copyright Zilliz12
Modern
Embedding
Models

13. 13 | © Copyright Zilliz13
Attention Is All You Need?
Source: Illustrated Transformer

14. 14 | © Copyright Zilliz14
Sentence BERT outputs a single vector per sequence
Source: Reimers & GurevychSource: Devlin et al.

15. 15 | © Copyright Zilliz15
Vision transformer
Source: CS230 notes
Source: Dosovitskiy et al.

16. 16 | © Copyright Zilliz16
Other unstructured data modalities

17. 17 | © Copyright Zilliz17
Other unstructured data modalities

18. 18 | © Copyright Zilliz18
Other unstructured data modalities

19. 19 | © Copyright Zilliz19
Other unstructured data modalities

20. 20 | © Copyright Zilliz20
Other unstructured data modalities

21. 21 | © Copyright Zilliz21
Multimodal
Embeddings

22. 22 | © Copyright Zilliz22
Visual + language embeddings
Source: CLIP blog

23. 23 | © Copyright Zilliz23
ImageBind
Source: Girdhar, et al.

24. 24 | © Copyright Zilliz24
Bonus:
Multimodal
RAG

25. 25 | © Copyright Zilliz25
Vanilla RAG
Your
Documents
Embedding Model
Milvus
Question
Question + Context
Search
Gen AI Model
Reliable Answers
What is the default
AUTOINDEX distance
metric in Milvus
Client?
The default
AUTOINDEX distance
metric in Milvus
Client is L2.

26. 26 | © Copyright Zilliz26
Multimodal RAG
Multimodal Model
Milvus
Question
Question + Context
Search
Gen AI Model
Reliable Answers
What kind of music
did they play in the
pre-show?
The musician played
improvised electronic
music.