The History of Embeddings & Multimodal Embeddings

chloewilliams62 274 views 26 slides Jul 17, 2024
Slide 1
Slide 1 of 26
Slide 1
1
Slide 2
2
Slide 3
3
Slide 4
4
Slide 5
5
Slide 6
6
Slide 7
7
Slide 8
8
Slide 9
9
Slide 10
10
Slide 11
11
Slide 12
12
Slide 13
13
Slide 14
14
Slide 15
15
Slide 16
16
Slide 17
17
Slide 18
18
Slide 19
19
Slide 20
20
Slide 21
21
Slide 22
22
Slide 23
23
Slide 24
24
Slide 25
25
Slide 26
26

About This Presentation

Frank Liu will walk through the history of embeddings and how we got to the cool embedding models used today. He'll end with a demo on how multimodal RAG is used.


Slide Content

Frank Liu

Multimodal Embeddings

2 | © Copyright Zilliz2
Speaker
Frank Liu
Head of AI & ML
[email protected]
https://www.linkedin.com/in/fzliu
https://www.twitter.com/frankzliu

3 | © Copyright Zilliz3
A Quick
Refresher

4 | © Copyright Zilliz4
Vectors unlock unstructured data
Knowledge Base
(Documents)
Embedding Models Vectors Vector Databases

5 | © Copyright Zilliz5
Embeddings models workhorses of AI apps

6 | © Copyright Zilliz6
History of
Embeddings

7 | © Copyright Zilliz7
Back in the day…

8 | © Copyright Zilliz8
…feature vectors were handcrafted
SIFT
TF-IDF
Harris Corner Detector

9 | © Copyright Zilliz9
Circa 2012, convnets became immensely popular
Source: CS230 notes

10 | © Copyright Zilliz10
And many people discovered the power of vectors

11 | © Copyright Zilliz11
Recurrent neural networks
Source: CS230 notes

12 | © Copyright Zilliz12
Modern
Embedding
Models

13 | © Copyright Zilliz13
Attention Is All You Need?
Source: Illustrated Transformer

14 | © Copyright Zilliz14
Sentence BERT outputs a single vector per sequence
Source: Reimers & GurevychSource: Devlin et al.

15 | © Copyright Zilliz15
Vision transformer
Source: CS230 notes
Source: Dosovitskiy et al.

16 | © Copyright Zilliz16
Other unstructured data modalities

17 | © Copyright Zilliz17
Other unstructured data modalities

18 | © Copyright Zilliz18
Other unstructured data modalities

19 | © Copyright Zilliz19
Other unstructured data modalities

20 | © Copyright Zilliz20
Other unstructured data modalities

21 | © Copyright Zilliz21
Multimodal
Embeddings

22 | © Copyright Zilliz22
Visual + language embeddings
Source: CLIP blog

23 | © Copyright Zilliz23
ImageBind
Source: Girdhar, et al.

24 | © Copyright Zilliz24
Bonus:
Multimodal
RAG

25 | © Copyright Zilliz25
Vanilla RAG
Your
Documents
Embedding Model
Milvus
Question
Question + Context
Search
Gen AI Model
Reliable Answers
What is the default
AUTOINDEX distance
metric in Milvus
Client?
The default
AUTOINDEX distance
metric in Milvus
Client is L2.

26 | © Copyright Zilliz26
Multimodal RAG
Multimodal Model
Milvus
Question
Question + Context
Search
Gen AI Model
Reliable Answers
What kind of music
did they play in the
pre-show?
The musician played
improvised electronic
music.
Tags