Small to Slide: RAG using MultiModal LLMs.

With the advent of LLMs understanding multimodal inputs, document understanding has a new tool in the arsenal. Elliot Kang and I built an example using Milvus and a real slide deck to show how you can use retrieval in combination with multimodal input to get better results from your existing powerpoint slides.

1. Small to Slide:
Multimodal for Better RAG
Yi Ding, formerly LlamaIndex.TS

2. A bit about me
●Cofounded the Messaging Apps team at Apple and was there for 8 years.

●Started using LLMs on January 21, 2023.

●Joined LlamaIndex. Shipped LlamaIndex.TS, create-llama, LlamaCloud,
LlamaParse + many partnerships incl. w/ Zilliz.

3. A quick survey
●When did you start building with LLMs?

4. A quick survey
●When did you start building with LLMs?
○This past week?

5. A quick survey
●When did you start building with LLMs?
○This past week?
○This past month?

6. A quick survey
●When did you start building with LLMs?
○This past week?
○This past month?
○2024?

7. A quick survey
●When did you start building with LLMs?
○This past week?
○This past month?
○2024?
○2023?

8. A quick survey
●When did you start building with LLMs?
○This past week?
○This past month?
○2024?
○2023?
○Before?

9. The elephant in the room: OpenAI o1-preview
●Came out Thursday.

10. What’s the big deal with strawberries?

11. What’s the big deal with strawberries?
●LLMs “see” tokens, not characters or words.

12. What’s the big deal with strawberries?

13. What’s the big deal with strawberries?

14. What’s the big deal with strawberries?

15. What’s the big deal with strawberries?

16. o1 stands for reasoning

17. Look familiar?

18. Thinking Longer = More Intelligent
https://www.maximumtruth.org/p/ma
ssive-breakthrough-in-ai-intelligence

19. Superhuman language capabilities

20. So all good right?
●Well…

21. So all good right?
●Well…

●More thinking = more chances to fail.

22. From the official OpenAI docs

23. Overthinking
Thanks @hanchunglee

24. Overthinking

25. Hallucinations

26. Hallucinations

27. Hallucinations
●We are used to hallucinations being negative.

●But they can also be “positive.”

●Suspect that in model training, o1-preview is incentivized to “trust its gut.”

28. Hallucinations

29. It’s still early

30. Small to Slide:
Multimodal for Better RAG
Yi Ding, formerly LlamaIndex.TS

31. A quick survey
●When did you start building with LLMs?
○This past week?
○This past month?
○2024?
○2023?
○Before?

●RAG?

32. What is RAG?

33. What is RAG?
●Retrieval
●Augmented
●Generation

34. What is RAG?
●Retrieval
●Augmented
●Generation


●Better
●Output
●With
●Search

35. What is RAG?

36. Small to Big RAG
●Embed small, retrieve big.

●Key concept: what is embedded does not have to match what is retrieved
and given to the LLM.

●Coined at the Unstructured Meetup!

37. Small to Big RAG
Chris Churilo works in Marketing. She is currently VP of Marketing at Zilliz, the
makers of the Milvus Vector Database. Prior to that Ms. Churilo was VP of
Marketing at InfluxData.
Chris Churilo is a graduate of Berkeley. Go Bears! In her spare time she enjoys
stand up paddleboarding.

38. Small to Big RAG
Chris Churilo works in Marketing. She is currently VP of Marketing at Zilliz, the
makers of the Milvus Vector Database. Prior to that Ms. Churilo was VP of
Marketing at InfluxData.
Chris Churilo is a graduate of Berkeley. Go Bears! In her spare time she enjoys
stand up paddleboarding.

Q: Where does Ms. Churilo work?

39. Small to Big RAG
Chris Churilo works in Marketing. She is currently VP of Marketing at Zilliz, the
makers of the Milvus Vector Database. Prior to that Ms. Churilo was VP of
Marketing at InfluxData.
Chris Churilo is a graduate of Berkeley. Go Bears! In her spare time she enjoys
stand up paddleboarding.

Q: Where does Ms. Churilo work?

40. Why?

41. Small to Slide RAG
●Embed text, retrieve image.

●Key concept: what is embedded does not have to match what is retrieved
and given to the multimodal LLM.

●Built with Elliot Kang (LlamaIndex.TS contributor!)

42. Demo

43. But do we need to convert slide -> text at all?

44. ColPali
Instead of transforming PDFs to text,

Transform PDF to image and search over them.

https://huggingface.co/blog/manu/colpali

45. ColPali
The search is performed using not a single vector per image (like image
embedding models) but many (10^n) embeddings per image.

Vectors are compared against embeddings for query.

46. ColPali

47. ColPali

48. ColPali: How I think about it
●Visual ngrams*

●*Not a deep learning researcher

●Generate using image context (like Small to Slide!)

49. ColPali: Downsides
●Needs GPU (Modal deployable version coming!)

●Models are language and training specific.

●Far less exploration/tooling so far.

50. RAG is Evolving Very Fast
Text based RAG Small to Slide ColPali
Pros Easiest to get
started
Best tooling support
Better capture
multimodal
information
No text conversion
needed
Built in attribution
Cons Lose multimodal
information
Chunking is not
perfect
A bit more setup
than text based
Specific to certain
data types
Need GPU and
custom models

51. One Last Favor