Implement Agentic RAG Using Claude 3.5 Sonnet, LlamaIndex, and Milvus

In this talk, we review the cutting-edge techniques for implementing Agentic Retrieval-Augmented Generation (RAG) systems, leveraging the power of Claude 3.5 Sonnet, LlamaIndex, and Milvus. Retrieval-Augmented Generation has become a cornerstone in building intelligent systems that require both generative and retrieval capabilities, enabling more accurate, context-aware, and dynamic responses. The session will cover practical implementation details, including how to set up the RAG pipeline, integrate the components seamlessly, and optimize the system for performance and scalability. We will also explore use cases and real-world applications of Agentic RAG, demonstrating its potential in enhancing AI-driven solutions.

1. Build Agentic RAG Using
Claude 3.5 Sonnet,
LlamaIndex,
and Zilliz

Bill Zhang

2. System Evolving
2022, 2023, 2024
●LLM
●Compound AI Systems
●Agents
Shift from Models to Compound Systems
Compound AI Systems
User
Query
Generate
(LLM)
Answer
User
Query
Search
Query(LL
M)
Generate(
LLM)
Generate(
LLM)
Compound AI
System

3. Limitations of models
●Limited Data/Knowledge
●Hard to Adapt

4. Benefits of Compound Systems
●Modular
●Easy to Adapt
●E.g. RAG
○Very Specific
Control Logic
Programmatic
Autonomy
(LLM)
Fast Slow
- +

5. LLM Agents
●Reason
●Act (Using Tools)
○Search
○Calculate
○Coding
○Another LLM
●Access Memory

LLM Powered Autonomous Agents
User
Query
Plan/
Think
Act/
Tools
Observe Answer

6. RAG

7. Weakness
Certain Questions where top-k retrieval will fail:
●Summarization Questions: Summarize a document
●Comparison Questions: Compare the performance of employee A and B
●Structured Analytics + Semantic Search: Analyze the risk factors of highest
performing vector database companies globally.
●General multi-part questions: Tell me A, and Tell me B, Make a table and
generate a conclusion.

8. What is Agentic RAG?
RAG is just a tool for Agents.

9. Agents from Phase I to X
●Routing: Given user query and set of choices, output subset of
choices to route to.
●Query Planning: Break down query into parallelizable
Sub-queries. Each subquery can be executed against any RAG
●Tool Use: Use an LLM to call an API, LLM will determine the
parameters of the API.

10. ●ReAct: Reasoning + Acting with LLMs
○Sequential multi-part problem? (Make it Loop)
○Stateful (Add memory)

ReAct Agent Model

11. ●Dynamic Query Planning
○Plan Ahead at each Step
○Parallelize execution as much as possible
LLMCompiler

12. Observability, Control, Customize……
●

13. Implement Agentic RAG with Claude 3.5 Sonnet

14. Embedding Generation

17. Zilliz Cloud Vector Database Integration
●Zilliz Registration
○https://cloud.zilliz.com/signup
●Data ingestion

18. Data Retrieving

19. Create AI Agent

20. Bring everything together

21. Data
Ingestion