MemGPT: Introduction to Memory Augmented Chat

Read more: https://zilliz.com/blog/introduction-to-memgpt-and-milvus-integration
Why we need memory-augmented LLMs

1. MemGPT
why we need memory-augmented LLMs

2. ?????? Charles Packer
●PhD candidate @ Sky / BAIR, focus in AI
●Author of MemGPT
○First paper demonstrating how to give GPT-4
self-editing memory (AI that can learn over time)
●Working on agents since 2017
○“the dark ages”
○5 BC = Before ChatGPT
?????? [email protected]
?????? @charlespacker

3. Agents in 2017 ??????

4. For LLMs, “memory” is everything
memory is context
context includes long-term memory, tool use, ICL, RAG, …

5. For LLMs, “memory” is everything
“memory” =

6. MemGPT - giving LLMs real “memory”
GPT

7. Why is this the “best” AI product?

8. What about this?

9. Search engine AI assistant

10. Search engine AI assistant

12. Search engine AI assistant

13. Search engine AI assistant

15. tl;dr
LLMs doing constrained Q/A
�

16. tl;dr
LLMs doing long-range, open-ended tasks
�

18. 90%+ of questions are
related to one project
No shared context! Why?
We don’t know how to do it…

19. How to get an LLM to use
●hundreds of chats
●+ code base (1M+ LoC)
●+ …

20. ●…RAG?
●Lots of retrieval?
●Multi-step retrieval?
●Retrieval that works?
●What about writing?

21. …long-context LLMs?
Cost + latency
Context pollution

22. No shared context! Why?
We don’t know how to do it…

23. Search engine AI assistant
state management

24. MemGPT -> giving LLMs real “memory”

25. MemGPT -> memory via tools
LLM
tools
??????
Memory

26. Text
User message??????
GPT-4
Context window
8k max token limit
ChatGPT
Text
Agent reply??????
Standard LLM setup
e.g., ChatGPT UI + GPT-4 model

27. Event
User message??????
Document upload??????
System alert??????
Function
Send message??????
Query database
Pause interrupts??????
LLM
Virtual context
Main context
External context
∞ tokens
Max token limit
MemGPT
parse
parse
MemGPT LLM OS setup
Event loop + functions + memory hierarchy

28. Event
User message??????
Document upload??????
System alert??????
Function
Send message??????
Query database
Pause interrupts??????
LLM
Virtual context
Main context
External context
∞ tokens
8k max token limit
MemGPT
parse
parse
Fixed-context LLM
e.g., GPT-4 with 8k max tokens

29. Event
User message??????
Document upload??????
System alert??????
Function
Send message??????
Query database
Pause interrupts??????
LLM
Virtual context
Main context
External context
∞ tokens
8k max token limit
MemGPT
parse
parse
LLM inputs are “events” (JSON)
System alerts help the LLM manage memory
Virtual context
Main context
External context
∞ tokens
Max token limit
LLM

30. Event
User message??????
Document upload??????
System alert??????
LLM inputs are “events” (JSON)
System alerts help the LLM manage memory
{ “type”: “user_message”,
“content”: “how to undo git commit
-am?” }
{ “type”: “document_upload”,
“info”: “9 page PDF”,
“summary”: “MemGPT research paper” }
{ “type”: “system_alert”,
“content”: “Memory warning: 75% of
context used.” }

31. Event
User message??????
Document upload??????
System alert??????
Function
Send message??????
Query database
Pause interrupts??????
LLM
Virtual context
Main context
External context
∞ tokens
Max token limit
MemGPT
parse
parse
LLM outputs are functions (JSON)
Event loop + functions that allow editing memory

32. Function
Send message??????
Query database
Pause interrupts??????
Agent can query out-of-context
information with functions
{
“function”: “archival_memory_search ”,
“params”: {
“query”: “Berkeley LLM Meetup”,
“page”: “0”
}
}

33. Function
Send message??????
Query database
Pause interrupts??????
Pages into (finite) LLM context
{
“function”: “archival_memory_search ”,
“params”: {
“query”: “Berkeley LLM Meetup”,
“page”: “0”
}
}
LLM

34. Function
Send message??????
Edit context
Pause interrupts??????
Agent can edit their own memory
including their own context

{
“function”: “core_memory_replace”,
“params”: {
“old_content”: “OAI Assistants API”,
“new_content”: “MemGPT API”
}
}

35. Function
Send message??????
Edit context
Pause interrupts??????
Core memory is a reserved block
System
prompt
In-context
memory block
Working
context queue
{
“function”: “core_memory_replace”,
“params”: {
“old_content”: “OAI Assistants API”,
“new_content”: “MemGPT API”
}
}

36. Function
Send message??????
Query database
Pause interrupts??????
{
“function”: “send_message”,
“params”: {
“message”: “How may I assist you?”
}
}
User messages are a function
Allows interacting with system
autonomously w/o user inputs

37. { “type”: “user_message”,
“content”: “what’s happening on may 21 2024? ” }
{
“function”: “archival_memory_search”,
“params”: {
“query”: “may 21 2024”,
}
}
{
“function”: “send_message”,
“params”: {
“message”: “Have you heard about Milvus? ”
}
}
??????
??????

38. what’s happening on may 21 2024?
Have you heard about Milvus?
??????
??????
(User’s POV)

39. Event
User message??????
Document upload??????
System alert??????
Function
Send message??????
Query database
Pause interrupts??????
LLM
Virtual context
Main context
External context
∞ tokens
Max token limit
MemGPT
parse
parse
MemGPT LLM OS setup
Event loop + functions + memory hierarchy

40. Calling & executing custom tools
MemGPT -> Building LLM Agents
Long-term memory management??????
??????
Loading external data sources (RAG)
??????

41. MemGPT
= the OSS platform for building ?????? and hosting ?????? LLM agents

42. DeveloperUser
MemGPT
Dev Portal
MemGPT CLI
$ memgpt run

43. MemGPT server
User-facing
application
REST API
Users
Agents
Tools
Sources
user_id: …
agent_id: …
Personal Assistant
State
Memories
Documents

44. MemGPT server
User-facing
application
REST API
Users
Agents
Tools
Sources
user_id: …
agent_id: …
Personal Assistant
State
Memories
Documents
Webhooks

45. MemGPT
may 21 developer update ??????

47. Docker integration - the fastest way to create a MemGPT server
Step 1: docker compose up
Step 2: create/edit/message agents using the MemGPT API
MemGPT ❤

48. MemGPT streaming API - token streaming
CLI: memgpt run --stream
REST API: use the stream_tokens flag [PR #1280 - staging]

49. MemGPT streaming API - token streaming
MemGPT API works with both non-streaming + streaming endpoints
If the true LLM backend doesn’t support streaming, “fake streaming”

50. MemGPT /chat/completions proxy API
Connect your MemGPT server to any /chat/completions service!
For example - ?????? voice call your MemGPT agents using VAPI!

MemGPT ??????