LLM Inference Optimization by Chip Huyen

This talk will discuss why LLM inference is slow and key latency metrics. It also covers techniques that make LLM inference fast, including different batching, parallelism, and prompt caching. Not all latency problems are engineering problems though. This talk will also cover interesting tricks to hide latency at an application level.

2. Inference optimization
Chip Huyen
huyenchip.com
@chipro

3. LLM compute
3Chip Huyen | @chipro
Training Finetuning
Inference

4. LLM economics
4Chip Huyen | @chipro
Training Finetuning
Inference

5. LLM economics
5Chip Huyen | @chipro
Training Finetuning
Inference w/o reasoning
Training Finetuning
Inference w/ reasoning

6. LLM economics
6Chip Huyen | @chipro
Training compute
Inference compute
:

7. Inference cost determines LLM profitability


7Chip Huyen | @chipro

8. Optimization: quality, cost, latency
8Chip Huyen | @chipro
Good Fast
Cheap

9. Optimization: quality, cost, latency
9Chip Huyen | @chipro
Good Fast
Cheap
What’s missing?

10. ⚠⚠
Many, many techniques
Will only focus on core, easier-to-apply techniques
⚠⚠

10Chip Huyen | @chipro

11. JamesWho wrote the book Chaos? Gleick did .
Input Output
TTFT
Time To First Token
TPOT
Time Per Output Token
Total latency = TTFT + TPOT x (N-1)

12. JamesWho wrote the book Chaos? Gleick did .
Input Output
TTFT*
Time To First Token
TPOT
Time Per Output Token
Total latency = TTFT + TPOT x (N-1)
First generated token != First visible token

13. JamesWho wrote the book Chaos? Gleick did .
Input Output
TTFT
Time To First Token
TPOT
Time Per Output Token
Total latency = TTFT + TPOT x (N-1)
Which metric is more important to you?

14. JamesWho wrote the book Chaos? Gleick did .
Input Output
TTFT
Time To First Token
TPOT
Time Per Output Token
Total latency = TTFT + TPOT x (N-1)
TTFT vs. TPOT tradeoff?

15. Throughput vs. goodput
Examples:
●Goal
○TTFT < 200 ms
○TPOT < 100 ms
●Throughput: 10 requests/m

15Chip Huyen | @chipro
What’s goodput?

16. Inference server
Hardware
Model 2Model 1 Model 3
Apps
Requests
Responses
How to make LLMs run fast?

17. Inference server
Hardware
Model 2Model 1 Model 3
Apps
Requests
Responses
How to make LLMs run fast?
Mm, can we just use a lot
of compute?

18. 34B
8B
8B
8B
8B
70B
13B
13B
13B
13B8B
34B
34B
70B
Given X machines & Y models,
how to efficiently load them?

19. Inference server
Hardware
Model 2Model 1 Model 3
Apps
Requests
Responses
How to make LLMs run fast?
Make more powerful hardware0

20. Inference server
Hardware
Model 2Model 1 Model 3
Apps
Requests
Responses
Make more powerful hardware0
Make models more efficient1

21. Inference server
Hardware
Model 2Model 1 Model 3
Apps
Requests
Responses
Make more powerful hardware0
Make models more efficient1
Make service more efficient2

22. 22Chip Huyen | @chipro

23. Inference server
Hardware
Model 2Model 1 Model 3
Apps
Requests
Responses
Make models more efficient1
Make service more efficient2
Can change
model quality
Shouldn’t modify
output quality
Make more powerful hardware0

24. Make models more efficient
●Quantization
○Reducing precision
Ex: 32 bit → 8 bit
7 GB x 32 bit = 28 GB
7 GB x 8 bit = 7 GB
24Chip Huyen | @chipro

25. Make models more efficient
●Quantization
●Distillation

25Chip Huyen | @chipro
Teacher
Student
Data
Can student outperform teacher?

26. Make service more efficient


26Chip Huyen | @chipro

27. Requests
Static batching
Dynamic batching
Time
Batching

28. R1
R2
R3
R4
R5
R6
R7
R8
R2
R3
R4
R1
R5
R6
R7
R8
Normal
batching
Continuous
batching
Batch 1 Batch 2
…
…
…
…
…
Time
Continuous batching

29. JamesWho wrote the book Chaos? Gleick did .
Prefill Decode Decode Decode Decode
Input Output
Input tokens can be
processed in parallel
Output tokens are
generated sequentially

30. JamesWho wrote the book Chaos? Gleick did .
Prefill Decode Decode Decode Decode
Input Output
Compute-bound Memory bandwidth-bound
Decoupling prefill & decode

31. Non
distributed
Device 1
Device 2




Gather
Tensor
parallelism

32. Layer 1
Layer 2
Layer 3
Layer 4
Layer 1 Layer 1
Layer 2
Layer 3
Layer 1
Layer 2 Layer 2
Layer 3
Layer 4
Layer 3
Layer 4 Layer 4
MB1 MB2 MB3 MB4
Device 1
Device 2
Device 3
Device 4
Input
Output MB1 MB2 MB3 MB4
Time
Pipeline
parallelism

33. Prompt caching
33Chip Huyen | @chipro

34. Prompt caching
34Chip Huyen | @chipro
Structure your prompt to maximize prompt
cache hit!

35. Claude Code: prompt cache
35Chip Huyen | @chipro
sniffly.dev

36. Key takeaways
1.Inference optimization is essential for LLM economics
2.Evaluate an inference service not just by cost & latency but also quality
(techniques used can change a model’s quality)
3.Structure your prompts to maximize prompt cache hit
36Chip Huyen | @chipro

37. Thank you!

@chipro
chiphuyen
chiphuyen
huyenchip.com/blog
huyenchip.com/llama-policeTrack top 1200+ AI repos