Leveraging Open-Source LLMs for Production

This talk explores the practical use of open-source LLMs in real-world applications, discussing their pros and cons, such as privacy benefits and cost-efficiency, alongside challenges like technical expertise and deployment techniques. It covers the economics of fine-tuning and deployment, infrastructure needs, and various frameworks, while providing an overview of key open-source LLMs and their features. The goal is to equip attendees with a technical understanding of how to adopt open-source LLMs models.

1. Leveraging
Open-Source LLMs
for Production


Andrey Cheptsov, Founder at

InfoQ Dev Summit, Munich, September, 2024

2. Closed-source vs open-source
Looking back at
predictions

3. Closed-source vs open-source
Source: Maxime Labonne
An open-source model closes
the gap for the first time
MMLU Pro
(5-shot)

4. Benchmarks
Source: Meta
Llama 3.1
Instruct

5. Benchmarks
Source: Qwen
Qwen 2.5
Instruct

6. When to use open-source models
Closed-source vs open-source
●Control (1)
●Customization (3)
●Transparency (2)
●Ecosystem (5)
●Cost-effectiveness (4)

7. Llama 3.1 (no
optimization)
Hardware requirements
Add text
Inference Training
A100:80Gx8 x 2 nodes A100:80Gx8 x 6 nodes

8. Hardware requirements
How will you even run
this?

9. Source: HuggingFace
Quantization
Optimization techniques

10. Llama 3.1 (with
optimization)
Hardware requirements
Training Inference

11. Optimization techniques
Low Rank Adaptation
(LoRA)
Source: HuggingFace

12. Llama 3.1 (with
optimization)
Hardware requirements
Training Inference

13. Llama 3.1 (with
optimization)
Hardware requirements
TrainingInference

14. Development process
Source: Sebastian Raschka
Pre-training and
Post-training

15. Development process
Source: Sebastian Raschka
Pre-training and
Post-training

16. Development process
Supervised fine-tuning
(SFT)
Source: Chip Huen

17. Development process
Source: Sebastian Raschka
Pre-training and
Post-training

18. Development process
Source: Sebastian Raschka
Pre-training and
Post-training

19. Reinforcement Learning
from Human Feedback
(RLHF)
Source: Chip Huen
Development process

20. Development process
Source: Sebastian Raschka
Pre-training and
Post-training

21. Source: archive.org
RLHF vs DPO
Development process

22. ●CUDA/ROCm/XLA
●vLLM/TGI/NIM
●TRL/Axolotl
Frameworks & tools
Development process