Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

1. Kubernetes & AI
?????? Beauty and the ?????? Beast!?!

2. Tobias Schneck
@
[email protected]
@toschneck
Principal Architect
@toschneck

3. As Kubernetes folks
why should we care about AI?
??????

4. … will it be the next big thing?!
??????

5. By 2028, the adoption of AI will culminate in
over 50% of cloud compute
resources devoted to AI workload, up from
less than 10% in 2023.
Gartner® states, 2023

6. OK … so what’s about this AI thingy?
????????????

7. Drawings created with Excalidraw, thanks Koray Oksay (@korayoksay) for the hint ??????

12. ??????

13. … a lot of Data and Math for an
Infrastructure guy ??????

… how does such data get compute?
?????? ??????

14. Credits to Andrej Karpathy ?????? Awesome Intro to LLMs
[1hr Talk] Intro to Large Language Models

15. Credits to Andrej Karpathy ??????

16. Credits to Andrej Karpathy ??????

17. Credits to Andrej Karpathy ??????

18. Credits to Andrej Karpathy ??????

19. Credits to Andrej Karpathy ??????

20. How does our normal Job look like?
??????

21. Based on Adel Zaalouk (@ZaNetworker) drawings from the CNCF Cloud Native AI white paper ??????

22. What will change in our Infra?
??????

23. Based on Adel Zaalouk (@ZaNetworker) drawings from the CNCF Cloud Native AI white paper ??????

24. Based on Adel Zaalouk (@ZaNetworker) drawings from the CNCF Cloud Native AI white paper ??????

26. So, Why Kube ?

27. Flexibility & Standardization
Standard
Container
High-Cube
Container
Hardtop
Container
Open Top
Container
Flat Platform (Plat) Ventilated
Container
Cooling Container Bulk Container
Tank
Container
Container Types

28. Data Center I




Infrastructure Layer
Standardization with Kubernetes





App Services
IT Space I IT Space II IT Space III





Backend Services
DB Services Analytics Observability
Data Center II




Data Center III




Edge

Cloud Providers

Caches AI / ML
DDoS
Protect
Managed
Services
Real Time
Analysis
Intelligent
Edge Devices
Smart
Automation
Data
Processing

29. Kube for AI ⇔ Kube for Applications
?????? Kube is a de facto standard as “cloud operating systems”
✅ API abstraction layer for multiple types of network, storage, and compute resources
?????? Standard interfaces for support of DevOps best practices like GitOps
?????? Variation of Cloud Providers and Services are consumable via standard APIs

30. … and OpenAI uses Kube already 2017 ??????

31. So …. ??????

How to build an AI Platform?
?????? ??????

38. How does the ecosystem look
like!?
?????? ??????

39. AI Frameworks

40. Options to Adapt the Frameworks ??????
Feature
“Kube Native”
via Kubeflow
Managed Platforms
(e.g., SageMaker)
Focused Tools
(e.g., MLflow)
Scope End-to-end MLOps platformManaged MLOps service
Specific functionalities within
ML lifecycle
Open Source Yes No Yes
Scalability & Portability High Depends on cloud provider Moderate
Setup & Management Complex Simpler Simpler
Portability Everywhere Mostly Cloud Mostly Machine based
Vendor Lock-in No Yes (to specific cloud provider)No

41. AI Frameworks
Could use

42. KubeFlow ⁉

?????? Currently the most
feature complete
choice for Kube

?????? But Setup is complex!

43. KubeFlow ⁉

The Beauty ?????? :
●Incubating CNCF Project
●Serving AI Platform in Multi-Tenancy
●Popularity 13.7k ⭐ ~ long-term Maintenance Chance
●Alternatives like MLflow / KServe are integrated

The Beast ??????
●Mostly vendor specific installer instructions ??????
○No maintained automated installer for generic Kubernetes
○Helm chart issue #3173
●Dependency “hell”
○A lot of different 3rd party dependencies constraints
○Hard to adapt again to existing company defaults
●Only support EOL Kubernetes <= 1.26❗
○Usability is then questionable in production

44. Sounds good, but what about
on-prem / offline cases?
??????

45. [Cloud] Data Center I









GPU / TPU Powered Services
based on Argo CD




AI Model Serving
[AI] Application Service
Application
⚙ Separate Model Training / Model Usage Example
Infrastructure Layer
Data Center II




Data Center III




Edge

Cloud Providers

Real Time
Analysis
Intelligent
Edge Devices
Smart
Automation
Data
Processing
Data Delivery
Model
Export
Local AI
Consume
Scale for
Training
Vanilla Setup

46. Starting a POC
??????
github.com/toschneck/kubernetes-and-ai

47. Kubeflow | Katib Architecture for Hyperparameter
Tuning (aka optimization run)

48. Server & Local Model with LocalAI

50. Any Questions?

51. THANKS FOR JOINING!
kubermatic
@toschneck
[email protected]