AI/ML Infra Meetup | Optimizing ML Data Access with Alluxio: Preprocessing, Pretraining, & Inference at Scale

AI/ML Infra Meetup
Mar. 06, 2025
Organized by Alluxio

For more Alluxio Events: https://www.alluxio.io/events/

Speaker:
- Bin Fan (VP of Technology @ Alluxio)

In this talk, Bin Fan shares his insights on data access challenges in ML applications, with particular emphasis on how Alluxio's distributed caching helps bridge the gap between storage and compute in preprocessing, pretraining and inference.

1. Optimizing ML Data Access with Alluxio
Preprocessing, Pretraining, & Inference at
Scale
Bin Fan
Founding Engineer, VP of Technology @ Alluxio
March 6th 2025

2. About Me

2
Bin Fan
○Founding Engineer, VP of Technology @ Alluxio
○Email: [email protected]
○Linkedin: https://www.linkedin.com/in/bin-fan/
○Previously worked in Google
○PhD in CS at Carnegie Mellon University

3. Powered by Alluxio
ZhihuTELCO & MEDIA
E-COMMERCE
FINANCIAL SERVICES
TECH & INTERNET
OTHERS

4. 4
Alluxio Data Platform
Accelerate data-intensive AI & Analytics workloads

5. Pretraining

6. DeepSeek: Redefining Open-Source LLMs
●Performance on Par with SOTA Models Like GPT-4 at a fraction of the cost
●Disrupting the Competitive Landscape
○Expanding accessibility to much broader audiences
○Higher bar for upcoming general-purpose LLMs
○Potentially more possibility on LLMs with private domain adaptation
●A key lesson: great LLMs can be created by small teams with extremely
efficient resource utilization

7. Engineering/Resource Efficiency in Pre-training
Data Lake (All Data)
us-east-1
Training
Distributed Cache
(Alluxio)
…
Fast Access with
Only Hot Data Cached
Only retrieve
Data on Demand
Distributed Cache
(Alluxio)
us-west-1
Training
●High and consistent I/O performance
→ Comparable I/O performance to HPC storage

●Cloud agnostic
→ Easy to extend the prod env to multi-region/cloud

●Transparent Cache Mgmt
→ Avoid repeatedly preparing (same) data, and the
overhead to maintain local storage

8. Inference

9. LLM Inference: Two Key Metrics
Throughput (System Perspective)
●Measures tokens / sec
●Higher throughput → Better resource utilization, lower system cost
First-time to token (User Perspective)
●Measures time from request submission to the first token generation
●< 100ms → Smooth user experience

10. GPU Memory capacity: Primary Bottleneck
●VRAM is needed for Model Weight & KV-cache
●A typical 13B model inference on A100
●GPT-3 (175B) requires 350GB GPU RAM to load
model weights.
●Large KV-cache is needed for longer context
windows

11. KV Cache Offloading
●A critical optimization for speeding up Transformer models
○Significantly speeding up text generation by reusing previous context instead of recalculating
attention for all tokens at each step.
○Example KV Cache systems :
■LMCache (vLLM Production Stack), MoonCake, etc
●Experimenting Alluxio as a Tiered KV cache
○Talk to me if you are interested in this


Mooncake

12. Deepseek 3FS

13. DeepSeek 3FS: High-Performance Parallel Filesystem
●Newly Open-Source Parallel Filesystem by DeepSeek
○Purpose-Built for RDMA + NVMe hardware
○Powered by FoundationDB Scalable metadata
○Achieves 40GB/s per node throughput (8TB/s with 180 nodes)
●Optimized for High-Throughput Workloads
○Focused on large file read/write performance (not for general-purpose use)
○Recommended using FFRecord format for efficient small file aggregation

14. Complementary Technologies
●3FS: Modern Parallel Filesystem (Similar to GPFS, Lustre)
○Optimized for I/O-intensive workloads with RDMA + NVMe
●Alluxio: Distributed Caching & Access Layer
○Bridges Compute & Data Lakes, accelerating I/O workloads
○Achieves RDMA-comparable read speeds with intelligent caching
○Provides namespace abstraction & indirection for S3, HDFS, GCP, and more → Cloud-agnostic I/O
●Alluxio can integrate with 3FS, just like S3 or HDFS
○Enables high-mid-low tiered I/O solutions, allowing applications to optimize performance and cost

15. <<Scan code to register