Minimizing Request Latency of Self-Hosted ML Models by Julia Kroll

Join our session on minimizing latency in self-hosted #ML models in cloud environments. Learn strategies for deploying Deepgram's speech-to-text models on your hardware, including concurrency limits, auto-scaling, input chunk granularity, and efficient model loading. Optimize your ML inference.

1. A ScyllaDB Community
Minimizing Request Latency of
Self-Hosted ML Models
Julia Kroll
Applied Engineer @ Deepgram

2. Julia Kroll (she/her)

Applied Engineer @ Deepgram
■I’ve contributed to global language AI products
spanning dozens of languages
■P99s are outliers and should be removed accordingly
■Dark mode is overrated
■I brew my own kombucha

3. ML Models 101
■Training: learning by looking at lots of data
■Inference: predicting based on past learning

■Both require GPUs
■Training is more compute-intensive than inference
●Meta is training Llama 3 on a 24,576-GPU cluster
●Llama can run inference for a request on 1 GPU
■We’ll focus on inference

4. What is self-hosting?

■You have a trained ML model
■You want to serve inference requests with the model
■You use your own hardware (bare-metal or cloud VMs) to do so

5. Which models can you self-host?
✔ Open-source
●Llama, Falcon, Mistral
●Models on Hugging Face
✔ Encrypted, distributed models
●Deepgram
✔ Privately trained by you
●Your proprietary, in-house models
❌ Not distributed
●GPT, Gemini, Claude

6. How do you self-host at scale?

■Containerization via Docker, Kubernetes, etc
■Each container has a copy of the model
■Each container serves some inference requests
■You may scale to more or fewer containers as needed

7. Pros and cons of hosted API versus self-hosting
Hosted
■Pros:
●You don’t have to think about it.
●It’s abstracted.
●It just works.
■Cons:
●You lack control.
●It may not suit your needs.
●It may not scale.
●It’s expensive.
●You send data to a 3rd party.

Self-Hosted
■Pros:
●You control everything.
●You can trick out your perfect setup.
●It’s low-margin.
●You retain your own data.
■Cons:
●You’re responsible for everything.
●It takes engineering time and effort.
●You might have to remember that
servers exist in the physical world.

8. Demo: self-hosted Deepgram

9. So you want to self-host
in prod?

10. 3 questions to ask yourself…

11. 1. How many requests can I serve?

■Each server will have a maximum level of traffic it can serve before it falls over.
■Latency will likely degrade before total failure.
■To determine a threshold:
●Define maximum acceptable latencies
●Benchmark to obtain maximum number of requests
●Monitor and set limits in prod

12. 1. How many requests can I serve?
n requests: g4dn, T4 GPU
P99 = 560 ms
Spiky P99 over time
2n requests: g6, L4 GPU
P99 = 360ms
Smooth P99 over time

13. 2. How many servers do I need?
■Now that you know how many requests 1 unit of compute can serve,
how many units do you need?
●Baseline, min, max, periodicity (daily, weekly)
■Scale on number of requests, given max requests for acceptable latency
●Kubernetes: HorizontalPodAutoscaler
●Example:
■Values of max_requests = 100, target_ratio 0.9
■Active requests = 91 → utilization ratio > 0.9 → scale up

14. 3. Which set of models am I serving?

■For low-latency inference, models must stay in memory.
■Are you serving one model, a few, or many?
■How large are your models? How much memory do your instances have?

15. 3. Which set of models am I serving?
5 models
P99 = 800 ms
15 models
P99 = 2000ms
1 model
P99 = < 100 ms
not pictured

16. 3. Which set of models am I serving?

To serve many models with low latency:
■Several servers, each with a subset of models
■Load-balance to distribute traffic across the servers
■Send each request to a server containing the right model

17. Conclusion
For low-latency self-hosted ML inference:
■Select your model(s)
■Define and benchmark your request limits
■Auto-scale on those known limits
■Distribute many models (and their requests) across servers

18. Thank you! Let’s connect.
Julia Kroll
[email protected]
https://www.linkedin.com/in/juliakroll