Prometheus Multi Tenancy

Monitoring Kubernetes multi-tenant environments. Thanos vs Cortex vs Mimir vs Federation.

1. Multi-tenant Kubernetes
observability with Prometheus
robusta-dev Natan Yellin aantn
Natan Yellin, robusta.dev

2. $ whoami
Co-founder of robusta.dev
Multi-cluster Kubernetes observability
Add-on to Prometheus
Substack newsletter: Why this Kubernetes thing?
Natan Yellin aantnrobusta-dev

3. How should I gather
Prometheus metrics from
all my tenants?
Natan Yellin aantnrobusta-dev

4. Assumptions
Natan Yellin aantn
Clusters
Namespaces
Virtual clusters (e.g. capsule, kamaji, vcluster)
etc...
1. Many Kubernetes tenants
2. Tenants need some form of isolation
3. We want to monitor with Prometheus
robusta-dev

5. What should I use?
Natan Yellin aantnrobusta-dev

6. In the beginning there was one
Natan Yellin aantnrobusta-dev

7. In the beginning there was one
Natan Yellin aantn
Simple
No security isolation/RBAC
No performance isolation
If tenants are clusters, discovery is
annoying
Advantages:
Disadvantages:
"One team broke Prometheus for
everyone else"
robusta-dev

8. Then there were many
Natan Yellin aantnrobusta-dev

9. Then there were many
Natan Yellin aantn
Simple
Security isolation
Performance isolation
Scalable?
No unified queries
No unified management
More resources?
Advantages:
Major Disadvantage:
Minor Disadvantages:
"If you break it, it only breaks for your
product line."
robusta-dev

10. What we want
Natan Yellin aantn
Isolation
Scalability
Decentralized:
Query all Prometheuses at once
Centralized:
robusta-dev

11. What else we want?
Natan Yellin aantn
Scalability
Long term storage of metrics
1.
2.
robusta-dev

12. Three approaches
Natan Yellin aantnrobusta-dev

13. Solve it outside Prometheus
Natan Yellin aantnrobusta-dev

14. Solve it outside Prometheus
Natan Yellin aantn
Doesn't touch Prometheus itself
Delegates problem to other tool
Queries need to address one
Prometheus at a time
Key advantages:
Key disadvantage:
robusta-dev

15. Multiple + Centralized (take 1)
Natan Yellin aantnrobusta-dev

16. Multiple + central (take 1)
Natan Yellin aantn
Reuses existing Prometheus
Federated can do roll-up
Federated can selectively scrape
With roll-up/selective you can't
actually query all Prometheuses
Scaling
Key advantages:
Key disadvantages:
robusta-dev

17. Natan Yellin aantn
Disclaimer: Thanos has lots of options, I'm simplifying a little
robusta-dev

18. Multiple + central (take 2)
Natan Yellin aantnrobusta-dev

19. Multiple Prometheuses + central Prometheus (take 2)
Natan Yellin aantn
Super scalable!
Reuses existing Prometheus
Very common solution, lots of tooling
No RBAC built-in
Key advantages:
Key disadvantages:
"Most mature option" - most people
robusta-dev

20. One Prometheus to Rule them All
Natan Yellin aantnrobusta-dev

21. One Prometheus to Rule them All
Natan Yellin aantnrobusta-dev
Cortex
Grafana Mimir
VictoriaMetrics
TimescaleDB
M3DB
Options:
...

22. Grafana Mimir
Natan Yellin aantnrobusta-dev
Native multi-tenancy!
Backed by Grafana
Complexity
Key advantages:
Key disadvantages:

23. Other useful tools
Natan Yellin aantn
Add prom-label-proxy to Thanos
(and others) to enforce RBAC
robusta-dev

24. Thank you!
Natan Yellin aantn
A special thank you to Shalom Cohen and Evgeny Uklist + Racoons team for
providing inputs
robusta-dev

25. Questions?
Natan Yellin aantnrobusta-dev