Cache Me If You Can: How Grafana Labs Scaled Up Their Memcached 42x & Cut Costs Too

Our cloud database stores billions of files in object storage. With petabytes of data being queried every day, we started bumping into our cloud storage providers' rate-limits, resulting in decreased reliability & performance. We had large memcached clusters in place to absorb & deamplify reads to object storage - but these could hold at most a few hours’ worth of data, and constantly churned due to the excessive volume of data passing through. The conclusion we came to was: we needed much larger caches, ideally without inflating our cloud costs and adding operational complexity.

I'll show how we managed to increase our cache size by 45x and reduce our costs by using a little-known feature of memcached called ""extstore"". Extstore enables offloading of objects to SSDs which can't fit into memory. In this talk I’ll be covering how we use it, how to monitor it, why we chose it, and other considerations. I'll also cover how we use ephemeral storage provided by public cloud vendors in the form of physically-attached SSDs with incredibly high throughput, low latency, and best of all - low cost!

This talk is also a story of how products evolve, and how we as a team are buying time in the short term to keep up our reliability while we evolve our storage design in the medium-long term.

1. Cache Me If You Can How Grafana Labs Scaled Up Their Memcached 42x and Cut Costs Too Danny Kopping Senior Software Engineer @ Grafana Labs

2. Cutting to the chase rollout rollout 65% reduction in object storage reqs using memcached + 50TB of SSD 2% overall TCO reduction No degradation in performance

3. But first… Loki internals

4. 2023-06-16T06:09:05.123456789Z {app=”nginx”, env=”dev”} Timestamp with nanosecond precision Content log line Labels/Selectors key-value pairs indexed unindexed How does Loki work? 192.168.1.1 "GET / HTTP/1.1" 200 Object Storage chunks index

5. ingester logs 192 .168.1.1 - - [06/16/2023:06:09:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.74.0" 192.168.1.2 - - [06/16/2023:06:09:06 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "wget/1.21.1" 192.168.1.3 - - [06/16/2023:06:09:07 +0000] "GET /about.html HTTP/1.1" 200 396 "-" "Safari/15.4" 192.168.1.4 - - [06/16/2023:06:09:08 +0000] "GET /contact.html HTTP/1.1" 200 299 "-" "Firefox/92.0" 192.168.1.5 - - [06/16/2023:06:09:09 +0000] "GET /blog/index.html HTTP/1.1" 200 824 "-" "Chrome/92.0.4515.131" 192.168.1.6 - - [06/16/2023:06:09:10 +0000] "GET /blog/post-1.html HTTP/1.1" 200 543 "-" "Opera/12.15" 192.168.1.7 - - [06/16/2023:06:09:11 +0000] "GET /blog/post-2.html HTTP/1.1" 200 456 "-" "Edge/102.0.1245.153" 192.168.1.8 - - [06/16/2023:06:09:12 +0000] "GET /blog/post-3.html HTTP/1.1" 200 369 "-" "IE/11.0" {app=”nginx”, pod=”nginx-a”, az=”us-east-1a”} 192.168.1.1 - - [06/16/2023:06:09:13 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.74.0" 192.168.1.2 - - [06/16/2023:06:09:14 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "wget/1.21.1" 192.168.1.3 - - [06/16/2023:06:09:15 +0000] "GET /about.html HTTP/1.1" 200 396 "-" "Safari/15.4" 192.168.1.4 - - [06/16/2023:06:09:16 +0000] "GET /contact.html HTTP/1.1" 200 299 "-" "Firefox/92.0" 192.168.1.5 - - [06/16/2023:06:09:17 +0000] "GET /blog/index.html HTTP/1.1" 200 824 "-" "Chrome/92.0.4515.131" 192.168.1.6 - - [06/16/2023:06:09:18 +0000] "GET /blog/post-1.html HTTP/1.1" 200 543 "-" "Opera/12.15" 192.168.1.7 - - [06/16/2023:06:09:19 +0000] "GET /blog/post-2.html HTTP/1.1" 200 456 "-" "Edge/102.0.1245.153" 192.168.1.8 - - [06/16/2023:06:09:20 +0000] "GET /blog/post-3.html HTTP/1.1" 200 369 "-" "IE/11.0" {app=”nginx”, pod=”nginx-b”, az=”us-east-1a”} 192.168.1.1 - - [06/16/2023:06:09:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.74.0" 192.168.1.2 - - [06/16/2023:06:09:06 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "wget/1.21.1" 192.168.1.3 - - [06/16/2023:06:09:07 +0000] "GET /about.html HTTP/1.1" 200 396 "-" "Safari/15.4" 192.168.1.4 - - [06/16/2023:06:09:08 +0000] "GET /contact.html HTTP/1.1" 200 299 "-" "Firefox/92.0" 192.168.1.5 - - [06/16/2023:06:09:09 +0000] "GET /blog/index.html HTTP/1.1" 200 824 "-" "Chrome/92.0.4515.131" 192.168.1.6 - - [06/16/2023:06:09:10 +0000] "GET /blog/post-1.html HTTP/1.1" 200 543 "-" "Opera/12.15" 192.168.1.7 - - [06/16/2023:06:09:11 +0000] "GET /blog/post-2.html HTTP/1.1" 200 456 "-" "Edge/102.0.1245.153" 192.168.1.8 - - [06/16/2023:06:09:12 +0000] "GET /blog/post-3.html HTTP/1.1" 200 369 "-" "IE/11.0" {app=”nginx”, pod=”nginx-c”, az=”us-east-1b”} 192.168.1.1 - - [06/16/2023:06:09:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.74.0" 192.168.1.2 - - [06/16/2023:06:09:06 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "wget/1.21.1" 192.168.1.3 - - [06/16/2023:06:09:07 +0000] "GET /about.html HTTP/1.1" 200 396 "-" "Safari/15.4" 192.168.1.4 - - [06/16/2023:06:09:08 +0000] "GET /contact.html HTTP/1.1" 200 299 "-" "Firefox/92.0" 192.168.1.5 - - [06/16/2023:06:09:09 +0000] "GET /blog/index.html HTTP/1.1" 200 824 "-" "Chrome/92.0.4515.131" 192.168.1.6 - - [06/16/2023:06:09:10 +0000] "GET /blog/post-1.html HTTP/1.1" 200 543 "-" "Opera/12.15" 192.168.1.7 - - [06/16/2023:06:09:11 +0000] "GET /blog/post-2.html HTTP/1.1" 200 456 "-" "Edge/102.0.1245.153" 192.168.1.8 - - [06/16/2023:06:09:12 +0000] "GET /blog/post-3.html HTTP/1.1" 200 369 "-" "IE/11.0" {app=”nginx”, pod=”nginx-d”, az=”us-east-1b”} Object Storage 2023-06-16 11:02.123456789 2023-06-16 11:12.123456789 2023-06-16 11:02.123456789 2023-06-16 11:12.123456789 2023-06-16 11:02.123456789 2023-06-16 11:12.123456789 2023-06-16 11:02.123456789 2023-06-16 11:12.123456789 00ba6fb0 14a8d9b3 94d10ab7 72447153 Ingestion chunks

6. 00ba6fb0 14a8d9b3 94d10ab7 72447153 {app=”nginx”} {pod=”nginx-a”} {az=”us-east-1a”} {pod=”nginx-b”} {pod=”nginx-c”} {pod=”nginx-d”} {az=”us-east-1b”} Indexing

7. 00ba6fb0 14a8d9b3 94d10ab7 72447153 {app=”nginx”} Querying

8. 00ba6fb0 72447153 {app=”nginx”} {az=”us-east-1a”} Querying

9. 100GB Raw Logs Timeframe Brute force search, heavily parallelized Label selector 10GB 1TB/s {app=”nginx”, az=”us-east-1a”} |= “12.34.56.78” index 100TB Fast queries!

10. 100GB Raw Logs Timeframe Brute force search, heavily parallelized Label selector 10GB 1TB/s {app=~”.+”} |= “12.34.56.78” index 100TB Slow queries…

11. +

12. Rate-limiting!

13. Rate-limiting => decreased throughput => slower queries => frustrated users => SLO budget burn alerts => operator toil

14. Two choices: query less or cache more

15. Choice A: query less improve labels secondary indexes chunk compaction smarter query engine

16. The problem: this shit takes time

17. Choice B: cache more buy time to do the right thing™ easier lever to pull shorter delivery timeline

18. Caching

19. ingester logs 192 .168.1.1 - - [06/16/2023:06:09:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.74.0" 192.168.1.2 - - [06/16/2023:06:09:06 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "wget/1.21.1" 192.168.1.3 - - [06/16/2023:06:09:07 +0000] "GET /about.html HTTP/1.1" 200 396 "-" "Safari/15.4" 192.168.1.4 - - [06/16/2023:06:09:08 +0000] "GET /contact.html HTTP/1.1" 200 299 "-" "Firefox/92.0" 192.168.1.5 - - [06/16/2023:06:09:09 +0000] "GET /blog/index.html HTTP/1.1" 200 824 "-" "Chrome/92.0.4515.131" 192.168.1.6 - - [06/16/2023:06:09:10 +0000] "GET /blog/post-1.html HTTP/1.1" 200 543 "-" "Opera/12.15" 192.168.1.7 - - [06/16/2023:06:09:11 +0000] "GET /blog/post-2.html HTTP/1.1" 200 456 "-" "Edge/102.0.1245.153" 192.168.1.8 - - [06/16/2023:06:09:12 +0000] "GET /blog/post-3.html HTTP/1.1" 200 369 "-" "IE/11.0" {app=”nginx”, pod=”nginx-a”, az=”us-east-1a”} 192.168.1.1 - - [06/16/2023:06:09:13 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.74.0" 192.168.1.2 - - [06/16/2023:06:09:14 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "wget/1.21.1" 192.168.1.3 - - [06/16/2023:06:09:15 +0000] "GET /about.html HTTP/1.1" 200 396 "-" "Safari/15.4" 192.168.1.4 - - [06/16/2023:06:09:16 +0000] "GET /contact.html HTTP/1.1" 200 299 "-" "Firefox/92.0" 192.168.1.5 - - [06/16/2023:06:09:17 +0000] "GET /blog/index.html HTTP/1.1" 200 824 "-" "Chrome/92.0.4515.131" 192.168.1.6 - - [06/16/2023:06:09:18 +0000] "GET /blog/post-1.html HTTP/1.1" 200 543 "-" "Opera/12.15" 192.168.1.7 - - [06/16/2023:06:09:19 +0000] "GET /blog/post-2.html HTTP/1.1" 200 456 "-" "Edge/102.0.1245.153" 192.168.1.8 - - [06/16/2023:06:09:20 +0000] "GET /blog/post-3.html HTTP/1.1" 200 369 "-" "IE/11.0" {app=”nginx”, pod=”nginx-b”, az=”us-east-1a”} 192.168.1.1 - - [06/16/2023:06:09:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.74.0" 192.168.1.2 - - [06/16/2023:06:09:06 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "wget/1.21.1" 192.168.1.3 - - [06/16/2023:06:09:07 +0000] "GET /about.html HTTP/1.1" 200 396 "-" "Safari/15.4" 192.168.1.4 - - [06/16/2023:06:09:08 +0000] "GET /contact.html HTTP/1.1" 200 299 "-" "Firefox/92.0" 192.168.1.5 - - [06/16/2023:06:09:09 +0000] "GET /blog/index.html HTTP/1.1" 200 824 "-" "Chrome/92.0.4515.131" 192.168.1.6 - - [06/16/2023:06:09:10 +0000] "GET /blog/post-1.html HTTP/1.1" 200 543 "-" "Opera/12.15" 192.168.1.7 - - [06/16/2023:06:09:11 +0000] "GET /blog/post-2.html HTTP/1.1" 200 456 "-" "Edge/102.0.1245.153" 192.168.1.8 - - [06/16/2023:06:09:12 +0000] "GET /blog/post-3.html HTTP/1.1" 200 369 "-" "IE/11.0" {app=”nginx”, pod=”nginx-c”, az=”us-east-1b”} 192.168.1.1 - - [06/16/2023:06:09:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.74.0" 192.168.1.2 - - [06/16/2023:06:09:06 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "wget/1.21.1" 192.168.1.3 - - [06/16/2023:06:09:07 +0000] "GET /about.html HTTP/1.1" 200 396 "-" "Safari/15.4" 192.168.1.4 - - [06/16/2023:06:09:08 +0000] "GET /contact.html HTTP/1.1" 200 299 "-" "Firefox/92.0" 192.168.1.5 - - [06/16/2023:06:09:09 +0000] "GET /blog/index.html HTTP/1.1" 200 824 "-" "Chrome/92.0.4515.131" 192.168.1.6 - - [06/16/2023:06:09:10 +0000] "GET /blog/post-1.html HTTP/1.1" 200 543 "-" "Opera/12.15" 192.168.1.7 - - [06/16/2023:06:09:11 +0000] "GET /blog/post-2.html HTTP/1.1" 200 456 "-" "Edge/102.0.1245.153" 192.168.1.8 - - [06/16/2023:06:09:12 +0000] "GET /blog/post-3.html HTTP/1.1" 200 369 "-" "IE/11.0" {app=”nginx”, pod=”nginx-d”, az=”us-east-1b”} Object Storage 2023-06-16 11:02.123456789 2023-06-16 11:12.123456789 2023-06-16 11:02.123456789 2023-06-16 11:12.123456789 2023-06-16 11:02.123456789 2023-06-16 11:12.123456789 2023-06-16 11:02.123456789 2023-06-16 11:12.123456789 00ba6fb0 14a8d9b3 94d10ab7 72447153 Ingestion chunks

20. Object Storage 00ba6fb0 72447153 {app=”nginx”} {az=”us-east-1a”} Querying Query Engine

21. Recency bias

22. 200 instances @ 1 vCPU, 6GB RAM Total capacity: ~1.2TB RAM running on shared n2-standard-32 nodes

23. Success metric: hit rate Looks kinda… good , right?

24. Problem: churn Items being evicted before being fetched even once!

25. How much cache do we need?

26. How much cache do we need?

27. How much cache do we need?

28. ~50TB cache ~500Gbps throughput

29. The challenge: can we do it cost-effectively, whilst maintaining performance, in an operationally-familiar way?

30. CPU: <list> / 2 / <cpus> = ~$17.72/vCPU/month RAM: <list> / 2 / <gb> = ~$4.43/GB/month 200 vCPUs => $3544 1200 GB RAM => $5316 => $8860 per month 200 instances @ 1 vCPU, 6GB RAM Total capacity: ~1.2TB RAM running on shared n2-standard-32 nodes

31. CPU: <list> / 2 / <cpus> = ~$17.72/vCPU/month RAM: ~$4.43/GB/month 200 vCPUs => $3544 1200 GB RAM => $5316 => $8860 per month 200 instances @ 1 vCPU, 6GB RAM Total capacity: ~1.2TB RAM running on shared n2-standard-32 nodes

32. The goal: drive down cost per GB

33. Use memcached …with SSDs!

34. memcached “extstore” the industry’s best-kept secret

35. keys value s DRAM SSD extstore

36. memcached <other flags> --extended ext_path=/mnt/disks/ssd0/datafile:345G Using Extstore

37. memcached <other flags> --extended ext_path=/mnt/disks/ssd0/datafile:345G, ext_path=/mnt/disks/ssd1/datafile:345G, ext_path=/mnt/disks/ssd2/datafile:345G... Using Extstore

38. SSDs in Cloud?

39. Physically attached to hypervisor No network involved

40. “Local SSDs” 375GB each @ $30 /month Add up to 24 SSDs to most machine types 660MB/s reads, 350MB/s writes

41. “Instance Storage” Varies by instance type Included in cost We use im4gn or i4i

42. TTL TTV: Time-to-value 2 weeks!

43. CPU: <list> / 2 / <cpus> = ~$13.08/vCPU/month RAM: <list> / 2 / <gb> = ~$13.08/GB/month SSD: $30 per disk (375GB) = $0.08/GB/month 198 vCPUs => $2590 165 GB RAM => $2158 132 SSDs => $3960 => $8708 per month 33 instances @ 6 vCPU, 5GB RAM, 4 SSDs Total capacity: ~50TB SSD, 528Gbps running on dedicated n2-highcpu-8 nodes (16Gbps each)

44. CPU: <list> / 2 / <cpus> = ~$13.08/vCPU/month RAM: <list> / 2 / <gb> = ~$13.08/GB/month SSD: $0.08/GB/month 198 vCPUs => $2590 165 GB RAM => $2158 132 SSDs => $3960 => $8708 per month 33 instances @ 6 vCPU, 5GB RAM, 4 SSDs Total capacity: ~50TB SSD, 528Gbps running on dedicated n2-highcpu-8 nodes (16Gbps each)

45. CPU: <list> / 2 / <cpus> = ~$17.72/vCPU/month RAM: <list> / 2 / <gb> = ~$4.43/GB/month 200 vCPUs => $3544 1200 GB RAM => $5316 => $8860 per month 200 instances @ 1 vCPU, 6GB RAM Total capacity: ~1.2TB RAM running on shared n2-standard-32 nodes

46. Google Cloud Storage Amazon S3 $5 per million writes $0.005 per 1000 $0.4 per million reads $0.0004 per 1000 Object Storage No charge for bandwidth within the same region

47. 65% reduction in object storage reqs using memcached + 50TB of SSD ~ 2% overall TCO reduction No degradation in performance rollout

48. Cache effectiveness TB/d rollout

49. Cache effectiveness rollout

50. Query performance P50 P99 rollout

51. Trade-offs

52. Latency P50 rollout

53. Latency P50 rollout

54. Latency

55. Latency

56. Durability vs volatility

57. Disk-related issues Disks fill up Sectors fail Performance degrades

58. Solution: Kill the disk!

59. Observability Prometheus memcached-exporter node-exporter Alertmanager Disk full alert Disk latency alert Memcached latency alert Grafana & Loki (shocker!)

60. Implications for the future of cloud databases? Cost is becoming compelling Capacious disks at near-DRAM speeds Higher network throughput Getting higher as network is offloaded from CPU

61. Thanks! Questions?

62. Why not Memorystore / Elasticache?