Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips

ScyllaDB monitoring provides a lot of useful information. But sometimes it’s not easy to find the root of the problem if something is wrong or even estimate the remaining capacity by the load on the cluster. This talk shares our team's practical tips on: 1) How to find the root of the problem by metrics if ScyllaDB is slow 2) How to interpret the load and plan capacity for the future 3) Compaction strategies and how to choose the right one 4) Important metrics which aren’t available in the default monitoring setup.

1. Getting The Most Out of ScyllaDB Monitoring: Rarely Mentioned Debugging and Tuning Tips Andrei Manakov, Staff Software Engineer at ShareChat

2. Andrei Manakov More than 13 years experience in industry Designed and developed multiple highload projects Passionate about performance problems in distributed systems Developing TikTok-like app with more than 20m DAU

3. Moj Overview Reconnection Storms Compaction Strategies Cluster Capacity Presentation Agenda

4. Moj Overview

5. Moj, a short video app

6. Feed Feature Store

7. (Re)Connection Storms

8. What Happened?

9. Internal CQL Inserts on Some Shards

10. (Re)Connection Storm fd sum(rate(scylla_transport_cql_connections)) by (instance)

11. Driver Hygiene (golang)

12. Driver Hygiene Rules (golang)

13. Still (Re)connection Storm.. fd https://github.com/scylladb/gocql/issues/124

14. Driver Hygiene Rules (golang)

15. Still Problems.. fd https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_next_upstream

16. Compaction Strategies

17. Compaction Strategies

18. Disks Reads/Writes fd sum(rate(node_disk_reads_completed_total[5m])) fd sum(rate(node_disk_writes_completed_total[5m]))

19. C ompaction CPU Usage fd avg(rate(scylla_scheduler_runtime_ms{group="compaction"}[2m])) by (group)/10

20. How Does ScyllaDB Reads?

21. SSTable Read s fd sum(rate(scylla_sstables_single_partition_reads[5m]))

22. ScyllaDB Cache Misses fd sum(rate(scylla_cache_reads_with_misses[5m]))

23. SSTable Read Efficiency fd sum(rate(scylla_sstables_single_partition_reads[5m])) / sum(rate(scylla_cache_reads_with_misses[5m]))

24. Real Life Situation

25. Cluster Capacity

26. Why Load is Not Really U seful ? fd https://github.com/scylladb/scylla-monitoring/issues/2003

27. CPU Capacity Analysis fd max(sum(rate(scylla_scheduler_runtime_ms{group!="compaction"}[2m])) by (instance, shard))/10 fd avg (sum(rate(scylla_scheduler_runtime_ms{group!="compaction"}[2m])) by (instance, shard))/10

28. Cache fd 1-sum(rate(scylla_cache_reads_with_misses[5m])) / sum(rate(scylla_cache_reads[5m]))

29. Disk Bytes Written/Read fd sum(rate(node_disk_written_bytes_total[5m])) fd sum(rate(node_disk_read_bytes_total[5m]))

30. Disk I/O and Bandwidth Capacity Capacity in io_properties.yaml disks: - mountpoint: /var/lib/scylla/data read_iops: 2400000 read_bandwidth: 5921532416 write_iops: 1200000 write_bandwidth: 4663037952

31. Conclusion disks: - mountpoint: /var/lib/scylla/data read_iops: 2400000 read_bandwidth: 5921532416 write_iops: 1200000 write_bandwidth: 4663037952

32. Stay in Touch Andrei Manakov [email protected] @AndreyManakov https://www.linkedin.com/in/andrei-manakov-69228a81/