Batch to near-realtime: inspired by a real production incident

This slide deck was used for the platformatory streams meetup in Bengaluru on July 7, 2024.

This is a real world account from an Apache Druid cluster in production. A story of 48 hours of debugging, learning and understanding batch vs stream better, filing a couple of issues in Druid open source projects and finally a stable production pipeline again thanks to the Druid community. We will discuss what parts of your design could be impacted, how you should change the related systems so the cascading failures don’t bring down your complete production availability. As an example, we will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how to mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our Grafana dashboards.

Finally we will list all the leanings and how we made sure we never repeat the same mistakes in production systems.

1. Batch (to) Near-Realtime Shivji kumar Jha, Staff Engineer, Nutanix Sachidananda Maharana, MTS 4, Nutanix

2. ABOUT US Software Engineer & Regular Speaker / Meetups Excited about: Distributed Databases & Streaming Open-Source Software & Communities MySQL / Postgres, Pulsar/NATS, Druid/Clickhouse Regular Platform Engineer Excited about: Distributed OLAP Databases Open-Source Enthusiast Shivji Kumar Jha Staff Engineer CPaaS Data Platform, Nutanix Sachidananda Maharana Software Engineer OLAP Ninja CPaaS Team, Nutanix

3. CONTENTS Background App Context The problem Statement Druid 101 Ingestion Issues & Fixes Query Issues & Fixes Fix in common libraries (OSS) Be a good citizen!

4. BACKGROUND: Application CONTEXT UI based slice and dice analytics with filters Easy Ops: One Multi-tenant DB (Druid) Cluster Fine grained isolation per customer / use-case Resilient Pipeline: Temporal for orchestration Durable / Scalable Storage: S3 Java Workers with Postgres storage for state.

5. BACKGROUND: APP ingestion pipeline

6. EARLIER: old batch system 3 hrs

7. NOW: near-real-time INGESTION nudge State Machine, absorb backpressure Cron : 5 mins

8. Batch to near-real-time system Cron : 5 mins Ingestion V 1 I ngestion V2 Druid Ingestion Tasks Big Tasks Many small tasks

9. ANY IDEAs how to handle THis ? https:// www.dreamstime.com /stock-image-diversity-show-hands-image9264971

10. IDEAS? Scale [UP / OUT] resources? Isolate tasks [ingest / compact / query]? Parallelize? Choose simple performant data assignment algos? Throttle & throw defined error codes Global Queues Per customer

11. MORE IDEAS: Compaction for better Query! A lot of task means a lot of small files During query, reading from lots of small files not efficient S3 also prefers smaller number of bigger files. Solution: Merge files in the background Enter Compaction: Combine smaller files into rolled up (fewer) bigger files. Reduce the granularity as data gets old! Example: Minute => hourly => daily => weekly… Can’t graphs for huge time at less granularity. Saves disk space, network throughput, latency

12. SUMMARY: Change in Requirements Change in Requirement: Batch (3 hours) to 5 minutes Earlier: Agent collects data, dumps to S3. Cron runs every 3 hour, ingests from S3 to Druid SLA : 3 hours New Design: SLA : 10 minutes Agent collects data, dumps to S3 every 5 minutes. Ingestion Pipeline ingests to Druid depending on what Druid likes. Ingestion Pipeline gobbles backpressure. Release Plan Data sources uploaded to cluster in a phased manner

13. Heard of DRUID? https:// www.dreamstime.com /stock-image-diversity-show-hands-image9264971

14. DRUID 101

15. DRUID 101 Write Read

16. Peon Processes(Slots) Segments(ready to query)

17. Druid Nos : 4+ years in Prod Cluster size Last 24 hrs

18. INGESTION ISSUEs

19. Deploying new design

20. Customer 1 Deploy for 1 customer

21. Customer 1 Customer 3 Customer 2 Deploy for more

22. Customer 1 Customer 3 Customer 2 Customer N Too many tasks Deploy for all… booom !

23. Customer 1 Customer 3 Customer 2 Customer N Too many tasks Deploy for all… booom !

24. Customer 1 Customer 3 Customer 2 Customer N Too many tasks No tasks Chill life Deploy for all… booom !

25. Proof of the Pudding!

26. Proof of the Pudding! 2

27. Proof of the Pudding! 3

28. Summary: Druid INGESTION struggling Ingested smaller, but more tasks. O nboarding a few large customers, all good for a day! More confidence  Onboarded all customers at once Ingest task queue kept piling up (till 25K), overwhelmed after 5K Soon, overlord (ingestion orchestrator) node CPU usage at 100%  All the tasks stuck in pending state Task count was 12x more than previous but smaller. Ingest tier nodes (MM) were sitting idle, no incoming tasks.  Ingestion task state not updating as overlord was overwhelmed. Druid Overlord

29. Get INGESTION ORCHESTRATOR Alive

30. Overlord Process Ingest tasks not Being assigned!!

31. Overlord Process Bigger VM Scale up VM (AWS EC2)

32. Overlord Process Bigger VM Bigger DB Also, scale up DB

33. Handling the Overlord… No support for horizontal scalability Only active/passive. Vertically scale overlord instance as well as its Postgres DB capacity. Optimize the Changed Druid process configs to optimize task assigment !

34. Handling the Overlord… Optimize the Changed Druid process configs to optimize task assigment ! Restrict Global task queue & Throttle! Set max pending tasks per customer to 1 Throttle, Don’t give up druid.indexer.queue.maxSize : 5000 GET /druid/indexer/v1/ pendingTasks?datasource =ds1

35. SCALE UP POSTGRES DB Queries to Postgres for task were taking long time. Add more CPU to DB server Overlord CPU utilization is less Number of pending tasks are less

36. Scaling INGESTION TIER

37. Middle managers Peon Processes More VMs

38. Middle managers Peon Processes More VMs

39. Middle managers Peon Processes Bigger Compact tasks More VMs Tiering Middle Managers

40. Middle managers More Slots Bigger Compact tasks More VMs More slots per Middle Managers

41. Middle managers More Slots Bigger Compact tasks Less VMs Right size Middle Managers

42. Summary : Scaling Middle Manager Increased number of middle manager as so that more task slots are available for overlord to assign tasks. Then we increased number of slots per middle manager as new tasks were small i.e. having less number of files to ingest. We created a separate tier for compaction as these tasks took more resource then the current index tasks. Then we right sized the middle manager count in each tier by reducing it. 12 MMs * 5 slots => 24MMs * 5 slots 24 MMs * 5 slots => 12MMs * 10 slots 12 MMs * 10 slots => 10 MMs * 10 slots + 2 MMs *5 slots Tiering

43. ISSUES IN QUERY TIER

44. Too many segments to assign !!!

45. Too many segments to assign !!!

46. Coordinator Process

47. Coordinator Process Bigger VM

48. Coordinator Process Bigger VM Same big DB

49. Handling the Coordinator… Increased Coordinator instance type as it is not scalable horizontally Tried the following coordinator dynamic configs:

50. Handling the Coordinator… Increased Coordinator instance type as it is not scalable horizontally Use the dynamic configs. Restart on scale is hard! reducing the number of segments per coordinator cycle maxSegmentsToMove : 1000 percentOfSegmentsToConsiderPerMove : 25

51. Handling the Coordinator… Increased Coordinator instance type as it is not scalable horizontally Tried the following coordinator dynamic configs: Assign segments In round-robin fashion first. Lazily reassign with chosen balancer strategy later maxSegmentsToMove : 1000 percentOfSegmentsToConsiderPerMove : 25 useRoundRobinSegmentAssignment : true

52. Finally, the QUERY Nodes

53. Handling the Historicals Until auto compaction done: More no of segments for queries More resources to power query tier Beware: More segments = more cache to load Check for linux configs max_map_count Query for recent data Query for recent data Older Historicals Current Historicals Larger segments Smaller segments Datasource 2 Datasource 1 Datasource 1 Datasource 2

54. FILE ISSUES in OSS

55. FILE ISSUES in OSS

56. Fix Issues TOO IF possible!

57. Happy State!!!

58. Thank You Questions Shivji Kumar Jha linkedin.com/in/shivjijha/ slideshare.net/shiv4289/presentations/ youtube.com/@shivjikumarjha Sachidananda Maharana https://www.linkedin.com/in/sachidanandamaharana/ Liked this? Checkout our upcoming meetups!