Mitigating the Impact of State Management in Cloud Stream Processing Systems

Stream processing is a crucial component of modern data infrastructure, but constructing an efficient and scalable stream processing system can be challenging. Decoupling compute and storage architecture has emerged as an effective solution to these challenges, but it can introduce high latency issues, especially when dealing with complex continuous queries that necessitate managing extra-large internal states.

In this talk, we focus on addressing the high latency issues associated with S3 storage in stream processing systems that employ a decoupled compute and storage architecture. We delve into the root causes of latency in this context and explore various techniques to minimize the impact of S3 latency on stream processing performance. Our proposed approach is to implement a tiered storage mechanism that leverages a blend of high-performance and low-cost storage tiers to reduce data movement between the compute and storage layers while maintaining efficient processing.

Throughout the talk, we will present experimental results that demonstrate the effectiveness of our approach in mitigating the impact of S3 latency on stream processing. By the end of the talk, attendees will have gained insights into how to optimize their stream processing systems for reduced latency and improved cost-efficiency.

1. Mitigating the Impact of State Management in Cloud Stream Processing Systems Yingjun Wu Founder at RisingWave Labs

2. Yingjun Wu ( he/him/his ) Founder at RisingWave Labs Chief Everything Officer Ex-AWS Redshift Ex-IBM Research Almaden PhD in database systems and stream processing

3. What is RisingWave? A distributed SQL streaming database Open sourced in April 2022 under Apache 2.0 License >5K GitHub stars >130 GitHub contributors >1K Slack members >100K K8s deployments

4. Background

5. Background

6. Background

7. Background

8. Stream processing systems continuously process large volume of data Background

9. Stream processing systems continuously process large volume of data Need to maintain internal states for stateful operators such as joins and aggregations Background

10. Consider joining two data streams Impression stream Click stream State Management

11. Consider joining two data streams Impression stream Click stream State Management

12. Consider joining two data streams Impression stream Click stream State Management

13. State Management: Existing Solutions MapReduce style Compute-storage coupled

14. MapReduce style Compute-storage coupled State Management: Existing Solutions

15. State Management: Existing Solutions MapReduce style Compute-storage coupled

16. State Management: Existing Solutions MapReduce style Compute-storage coupled

17. State Management: the Cloud Era Cloud -native style Compute-storage decoupled

18. State Management: the Cloud Era Cloud-native style Compute-storage decoupled +

19. Consider joining two data streams Impression stream Click stream State Management in the Cloud

20. Consider joining two data streams Impression stream Click stream State Management in the Cloud

21. Consider joining two data streams Impression stream Click stream State Management in the Cloud

22. Consider joining two data streams Impression stream Click stream State Management in the Cloud

23. How to mitigate the issue?

24. Tiered Storage EC2 local storage: the “cloud cache” Super fast! Data will get lost if machine crashes EBS: the “cloud disk” Fast 99.999% durability (5 nines) S3: the persistent storage Slow 99.999999999% durability (11 nines)

25. Tiered Storage EC2 local storage: the “cloud cache” Super fast! Data will get lost if machine crashes EBS: the “cloud disk” Fast 99.999% durability (5 nines) S3: the persistent storage Slow 99.999999999% durability (11 nines)

26. Tiered Storage Should we really leverage EBS?

27. Tiered Storage Should we really leverage EBS? No, at least for now…

28. Tiered Storage Should we really leverage EBS? No, at least for now… EC2 local storage vs EBS EBS is more expensive than EC2 local storage EBS is slower than EC2 local storage EBS is persistent while EC2 local storage is not

29. Tiered Storage How to manage data at different layers?

30. Log Structured Merge Tree Use log structured merge tree! Recently accessed data will be cached in local machine Upper level runs will be periodically compacted to lower levels

31. Compaction Compactions in LSM trees can cause huge latency spikes!

32. Compaction Compactions in LSM trees can cause huge latency spikes! Move the compaction to remote machines

33. Compaction Compactions in LSM trees can cause huge latency spikes! Move the compaction to remote machines

34. Compaction Compactions in LSM trees can cause huge latency spikes! Move the compaction to remote machines Build SST Upload SST

35. Compaction Compactions in LSM trees can cause huge latency spikes! Move the compaction to remote machines Build SST Upload SST

36. Compaction Accessing S3 is always expensive!

37. Compaction Accessing S3 is always expensive!

38. Compaction Accessing S3 is always expensive! S3 buckets

39. Compaction Accessing S3 is always expensive! Each bucket has performance limit! S3 buckets

40. Compaction Accessing S3 is always expensive! Each bucket has performance limit! Using too many buckets can cause fragmentation issues! We set #buckets to a magic number and scale based on workloads. S3 buckets

41. Conclusion State management is a challenging problem in stream processing systems. Decoupled compute-storage architecture helps achieve infinite and independent scalability. Tiered storage helps optimize performance. Using remote compaction and streaming compaction to optimize network. Set number of buckets wisely to reduce S3 access bottleneck.

42. Yingjun Wu @YingjunWu r isingwave.com/github risingwave.com/slack Thank you! Let’s connect.