Cost-Efficient Stream Processing with RisingWave and ScyllaDB

A technical look at how to achieve cost-efficient stream processing with ScyllaDB and RisingWave, an open-source distributed streaming database.

1. Cost-efficient Stream Processing with RisingWave and ScyllaDB Yingjun Wu, CEO at RisingWave

2. Yingjun Wu Chief Everything Officer Ex-AWS Redshift Ex-IBM Research Almaden PhD in databases and stream processing Your photo goes here, smile :)

3. State Management: Traditional Solutions MapReduce style Coupled compute and storage architecture

4. State Management: Traditional Solutions MapReduce style Coupled compute and storage architecture

5. State Management: Cloud-based Solution Cloud-native style Decoupled compute and storage architecture

6. State Management: Cloud-based Solution Cloud-native style Decoupled compute and storage architecture

7. State Management in the cloud Consider joining two data streams Impression stream Click stream

8. State Management in the cloud Consider joining two data streams Impression stream Click stream

9. State Management in the cloud Consider joining two data streams Impression stream Click stream

10. State Management in the cloud Consider joining two data streams Impression stream Click stream

11. Tiered Storage Log structured merge tree Recent data is cached in local machine Older data is perically compacted to lower levels

12. Comparison

13. State Management: Failure Recovery

14. State Management: Elastic Scaling

15. Demo Time! Extract, Transform, and Load (ETL) data from PostgreSQL to ScyllaDB 2 lines of code! Streaming ETL

16. Demo Time! Perform complex computation and then deliver data to ScyllaDB Streaming analytics Materialized view 1 Materialized view 2 Materialized view 3

17. Demo Time! Streaming analytics create materialized view mv1 as …; c reate materialized view mv2 as …; c reate materialized view mv3 as select … from mv1, mv2 …; create sink my_scylla as select … from mv3 with (connector = ‘cassandra’, …); Computation : Sink:

18. Modern companies need to gain insights from real-time data RisingWave + ScyllaDB can work collaboratively to reduce stream processing cost while attaining top performance Conclusion

19. Stay in Touch Yingjun Wu [email protected] yingjunwu yingjunwu https://www.linkedin.com/in/yingjun-wu/ risingwave.com/slack