Why We Chose ScyllaDB over DynamoDB for "User Watch Status"

Yichen Wei and Adam Drennan share the architecture and technical requirements behind "user watch status" for a major global media streaming service, what that meant for their database, the pros and cons of the many options they considered for replacing DynamoDB, why they ultimately chose ScyllaDB, and their lessons learned so far.

1. Why We Chose ScyllaDB over DynamoDB for "User Watch Status" Yichen Wei & Adam Drennan

2. Why choose ScyllaDB over others? The Story Reason behind the choice Options? SQL/NoSQL/NewSQL/DistSQL? Why ScyllaDB? The User Case Takeaways

3. Yichen Wei HA Distributed Platform domains including ML, microservices and HPC Evangelist HTAP, Rust and FP/CT WASM/WASI Your photo goes here, smile :)

4. Adam Drennan Backend software engineer with focus on HA, low-latency distributed systems Kotlin for backend evangelist Your photo goes here, smile :)

5. The User Story

6. The Story

7. Mickey: I want to resume the movie watched last time. Mickey: I want to know if I’ve seen this show before. The Story

8. The Story

9. User Experience First Speed Availability Scalability Fault tolerance Cloud first - AWS Reduced cost Design for future The Story and Goals

10. Multi-region 170k+ WPS Edge SLA - Nearline update Conditional updates based on event timestamp 78k+ RPS P99 Read latency < 10-20ms Simple point queries Large history data set > 10TB The Story and Goals

11. DB Choices

12. Choose the right DB is not easy https://dbdb.io/ 959+ databases Infra maintenance and upgrade - zero downtime, cloud International Distributed - Multi-region Multi-country Transaction, To be or not to be? SQL or noSQL? noSQL -> not only SQL CDC? … Options

13. Aurora? Aurora is good Disaggregated Storage and Compute Architecture Multi-Primary Write New Serverless(Wow) Not for us Write throughput Price Options

14. Why not DynamoDB? DynamoDB is good Multi-region Auto-scale Less maintenance Not for us Price SDK Options

15. Other databases ? NoSQL is - No SQL - 2010 Not only SQL - 2020 New/Dist SQL HTAP Scalability Not for us Write throughput Maintenance Options

16. We used to use C* heavily Delegated C* team Multi-region build-in clusters in On-Premise datacenters Multi-region build-in clusters in AWS How About Cassandra?

17. We used to use C* heavily But issues during maintenance and migration Spikes caused by GC -> GC tuning and upgrading Heterogeneous hardware in different datacenter/cloud -> Numa tuning CDC is complicated Didn’t utilize the modern hardwares -> RDMA, Numa aware … Many JVM related issues like above How About Cassandra?

18. DB Choice

19. DB Choice

20. Plan Create Scylla Cloud test cluster 6x i4i.4xlarge nodes with replication factor: 3 Pre-load cluster with 3 billion records Run combined load tests of 170k WPS and 78k RPS POC app uses Scylla shard-aware driver for Java Queries based on composite primary key Rely on custom query timestamps for event ordering Monitor metrics using Scylla Cloud Grafana dashboard Scylla Proof of Concept

21. Iterative Config Changes Working with Scylla Team Set speculative retry to 7ms Throttled max concurrent requests to 4 after Scylla team saw excessive queueing Final Results Able to hit combined load with no errors P99 read latency = 9ms P99 write latency < 1ms Scylla Proof of Concept

22. Features we like in ScyllaDB Performance Seastar -> C++, no GC, Numa aware ICS(Incremental Compaction Strategy) Shared nothing architecture - the scylla driver Flexible CL and RF for single/multi region(s) Leaderless - maintenance w/o end-user awareness DB Choice

23. Features we like in ScyllaDB WASM UDF Flexible Update Timestamp Active community DB Choice

24. Features we like in ScyllaDB Cloud Frugal architect - BYOC Zero maintenance Terraform DB Choice

25. Features we like in Scylla(Hope in Scylla Cloud in the future) DPDK support SAI Customized storage/index interface DB Choice

26. Choose a database is hard - One to rule them all Cloud makes is harder - Frugal architecture Serverless is not silver bullet Hardware will play an important role Takeaways

27. Thank you and Stay in Touch