Tracking Millions of Heartbeats on Zee's OTT Platform

Learn how Zee uses ScyllaDB for the Continue Watch and Playback Session Features in their OTT Platform. Zee is a leading media and entertainment company that operates over 80 channels. The company distributes content to nearly 1.3 billion viewers over 190 countries.

1. Tracking Millions of Heartbeats on Zee’s OTT Platform Srinivas Shanmugam , Principal Architect at ZEE5 Jivesh Threja , Tech Lead at ZEE5

2. About Zee5 Zee5 is an India-based OTT platform with over 500 thousand hours of on-demand content and 100M+ app downloads

3. Srinivas Shanmugam 20+ Yrs Of IT industry experience 9 + Yrs of experience as architect in large-scale software systems One of founding employee of charmboard built video discovery and commerce product from scratch for OTT platforms.

4. Jivesh Threja Tech Lead who loves to work and design complex use cases 8+ years of technical expertise in DRM, Blockchain, and Streaming Redesigned and redeveloped the entire heartbeat API for Zee5

5. Heartbeat Architecture Design Metadata Management Building Continue Watch Feature Concurrent Viewership Count Agenda

6. About This Topic A heartbeat is a request fired at regular intervals during video playback We help customers to continue from the exact point where user have left off the content Get Active viewership Count on real time basis To view previously watched contents

7. High Level Design

8. Architectural Goals Remove Dependency of Cloud Vendor Specific Architecture Open Source Cost optimisation High Throughput and Low Latency Multi Tenant Ready Easier to ship and launch new use cases Inserting New Episode in Continue Watching Rail Concurrent Viewership Count

9. Evaluated Multiple Databases

10. Why We Opted For ScyllaDB ? Persistent and In-memory DB Low Resource Consumption Low Latency Cross Region Replication Multi Cloud Ready Managed Support

11. Old Architecture Heartbeat Architecture in Cloud Vendor Specific Platform

12. New Architecture Heartbeat Architecture in Cloud-Agnostic Platform

13. Cost Comparison(Per Month) DynamoDB RDS (Postgresql) Redis Solr ScyllaDB $62000 $12000 5x

14. Low Level Design

15. Real Time Stream Processing Pipeline

16. Scalable API Layer

17. Metadata Management Managing metadata of such a large OTT platform was the first challenge we faced Solr and Redis were being used to manage large metadata This was strategically designed and replaced by ScyllaDB ScyllaDB provides faster queries by unique partitions along with materialized views for special use cases like new episode insertion; it proved to be more efficient than Redis and Solr

18. MetaData Model create keyspace.meta_data ( id text, title text, show_id text, …, …, PRIMARY KEY((id),show_id) ) with compaction = {‘class’: ‘LeveledCompactionStrategy’ }; Choosing the right keys and right clustering keys for further views helped us to get a single digit millisecond read latency.

19. Continue Watching Use Case Millions of users watching content at the same time Pausing and continuing on a different device from where they left off Needed a single digit millisecond API latency for the PUT and the GET

20. Viewership Count Use Case This is another additional interesting use case where we effectively used the concept of Time Compaction Strategy in ScyllaDB This helps us to get the concurrent viewership count per content and per user

21. Viewership Count Data Model CREATE TABLE keyspace.USER_SESSION_STREAM ( USER_ID text, DEVICE_ID text, ASSET_ID text, TITLE text, …, PRIMARY KEY((USER_ID), ASSET_ID) ) WITH default_time_to_live = 60 and compaction = { 'class' : 'TimeWindowCompactionStrategy' }; Populated on every heartbeat and cleared with the TTL

22. Load Test Report P99 latency of both Read and write operation is less than 4 ms at 41k QPS Average write latency Average 99% write latency Average Read latency Average 99% Read latency Requests / Sec

23. Interesting Facts Total Size Of ScyllaDB : 9 TB Daily Peak Concurrent Viewership : >1 Million Every Day Total HeartBeats Processed Per Day : > 100 Billion

24. Learnings Data modelling is the KEY to achieve single digit latencies in ScyllaDB Use the right Quorum and Compaction Strategies Choose partition and clustering keys wisely Use materialized views for fast lookup of data Avoid using filter queries Avoid Non Prepared statements Use Async queries for faster processing Use zone aware clients to avoid network bandwidth cost between AZ

25. Stay in Touch Srinivas Shanmugam [email protected] https://www.linkedin.com/in/srinivasshanmugam/ Jivesh Threja [email protected] @jthreja jivesh-threja https://www.linkedin.com/in/jthreja/