Real-Time or Analytics Workloads... Why Not Both?

ScyllaDB’s Workload Prioritization provides resource optimization and performance isolation across workloads with different performance needs, such as Analytics and Real-time. In this session, you will learn how Workload Prioritization works, how you can use it to run different types of workloads together under a single ScyllaDB cluster, and how to fine-tune priorities and resource allocation based on your specific requirements.

1. Real-Time or Analytics Workloads... Why Not Both? Felipe Cardeneti Mendes, Solution Architect at ScyllaDB

2. Felipe Mendes Published Author ScyllaDB Committer IT Specialist & Solution Architect Open Source Enthusiast Your photo goes here, smile :)

3. ScyllaDB as an Analytics Engine

4. An IoT Application Total amount of data points 526 billion temperature readings 1 ,000,000 sensors, representing homes in an area 365 days (1 year storage requirement) 1 reading per minute

5. Analytics over the entire data? How long would it take at normal speeds? We need more if analytics are a part of the pipeline W e need Scylla DB 200,000 points/second 730 hours (30 days) 1 million points/second 146 hours (almost a week)

6. Easy Peasy Scanning 3 months of data Finished Scanning. Succeeded 132,480,000,000 rows. Failed 0 rows. RPC Failures: 0. Took 110,892.91 ms Processed 1,194,666,083 rows/s Absolute min: 19.71, date 2019-08-27, sensorID 473869 Absolute max: 135.21, date 2019-08-27, sensorID 473869

7. Easy Peasy Scanning the entire dataset Finished Scanning. Succeeded 525,599,474,400 rows. Failed 0 rows. RPC Failures: 0. Took 542,191.31 ms Processed 969,398,554 rows/s Absolute min: 68.00, date 2019-05-28, sensorID 82114 Absolute max: 79.99, date 2019-03-19, sensorID 152594

8. We can efficiently process over 1.2 billion points per second (we’ll process whatever you need, too!)

9. Concurrency Challenges

10. ‹#›

11. P99 climbs to unacceptable values The Latency Problem

12. Throughput gradually drops, load distribution becomes unfair The Throughput Problem

13. Why Contention Happens? Primarily lack of system resources (disk I/O, CPU time) Not necessarily a problem Introduces queueing

14. Easy then, let's simply ... Addressing the Problem? Divide and conquer! Division in Space (Multi DC) Division in time (off peak OLAP)

15. Workload Prioritization

16. Isolation and Performance Optimization Background Tasks User Tasks

17. Shares Different workloads require different priorities Meet SLAs Flexible Configuration Adaptability to Changing Conditions

18. Getting Started Set up authentication, assign Roles to each workload Prioritization is wired on the authenticated users role Define your Service Levels For a primary workload: For a secondary one: Profit! CREATE SERVICE LEVEL main WITH shares = 200 CREATE SERVICE LEVEL secondary WITH shares = 600

19. Prioritization and Isolation are NOT enough!

20. Workload Characteristics

21. Workloads Characteristics: Time ‹#› The timeout dilemma: Timeout should follow: 𝑇𝑠𝑒𝑟𝑣𝑒𝑟 ≤ 𝑇𝑐𝑙𝑖𝑒𝑛𝑡 For Real-time : Can’t be too high Incurs retries or dropped requests Excessive retries result in wasted resources For Analytics: Can’t be too low Otherwise Batch will likely fail High throughput will typically increase latencies due to contention

22. Workloads Characteristics: Shedding ‹#› Overload response: Interactive workload: Throttling won't help Delaying response to user A will not cause some user B to stop sending requests Unbound concurrency Batch workload: Just throttle Allow us to have a knob that controls the pace of the analytics workload Bounded concurrency

23. Introducing Workload Characterization ‹#› Ideally – we want the database to behave differently: For Real-time: Have low timeout Load shedding (fail excessive requests), as the database can NOT slow down interactive workloads. Dedicate most of the resources to this workload. For Batch: Relatively higher timeout Apply back-pressure via throttling Use mostly unused resources

24. For Real-Time: For Analytics: Why not just hint the database with specifics? Introducing Workload Characterization ‹#› Have low timeout (30ms) timeout=30ms Load shedding AND workload_type=interactive Dedicate most of the resources AND shares=800 Have relatively high timeout (5s) timeout=5s Throttling AND workload_type=batch Use mostly unused resources AND shares=200

25. Takeaways ‹#› ScyllaDB powers both Analytics & Real-time intensive workloads Workload Prioritization helps with: Infrastructure Consolidation Resource Optimization Performance Isolation Workload Characterization provide: Workload specific settings Distinct overload and timeout responses

26. Stay in Touch Felipe Mendes [email protected] @cardeneti82118 fee-mendes Find me on LinkedIn