Pushing Your Streaming Platform to the Limit by Elad Leev

Join Elad for a hands-on session on Chaos Engineering for streaming platforms like Kafka, Pulsar, NATS, and RabbitMQ. Learn to stress test, benchmark, and fine-tune performance to ensure your system stays resilient under pressure.

1. A ScyllaDB Community
Pushing Your Streaming
Platform to the Limit
Elad Leev
Staff Engineer

2. Your photo
goes here,
smile :)
Elad Leev
#DistributedSystems #DataStream
#Scalability #DataMesh #Kafka
#Flink

4. No single metric can measure the performance of computer systems on all
applications. System performance varies enormously from one application
domain to another. Each system is typically designed for a few problem domains
and may be incapable of performing other tasks.
“
“

5. … The system that does the job with the lowest cost-of-ownership.
“
“

6. Your Vendor’s
Benchmarks
=
Marketing
Job

7. Know Your System Limits

8. Disk
Memor
y
CPU
Networ
k

9. The Key Criteria for a Good
Benchmark

10. The benchmark should test the exact same
deployment and configuration as we planned to use
in Production.
No special tricks. Environment Simulation

11. Conduct advanced research to understand your
traffic characteristics and SLAs/SLOs. Always
prepare for the unexpected.
Aim for the peak. Environment Simulation
Test peak performance

12. Failures are a given and everything will eventually fail over time: from routers to
hard disks, from operating systems to memory units corrupting TCP packets,
from transient errors to permanent failures. This is a given, whether you are
using the highest quality hardware or lowest cost components.
“
“
Werner Vogels, AWS CTO, 2016

13. Our system is a living organism; it scales up and
down. Embrace the change, know the impact.
Make it scalable. Environment Simulation
Test peak performance
Scalable & Portable

14. The benchmark must be understandable to maintain
credibility.
Environment SimulationSimplicity is key.
Test peak performance
Scalable & Portable
Transparent Design

15. What to look for?

16. For every resource in the system, we should check:

Utilization

Saturation

Errors
The USE Method

17. system, user, idle
loadavg
buffers, cache, mem-free,
mem-used
JVM heap, GC
bytes in-out, package
drops
io-ms, io-wait,
QueueLength

18. How to Benchmark

20. Open Messaging
Initiative
https://openmessaging.cloud

21. OMB
Driver
OMB
Worker
Task and Topic Orchestrator The Benchmark Executor

22. Kubernetes-based solution that's easy to scale. Runs
as a simple Producer/Consumer model, matching
our service architecture.
How to
Benchmark?

23. Kafka
Pulsar

24. Kafka
Pulsar

25. Kafka
Pulsar

26. Kafka
Pulsar

27. Kafka
Pulsar

28. Workload File

29. Workload File

31. Benchmark Insights

32. Average
Latency

33. Average
Latency
Test Our
Monitoring &
Dashboards

34. Average
Latency
Test Our
Monitoring &
Dashboards
Potential
Bottlenecks

35. Average
Latency
Test Our
Monitoring &
Dashboards
Potential
Bottlenecks
Scale-up
Estimates

36. Average
Latency
Test Our
Monitoring &
Dashboards
Potential
Bottlenecks
Scale-up
Estimates
Consumer/Producer
Configuration
Guidelines

37. Stay in Touch
Elad Leev
https://leevs.dev
@eladleev
@EladLeev
linkedin.com/in/elad-leev