Streaming vs batching (conundrum ai internal meetup)

Discussion about streaming & batching

1. Streaming vs Batching
Conundrum, May 2020
[email protected] — Mark Andreev

2. Agenda
-About data processing pipeline
-Batching approach
-Streaming approach
-Architecture
-Tools
-Conclusion
https://clck.ru/VQL9W

3. Data pipeline
-Event := (timestamp, payload)
-Pipeline := F(Event … Event ; state)

Examples:
-Web Banner CTR
-Forecast Ads Budget Consumption
-Fraud prevention
-Compute analytics over session windows
1 n
Batch
Latency - Hours
Streaming
Latency - Minutes/Seconds

4. Batching approach
-Fetch periodically
-Bounded stream approach
-Easy to write (look like SQL select/insert)
-Scheduled by standalone tool (like Airflow)

5. Streaming approach
-Fetch as soon as possible
-Unbounded stream approach
-Require KV storage for state

6. Streaming approach [Watermarks]
Event time
(when did it happen)
Processing time
(when did it fetched in streaming engine)
Ideal case
(no latency)
Outdated event
(fetched too late)
events older than watermark dropped
delay

7. Architecture [Lambda Architecture]
Data Source
Speed layer
Real time view
Batch layer
Pre compute view
Serve layer
https://clck.ru/VR4Lg

8. Architecture [Lambda Architecture]
Data Source
Speed layer
Real time view
Batch layer
Pre compute view
https://clck.ru/VR4Lg
Serve layer

9. Architecture [Lambda Architecture]
Data Source
Speed layer
Real time view
Batch layer
Pre compute view
https://clck.ru/VR4Lg
Serve layer

10. Architecture [Kappa Architecture]
Data Source
Speed layer
Real time view
Serve layer
https://clck.ru/VR4Bm

11. Architecture [Lambda vs Kappa]
Lambda Kappa
Batch + Streaming Streaming
Query all data Incremental algorithms on deltas
Batch is reliable
Streaming is approximate
Streaming with consistency
Two scripts for both approach Single script

12. Architecture [Lambda vs Kappa]
Lambda Kappa
Batch + Streaming Streaming
Query all data Incremental algorithms on deltas
Batch is reliable
Streaming is approximate
Streaming with consistency
Two scripts for both approach Single script

13. Architecture [Lambda vs Kappa]
Lambda Kappa
Batch + Streaming Streaming
Query all data Incremental algorithms on deltas
Batch is reliable
Streaming is approximate
Streaming with consistency
Two scripts for both approach Single script

14. Architecture [Lambda vs Kappa]
Lambda Kappa
Batch + Streaming Streaming
Query all data Incremental algorithms on deltas
Batch is reliable
Streaming is approximate
Streaming with consistency
Two scripts for both approach Single script

15. References
-Flink Concepts [https://clck.ru/VQLMU]
-Spark Streaming [https://clck.ru/VQLPA]
-Streaming Systems: The What, Where, When, and How of Large-Scale
Data Processing
-Stream Processing with Apache Flink: Fundamentals, Implementation, and
Operation of Streaming Applications