Large scale stream processing with Apache Flink

In today’s world it’s no longer enough to build systems that process big volumes of information. We now need applications that can handle large continuous streams of data with very low latency so we can react to the ever-changing environment around us. To efficiently handle such problems we need to deploy a stream processing solution. During the talk we’ll explore one of the most popular frameworks for stream processing – Apache Flink. We’ll see what unique capabilities it provides and how they apply to some real world problems. And we’ll also explore how it works under the hood and how to get the scalable and fault-tolerant stream processing that Flink provides.

1. Large scale stream
processing with
Apache Flink
Nikolay Stoitsev
Sr. Software Engineer at Uber Tech Sofia

2. Stream Processing?

3. Stream Processing?
User Interaction Logs

4. Stream Processing?
User Interaction Logs

Application Logs

5. Stream Processing?
User Interaction Logs

Application Logs

Sensor Data

6. Stream Processing?
User Interaction Logs

Application Logs

Sensor Data

Database Commit Logs

7. Infinite Dataset

8. Producer
Stream

9. Producer
Stream HDFS

10. Producer
Stream HDFS
Hive

11. Producer
Stream HDFS
Hive
Big Latency

12. Producer
Stream
HDFS
Real-time
service

13. Apache Storm
storm.apache.org

14. High-latency & accurate
vs.
Low-latency & approximation

15. Lambda architecture

16. https://www.oreilly.com/ideas/questioning-the-lambda-architecture

17. Kappa Architecture

18. Use Apache Kafka
Durable, scalable, fault-tolerant

19. Producer
Kafka
Stream
Processor

22. Metrics we want to track
Net payout
Daily items sold
Weekly items sold


Order acceptance rate
Order preparation speed
Item rating

23. Real time

24. Scalable

25. Granular

26. Highly available

27. Order Stream
Payment Stream
User Rating Stream

28. Order Stream
Payment Stream
User Rating Stream
Stream Processor
OLAP

29. samza.apache.org

30. Apache Flink
flink.apache.org

31. Everything is a batch
vs.
Everything is a stream

32. Single JVM Cluster Cloud
Runtime
DataSet API DataStream API

33. Dataflow graph

34. Source
Source
Operator
Operator
Operator Sinc
OLAP

35. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html

36. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html

37. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html

38. Flink Program
Optimizer
Graph Builder
Client

39. Flink Program
Optimizer
Graph Builder
Client
Job Manager
Task Manager Task Manager

40. Flink Program
Optimizer
Graph Builder
Client
Job Manager
Task Manager Task Manager
Snapshot Store

41. Fault tolerant

42. Flink Program
Optimizer
Graph Builder
Client
Job Manager
Task Manager Task Manager
Snapshot Store

43. Lightweight Asynchronous Snapshots for
Distributed Dataflows

Paris Carbone,
Gyula Fóra,
Stephan Ewen
Seif Haridi
Kostas Tzoumas

44. Barrier Msg MsgBarrierMsg MsgBarrier
Operator

45. Barrier Msg MsgBarrierMsg Msg
Operator
Msg
Snapshot Store

46. Exactly Once Processing

47. Can handle very large state

48. Flink Program
Optimizer
Graph Builder
Client
Job Manager
Task Manager Task Manager
Snapshot Store

49. Flink Program
Optimizer
Graph Builder
Client
Job Manager
Task Manager Task Manager
Snapshot Store
Job
Manager
Job
Manager
Zookeeper

50. Flink Program
Optimizer
Graph Builder
Client
Job Manager
Task Manager Task Manager
Snapshot Store
Job
Manager
Job
Manager
Zookeeper

51. Flink Program
Optimizer
Graph Builder
Client
Task Manager Task Manager
Snapshot Store
Job
Manager
Job
Manager
Zookeeper

52. Joining Streams

53. Order Stream
User Rating Stream

54. Order Stream
User Rating Stream

55. Order Stream
User Rating Stream
Local Join
Local Join

56. Order Stream
User Rating Stream
Local Join
Local Join

57. Apache Flink
●Can join streams
●Fault tolerant
●Exactly Once Processing
●Combines stream and batch processing

58. … but it requires Java/Scala code

59. Scalable, efficient and robust

60. github.com/uber/AthenaX

61. SQL → what data to analyze

Flink → how to analyze it

69. Resource estimation and
auto scaling

70. Monitoring and automatic
failure recovery

71. eng.uber.com/athenax

72. Thanks!
Nikolay Stoitsev @ Uber