Apache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARN

Overview of Apache Samza presented to the London HUG, October 22, 2013

1. Apache Samza Reliable Stream Processing Atop Apache Kafka and Hadoop YARN Jakob Homan London HUG

2. Who I am Samza for five months B efore that Hadoop , Hive, Giraph Say hi: @ blueboxtraveler

3. Things we would like to do (better)

4. Provide timely, relevant updates to your newsfeed

5. Update search results with new information as it appears

6. Sculpt metrics and logs into useful shapes

7. Tools? Response latency Samza Milliseconds to minutes RPC Synchronous Later. Possibly much later.

8. Frame(work) of reference Classic Hadoop Samza Storage layer Execution engine API HDFS Kafka Map-Reduce YARN map(k, v) => ( k,v ) reduce(k, list(v)) => ( k,v ) process( msg ( k,v )) => msg ( k,v )

9. Storage layer: Kafka

10. Apache Kafka Persistent, reliable, distributed message queue Shiny new logo!

11. At LinkedIn 10+ billion writes per day 172k messages per second (average) 55+ billion messages per day to real-time consumers

12. Quick aside… Kafka: First among (pluggable) equals LinkedIn: Espresso and Databus Coming soon? HDFS, ActiveMQ , Amazon SQS

13. Kafka in four bullet points Producers send messages to brokers Messages are key, value pairs Brokers store messages in topics for consumers Consumers pull messages from brokers

14. A Kafka Topic “Very sleepy” 534 “Car nicked!” 755 “The ref’s blind!” 234 “Nicked a car!” 534 Topic: StatusUpdateEvent Key: User ID of user who updated the status Value: Timestamp, new status, geolocation , etc.

15. Kafka topics are partitioned Message contents Key Message contents Key Message contents Key Message contents Key Message contents Key Message contents Key Message contents Key Message contents Key Message contents Key Message contents Key Partition 0 Partition 1 Partition 2 For our purposes, hash partitioned on the key!

16. A Samza job Input topics StatusUpdateEvent NewConnectionEvent LikeUpdateEvent Some code MyStreamTask implements StreamTask { …………. } Output topics NewsUpdatePost UpdatesPerHourMetric

17. Execution engine: YARN

18. What we use YARN for Distributing our tasks across multiple machines Letting us know when one has died Distributing a replacement Isolating our tasks from each other

19. Machine 1 Machine 1 YARN: Execution and reliability MyStreamTask: process () Samza TaskRunner : Partition 0 MyStreamTask: process () Samza TaskRunner : Partition 1 Node Manager 2 Node Manager 1 Samza App Master Kafka Broker Kafka Broker

20. Co-partitioning of topics MyStreamTask: process () Samza TaskRunner : Partition 0 StatusUpdateEvent , Partition 0 NewConnectionEvent , Partition 0 NewsUpdatePost An instance of StreamTask is responsible for a specific partition

21. API: process()

22. public interface StreamTask { void process ( IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator ) } getKey (), getMsg () sendMsg (topic, key , value) commit(), shutdown()

23. Awesome feature: State Generic data store interface Key-value out-of-box More soon? Bloom filter, lucene , etc. Restored by Samza upon task crash MyStreamTask: process () Samza TaskRunner : Partition 0 Store state

24. (Pseudo)code snippet: Newsfeed Consume StatusUpdateEvent Send those updates to all your conmections via the NewsUpdatePost topic Consume NewConnectionEvent Maintain state of connections to know who to send to

25. public class NewsFeed implements StreamTask { void process( envelope , collector , coordinator ) { msg = env.getMsg () userId = msg.get ( “ userID ” ); if( msg .get ( “type” )==STATUS_UPDATE) { foreach ( conn : kvStore .get ( userId ) { collector .send ( “ NewsUpdatePost ” , new Msg ( conn , msg .get ( “ newStatus ”) ) } } else { newConn = msg .get ( “ newConnection ” ) connections = kvStore .get ( userId ) kvStore .put ( userID , connections ++ newConn ) }

26. Current status

27. Hello, Samza ! Cool, eh? bit.ly /hello- samza Consume Wikipedia edits live Up and running in 3 minutes Generate stats on those edits

28. samza.incubator.apache.org bit.ly / samza_newbie_issues

29. Cheers! Quick start: bit.ly /hello- samza Project homepage: samza.incubator.apache.org Newbie issues: bit.ly / samza_newbie_issues Detailed Samza and YARN talk: bit.ly /samza_and_yarn Twitter: @ samzastream