Kafka streams - From pub/sub to a complete stream processing platform

A presentation on Kafka Streams APIs (part of Apache Kafka) and the innovative capabilities which brings in the world of open source stream processing engines. Simplicity (but powerful) and focus on developers being the biggest innovation.

1. Kafka Streams

From pub/sub to a complete
stream processing platform



Kafka Meetup Utrecht
Thursday, 8
th
June 2017
< paolo @ confluent.io >

2. https://www.confluent.io/blog/stream-data-platform-1/
Industry shift from Big Data
to Fast Data and Stream Processing

3. $ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt
Apache Kafka APIs and UNIX analogy

4. $ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt
Connect APIs
Apache Kafka APIs and UNIX analogy

5. $ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt
Producer/Consumer APIs
Apache Kafka APIs and UNIX analogy

6. $ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt
Streams APIs
Apache Kafka APIs and UNIX analogy

7. Streams APIs
part of Apache Kafka

http://kafka.apache.org/documentation/streams
http://docs.confluent.io/current/streams

8. Build applications, not clusters
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>0.10.2.1</version>
</dependency>

9. Spot the difference(s)

10. How do I run in production?

11. How do I run in production?
As any other Java applications...

12. How do I run in production?
Uncool Cool

13. How do I run in production?
http://docs.confluent.io/current/streams/introduction.html

14. Elastic and scalable
http://docs.confluent.io/current/streams/developer-guide.html#elastic-scaling-of-your-application

15. Elastic and scalable
http://docs.confluent.io/current/streams/developer-guide.html#elastic-scaling-of-your-application

16. Elastic and scalable
http://docs.confluent.io/current/streams/developer-guide.html#elastic-scaling-of-your-application

17. Typical high level architecture

18. Typical high level architecture
Real-time
Data
Ingestion

19. Typical high level architecture
Stream
Processing
Storage
Real-time
Data
Ingestion

20. Typical high level architecture
Data
Publishing /
Visualization
Stream
Processing
Storage
Real-time
Data
Ingestion

21. How many clusters do you count?
NoSQL
(Cassandra,
HBase,
Couchbase,
MongoDB, …)
or
Elasticsearch,
Solr,
…
Storm, Flink,
Spark
Streaming,
Ignite, Akka
Streams, Apex,
…
HDFS, NFS,
Ceph,
GlusterFS,
Lustre,
...
Apache Kafka

22. Simplicity is the ultimate sophistication
Apache Kafka
and Kafka Streams APIs
Stream Processing Platform

Publish & Subscribe
to streams of data like a
messaging system

Store
streams of data safely in a
distributed replicated cluster

Process
streams of data efficiently
and in real-time

Node.js

23. Duality of Streams and Tables
http://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables

24. Duality of Streams and Tables
http://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables

25. Interactive Queries
http://docs.confluent.io/current/streams/developer-guide.html#streams-developer-guide-interactive-queries

26. Interactive Queries
http://docs.confluent.io/current/streams/developer-guide.html#streams-developer-guide-interactive-queries

27. Kafka Streams DSL
http://docs.confluent.io/current/streams/developer-guide.html#kafka-streams-dsl

28. WorldCount (and Java 8)
WordCountLambdaExample.java
final Properties streamsConfiguration = new Properties();
streamsConfiguration.put (StreamsConfig.APPLICATION_ID_CONFIG , "wordcount-lambda-example");
streamsConfiguration.put (StreamsConfig.BOOTSTRAP_SERVERS_CONFIG , bootstrapServers);

...

final Serde<String> stringSerde = Serdes.String();
final Serde<Long> longSerde = Serdes.Long();

final KStreamBuilder builder = new KStreamBuilder();

final KStream<String, String> textLines = builder.stream(stringSerde, stringSerde, "TextLinesTopic");

final Pattern pattern = Pattern.compile("\\W+", Pattern.UNICODE_CHARACTER_CLASS );

final KTable<String, Long> wordCounts = textLines
.flatMapValues(value -> Arrays.asList(pattern.split(value.toLowerCase())))
.groupBy((key, word) -> word)
.count("Counts");

wordCounts.to(stringSerde, longSerde, "WordsWithCountsTopic ");

final KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration );

streams.cleanUp();
streams.start();

Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

29. Easy to Develop, Easy to Test
WordCountLambdaIntegrationTest.java
EmbeddedSingleNodeKafkaCluster CLUSTER =
new EmbeddedSingleNodeKafkaCluster ();

…

CLUSTER.createTopic(inputTopic);

…

Properties producerConfig = new Properties();
producerConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG ,
CLUSTER.bootstrapServers ());

30. Apache Kafka and Streams APIs benefits
•Build applications, not clusters
•Native integration with Apacke Kafka
•Elastic, fast, distributed, fault-tolerant, secure
•Scalable: S, M, L, XL, XXL
•Run everywhere: from containers to cloud
•Streams (with KStream) and tables (with KTable)
•Local state replicated to Kafka for fault-tolerance
•Windowing and event time semantics out of the box
•Supports late-arriving and out-of-order events

31. References
•http://kafka.apache.org/
•http://kafka.apache.org/documentation/streams/

•http://docs.confluent.io/
•http://docs.confluent.io/current/streams/

•http://docs.confluent.io/current/streams/javadocs/

•http://blog.confluent.io/

•http://github.com/confluentinc/examples/

•http://github.com/apache/kafka/tree/trunk/streams/

32. References

33. The easiest way to get you started
https://www.confluent.io/download/

34. SIMPLICITY
WE

35. YOUR FEEDBACK!

36. Discount code: kafcom17
‪Use the Apache Kafka community discount code to get $50 off
‪www.kafka-summit.org
Kafka Summit San Francisco: August 28
Presented by