Apache Kafka 101 by Confluent Developer Friendly

Read basic of implementing Apache Kafka.

1. Apache Kafka ® 101

2. What’s Covered Introduction Hands On: Your First Kafka Application in 10 Minutes or Less Topics Partitioning Hands On: Partitioning Brokers Replication Producers Consumers Hands On: Consumers Ecosystem Kafka Connect Hands On: Kafka Connect Confluent Schema Registry Hands On: Confluent Schema Registry Kafka Streams ksqlDB Hands On: ksqlDB

3. Introduction

4. Event Internet of Things

5. Event Internet of Things Business process change

6. Event Internet of Things Business process change User Interaction

7. Event Internet of Things Business process change User Interaction Microservice output

8. Notification + State

9. Key Value

10. Hands On: Your First Kafka Application in 10 Minutes or Less

11. GET STARTED TODAY Confluent Cloud cnfl.io/confluent-cloud PROMO CODE: KAFKA101 $101 of free usage

12. Hands On: Your First Kafka Application in 10 Minutes or Less

13. Topics

14. Topics

15. Topics Named container for similar events

16. Topics Named container for similar events System contains lots of topics Can duplicate data between topics

17. Topics Named container for similar events System contains lots of topics Can duplicate data between topics Durable logs of events

18. Topics Named container for similar events System contains lots of topics Can duplicate data between topics Durable logs of events Append only

19. Topics Named container for similar events System contains lots of topics Can duplicate data between topics Durable logs of events Append only Can only seek by offset, not indexed

20. Topics Named container for similar events System contains lots of topics Can duplicate data between topics Durable logs of events Append only Can only seek by offset, not indexed Events are immutable

21. Topics are durable

22. Retention period is configurable

23. Partitioning

24. Partition 0 Partition 1 Partition 2 Partitioned Topic

25. Partition 0 Partition 1 Partition 2

26. Partition 0 Partition 1 Partition 2

27. Partition 0 Partition 1 Partition 2 1 4 7 2 5 8 3 6 9 #

28. Partition 0 Partition 1 Partition 2 1 2 3 4 5 7 6 8 9 #

29. Partition 0 Partition 1 Partition 2 # 1 2 3 4 5 7 6 8 9

30. Hands On: Partitioning

31. Hands On: Partitioning

32. Brokers

33. Brokers An computer, instance, or container running the Kafka process

34. Brokers An computer, instance, or container running the Kafka process Manage partitions Handle write and read requests

35. Brokers An computer, instance, or container running the Kafka process Manage partitions Handle write and read requests Manage replication of partitions

36. Brokers An computer, instance, or container running the Kafka process Manage partitions Handle write and read requests Manage replication of partitions Intentionally very simple

37. Replication

38. Partition 0 Partition 1 Partition 2 Partitioned Topic

39. Replication Copies of data for fault tolerance

40. Replication Copies of data for fault tolerance One lead partition and N-1 followers

41. Replication Copies of data for fault tolerance One lead partition and N-1 followers In general, writes and reads happen to the leader

42. Replication Copies of data for fault tolerance One lead partition and N-1 followers In general, writes and reads happen to the leader An invisible process to most developers

43. Replication Copies of data for fault tolerance One lead partition and N-1 followers In general, writes and reads happen to the leader An invisible process to most developers Tunable in the producer

44. Producers

45. Partition 0 Partition 1 Partition 2 Partitioned Topic Producer

46. { final Properties props = KafkaProducerApplication. loadProperties (args[ ]); f inal String topic = props.getProperty( “output.topic.name” ); final Producer<String, String> producer = new KafkaProducer<>(props); final KafkaProducerApplication producerApp = new KafkaProducerApplication(producer, topic); }

47. { final ProducerRecord<String, String> producerRecord = new ProducerRecord <>( outTopic , key, value); return p roducer .send(producerRecord); }

48. Producers Client application Puts messages into topics Connection pooling Network buffering Partitioning

49. Partition 0 Partition 1 Partition 2 Partitioned Topic Producer

50. Consum ers

51. { final Properties consumerAppProps = KafkaConsumerApplication. load Properties(args[0]); final String filePath = consumerAppProps.getProperty( “file.path” ); final Consumer<String, String> consumer = new KafkaConsumer <>(consumerAppProps); final ConsumerRecordsHandler<String, String> recordsHandler = new FilewritingrecordsHandler(Paths.get(filePa … final KafkaConsumerapplication consumerApplication = new KafkaConsumerApplication(consumer, recordsHandler);

52. @ public void runConsume ( final Properties consumerProps) { try { consumer . subscribe (Collections. singletonList (consumerProps.getProperty( “input.topic.name” ))); while ( keepConsuming ) { final ConsumerRecords<String, String> consumerRecords = v consumer .poll(Duration. ofSeconds ( 1 )); recordsHandler .process(consumerRecords); }

53. @ public void process ( final ConsumerRecords<String, String> consumerRecords ) { final List<String> valueList = new ArrayList<>(); consumer Records .forEach(record -> valueList .add(record.value())); if (!valueList.isEmpty()) { try { Files. write ( path, valueList, StandardOpenOption. CREATE , StandardOpenOption. WRITE , StandardOpenOption … } catch (IOException e) { throw new RuntimeException(e); } } }

54. Consumers Client application Reads messages from topics Connection pooling Network protocol Horizontally and elastically scalable Maintains ordering within partitions at scale

55. Partition 0 Partition 1 Partition 2 Partitioned Topic Consumer A Consumer B

56. Partition 0 Partition 1 Partition 2 Partitioned Topic Consumer A

57. Partition 0 Partition 1 Partition 2 Partitioned Topic Consumer A Consumer A Consumer B

58. java application > configuration > dev.properties 1 group.id =movie_ratings 2 3 bootstrap.servers =localhost:29092 4 5 key.serializer =org.apache.kafka.common.serialization.StringSerializer 6 value .serializer =org.apache.kafka.common.serialization.StringSerializer 7 acks =all 8 9 #Properties below this line are specific to code in this application 10 input.topic.name =input-topic 11 output.topic.name =output-topic

59. Partition 0 Partition 1 Partition 2 Partitioned Topic Consumer A Consumer A Consumer B Consumer A

60. Hands On: Consumers

61. Hands On: Consumers

62. Ecosystem

63. Kafka Connect

64. Kafka Connect Data integration system and ecosystem Because some other systems are not Kafka External client process; does not run on brokers

65. Cluster Data Source Kafka Connect Kafka Connect Data Sink

66. Kafka Connect Data integration system and ecosystem Because some other systems are not Kafka External client process; does not run on brokers Horizontally scalable Fault tolerant

67. { “connector.class”: “io.confluent.connect.elasticsearch.ElasticsearchSinkConnector” , “connector.url”: “http://elasticsearch:9200” , “tasks.max”: “1” , “topics”: “simple.elasticsearch.data” , “name”: “simple-elasticsearch-connector” , “type.name”: “doc” , “value.converter”: “org.apache.kafka.connect.json.JsonConverter ” “value.converter.schemas.enable”: “false” }

68. Kafka Connect Data integration system and ecosystem Because some other systems are not Kafka External client process; does not run on brokers Horizontally scalable Fault tolerant Declarative

69. Connectors Pluggable software component

70. Connectors Pluggable software component Interfaces to external system and to Kafka

71. Connectors Pluggable software component Interfaces to external system and to Kafka Also exist as runtime entities

72. Connectors Pluggable software component Interfaces to external system and to Kafka Also exist as runtime entities Source connectors act as producers

73. Connectors Pluggable software component Interfaces to external system and to Kafka Also exist as runtime entities Source connectors act as producers Sink connectors act as consumers

74. Hands On: Kafka Connect

75. Hands On: Kafka Connect

76. Confluent Schema Registry

77. New consumers will emerge

78. Schemas evolve with the business

79. Schema Registry Server process external to Kafka brokers

80. Schema Registry Server process external to Kafka brokers Maintains a database of schemas

81. Schema Registry Server process external to Kafka brokers Maintains a database of schemas HA deployment option available

82. Schema Registry Server process external to Kafka brokers Maintains a database of schemas HA deployment option available Consumer/Producer API component

83. Schema Registry Defines schema compatibility rules per topic

84. Schema Registry Defines schema compatibility rules per topic Producer API prevents incompatible messages from being produced

85. Schema Registry Defines schema compatibility rules per topic Producer API prevents incompatible messages from being produced Consumer API prevents incompatible messages from being consumer

86. Supported Formats JSON Schema Avro Protocol Buffers

87. Hands On: Confluent Schema Registry

88. Hands On: Confluent Schema Registry

89. Kafka Streams

90. Kafka Streams Functional Java API

91. Kafka Streams Functional Java API Filtering, grouping, aggregating, joining, and more

92. Kafka Streams Functional Java API Filtering, grouping, aggregating, joining, and more Scalable, fault-tolerant state management

93. Kafka Streams Functional Java API Filtering, grouping, aggregating, joining, and more Scalable, fault-tolerant state management Scalable computation based on consumer groups

94. Kafka Streams Integrates within your services as a library

95. Kafka Streams Integrates within your services as a library Runs in the context of your application

96. Kafka Streams Integrates within your services as a library Runs in the context of your application Does not require special infrastructure

97. Insert Code: https://youtu.be/UbNoL5tJEjc?t=443

98. ksqlDB

99. ksqlDB A database optimized for stream processing

100. ksqlDB A database optimized for stream processing Runs on its own scalable, fault-tolerant cluster adjacent to the Kafka cluster

101. ksqlDB A database optimized for stream processing Runs on its own scalable, fault-tolerant cluster adjacent to the Kafka cluster Stream processing programs written in SQL

102. src > main > ksql > rate-movies.sql 1 2 CREATE TABLE rated_movies AS 3 SELECT title, 4 SUM (rating)/ COUNT (rating) AS avg_rating 5 FROM ratings 6 INNER JOIN movies 7 ON ratings.movie_id=movies.movie_id 8 GROUP BY title EMIT CHANGES;

103. ksqlDB Command line interface REST API for application integration

104. ksqlDB Command line interface REST API for application integration Java library

105. ksqlDB Command line interface REST API for application integration Java library Kafka Connect integration

106. Hands On: ksqlDB

107. Hands On: ksqlDB

108. Your Apache Kafka journey begins here developer.confluent.io