Introducing KSML: Kafka Streams for low code environments | Jeroen van Disseldorp and Ton van Bart, Axual

Kafka Streams has captured the hearts and minds of many developers that want to develop streaming applications on top of Kafka. But as powerful as the framework is, Kafka Streams has had a hard time getting around the requirement of writing Java code and setting up build pipelines. There were some attempts to rebuild Kafka Streams, but up until now popular languages like Python did not receive equally powerful (and maintained) stream processing frameworks. In this session we will present a new declarative approach to unlock Kafka Streams, called KSML. After this session you will be able to write streaming applications yourself, using only a few simple basic rules and Python snippets.

1. Introducing KSML: Kafka Streams for Low-code Environments Kafka Summit Europe 2021 Jeroen van Disseldorp CEO Ton van Bart Senior Engineer STREAMING MADE SIMPLE

2. Axual in a nutshell Page 2 Easy setups kafka ++ On-premise & Clouds s treaming m ade simple Battle-tested security Fully Supported

3. Data Governance Role-based access control Distributed governance Metadata catalogs Data lineage Security Network and storage encryption Strict access control Regularly tested by independent third parties Operations High availability Software-defined and automated deployments Metric collection and dashboards Log collection and analysis App development & process Self-service for DevOps DTAP Support Client libraries Test tooling CI / CD integration Platform Publish/Subscribe Messaging High Performance Multi-tenant Multi-cluster with transparent message replication Integration options & Protocol support Infrastructure Fully containerized for on-premise & clouds Axual provides Kafka in an Enterprise-Ready package Page 3 Logging and Monitoring Infrastructure Streaming Platform Data Governance VMs Kubernetes Kafka REST Connect Distributor Prometheus Axual Deploy Helm Discovery Schema Platform Manager Metadata Self-service UI Grafana Cluster Mgr Instance Mgr Alert Manager Logstash ElasticSearch Kibana Docker Linux Hard ware On-premise, AWS, Azure, Google, ...

4. Self-service allows DevOps teams to collaborate independently Page 4 Applications can be multiple types: Custom Custom code Java, Python, .NET Runs on Team’s own platform Connector Low code JDBC, Snowflake, ... Runs on a central platform Streaming App Low code Filter, Aggregate, ... Runs on a central platform ?

5. Last year’s hobby... Page 5

6. Scripting Page 6

7. YAML tags are interpreted as a DSL Requires no “coding” Domain Specific Language for HA Automations : tasks executed upon some trigger Scripts : functions that may be called upon from anywhere (function library) Sensors : inputs from very different plugins, like smart lights, smarts cars or even Jumbo supermarket ... No code compilation takes place, only interpretation Page 7

8. Page 8

9. What if we apply such a mechanism to Axual / Kafka With YAML we could specify a Kafka Streams topology Example pipeline: main: from: some_topic to: some_other_topic Page 9 Interpreter YAML + = But wait... what about custom functions like Predicates Transformations of keys and values ValueJoiners ... Could we somehow specify them without requiring compilation?

10. Enter Jython Page 10

11. KSML: a library to generate Kafka Streams topologies KSML reads YAML and generates a complete Kafka Streams topology Page 11 KSML Topology Generator KSML Input + =

12. Demonstration SensorData Name String The name of the sensor Timestamp Long The timestamp of the sensor reading Value String The value of the sensor, represented as string Type Enum The type of the sensor "AREA", "HUMIDITY", "LENGTH", "STATE", "TEMPERATURE" Unit String The unit of the sensor Color String The color of the sensor City String The city of the sensor Owner String The owner of the sensor Page 12

13. DEMO Inspect  Python Print Copy  copy all messages to given output Filter  copy only if input matches a criterium, Python condition Branch  multiple filters combined in one construct Router  output messages to dynamic topic name Page 13

14. Yes, it works! Page 14

15. Complete the Kafka Streams DSL All stream types implemented KStreams , KTables , GlobalKTables , KGroupedStream , KGroupedTable , SessionWindowedKStream , TimeWindowedKStream All stream operations implemented Filter, FilterNot , ForEach , Aggregate, Count, Reduce, GroupBy , ... All custom function types implemented Aggregator, ForEachAction , Initializer, KeyTransformer , Merger, Predicate, Reducer, ... Extending the list of supported data types Primitive types implemented (String, Int, Long, Float, Double, Binary) Basic AVRO works JSON support coming very soon Page 15

16. Next steps... Test, test, test Current code is alpha Build community Get feedback on the idea Imagine potential use cases Work out deployment model(s), such as Simple inclusion library for Java applications Docker image with Spring Boot Kafka Connect-like container with multiple threads Work towards beta release Implement unit tests Refine typing model  extend upon current AVRO-support Refine syntax (currently very much alpha state) Refine implementation Page 16 Join us at https:// ksml.io