Transforming Data Streams with Kafka Connect: An Introduction to Single Message Transforms

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way.



Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks.



Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case.



Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

1. Transforming Data Streams with Kafka Connect:
An Introduction to Single Message Transforms

Ranjan Rao

2. ❏Kafka Connect Overview
❏Single Message Transforms
❏Common Use Cases
❏Built-in SMTs
❏Example SMT Configuration
❏Alternatives
@ranjan_rao

3. Kafka Connect Overview
➔Tool for streaming data from various sources to Kafka and from Kafka to
various sink systems.



@ranjan_rao
Source
Source
Connector Sink
Sink
Connector

4. Single Message Transforms
➔Provides a powerful and flexible way to transform data
➔Simple function that accepts one record as an input and outputs a modified
record
➔Apache Kafka 0.10.2.0



@ranjan_rao

5. @ranjan_rao
Single Message Transforms
Database
Source
Connector
Transform #1Transform #N
Converter
Database
Sink
Connector
Transform #N
Converter
Transform #1

6. ✓Kafka Connect Overview
✓Single Message Transforms
❏Common Use Cases
❏Built-in SMTs
❏Examples of SMT Configurations
❏Alternatives
@ranjan_rao

7. Common Use Cases
●Data enrichment/enhancement
●Partitioning
●Field masking
●Event routing

@ranjan_rao

8. ✓Kafka Connect Overview
✓Single Message Transforms
✓Common Use Cases
❏Built-in SMTs
❏Example of SMT Configurations
❏Alternatives
@ranjan_rao

9. Built-in SMTs
●InsertField
●ReplaceField
●MaskField
●ValueToKey
●ExtractField


@ranjan_rao
●TimestampRouter
●RegexRouter
●InsertHeader
●HeadersFrom
●DropHeaders

10. ✓Kafka Connect Overview
✓Single Message Transforms
✓Common Use Cases
✓Built-in SMTs
❏Example of SMT Configurations
❏Alternatives
@ranjan_rao

11. Example SMT configuration
{“Firstname”: “John”, “LastName”: “Doe”,”Phone”: “000-000-0000” }


{“Firstname”: “John”, “LastName”: “Doe”, “Phone”: “123-456-7890”}



InsertField
MaskField
{“Firstname”: “John”, “LastName”: “Doe”,”Phone”: “ 000-000-0000”,
“Origin”: “Source Connector”,
“RecordTimestamp”: 1707940094
}

12. Example SMT configuration
"transforms": "MaskField,InsertField" ,

"transforms.MaskField.type" : "org.apache.kafka.connect.transforms.MaskField$Value" ,
"transforms.MaskField.fields" : "Phone",
"transforms.MaskField.replacement" : "000-000-0000"

"transforms.InsertField.type" : "org.apache.kafka.connect.transforms.InsertField$Value" ,
"transforms.InsertField.static.field" : "Origin",
"transforms.InsertField.static.value" : "Source Connector",
"transforms.InsertField.timestamp.field" : "RecordTimestamp"




{“Firstname”: “John”, “LastName”: “Doe”,”Phone”: “000-000-0000”,
“Origin”: “Source Connector”,
“RecordTimestamp”: 1707940094
}

13. ●Single Message Transforms for Confluent Platform
●Single Message Transforms for Kafka Connectors in Confluent Cloud


@ranjan_rao
Example SMT configuration

14. ✓Kafka Connect Overview
✓Single Message Transforms
✓Common Use Cases
✓Built-in SMTs
✓Example of SMT Configurations
❏Alternatives

@ranjan_rao

15. Alternatives

➢Kafka Streams
➢ksqlDB
➢Apache Flink


@ranjan_rao

16. Thank You!