Qlik Attunity Replicator and Kafka synco

This is introduction for Attunity data replication technology based on kafka

1. Kafka and Replicate

2. What is Kafka Distributed message queue High throughput – designed for large amount of small messages (default: 1MB) Scalable – cluster can be extended easily Persistent – messages are stored on disk, replicas are stored on different servers Runs on Java (Linux, Windows …) Apache open-source

3. Kafka Terminology Broker Kafka server Cluster Multiple brokers form a cluster Producer Kafka client, sends messages to brokers Consumer Kafka client, r eceives messages from brokers Topic Logical queue, divided into partitions Partition Contains Kafka messages, saved as a file on disk. Message Any buffer ( text/binary), f ormat should be coordinated between producers and consumers Message Key Optional, used for partitioning and compaction Offset Position of a message in a partition , u sed by consumer

4. Kafka Architecture

5. Anatomy of a Topic

6. Kafka Target Endpoint Replicate is a Kafka Producer Connects to brokers and sends messages Each message represents a record Message format is JSON, UTF-8 encoded, or Avro Message can have a Key Message can be compressed (snappy, gzip) Full Load and CDC are supported Supported Kafka versions: 0.8, 0.9, 0.10

7. Kafka – Metadata Messages Replicate produces messages to Kafka Each message contains a data record Metadata messages: Replicate Server name Task name Source schema, Source table Columns information Table version

8. (new) JSON Message Structure { "magic": " atMSG ", "type": "DT", "message": { "data": { "ID": "4", "NAME": " yyyyy " }, " beforeData ": { "ID": "4", "NAME": " xxxxx " }, "headers": { "operation": "UPDATE",  include FRESH (full load), UPDATE, DELETE, INSERT (CDC) " changeSequence ": "20170622101040000000000000000000005", "timestamp": "2017-06-22T10:10:40.000", " streamPosition ": "00000000.0095508c.00000001.0000.02.0000:237.36778.16", " transactionId ": "000000000000000000000000001B0007" } } }

9. (new) JSON Message Structure

10. Endpoint Configuration

11. Troubleshooting Connection Errors Test the connection to Kafka broker Check the broker’s address and port Verify that port is accessible by using telnet

12. Thank You 