Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide

The slide that goes with the workshop.

- Demonstrate Transitioning to Real-Time Data Systems: This track shows how to transition from traditional batch processing to real-time data handling, emphasizing the critical importance of timely updates in scenarios like air traffic control where delays can be hazardous.
- Implement Real-Time Data Solutions: Participants learn to implement and integrate a technology stack featuring Apache Spark, Cassandra, and PostgreSQL to effectively manage and process live data feeds from various sources, suitable for high-stakes environments.
- Employ Streaming and Data Capture Technologies: The workshop demonstrates how to utilize streaming platforms and Change Data Capture (CDC) with tools like Redpanda and Debezium to ensure continuous data flow and immediate update capture, streamlining operations and data accuracy.
- Apply Data Querying Techniques: It concludes by teaching how to apply advanced querying methods using Flink’s SQL client to incorporate real-time data queries into existing systems, ensuring efficient data manipulation and integration into relational and NoSQL databases.

1. BATCH
TO
STREAM
DATA PIPELINE
Christina Lin
Redpanda
Developer Advocate

2. Agenda
•Quick Intro – 10-15 mins
•HANDS-ON – 35-40 mins
•Q & A – 5-10 mins
© 2024 REDPANDA DATA

3. Christina Lin
Developer Advocate, Redpanda
aka. The Redpanda Lady
SOA
WebSphere
DB2
Sybase
Oracle
MQ
J2EE
EJB
DevOps
Microservice
EIP
K8s
Agile
Integration
Data
Mesh
Active MQ
Living data stack
Resilience - handle failures and scale gracefully
Elasticity – infrastructure that can scale dynamically
Decentralization - data ownership, empowering
individual teams
Performance - low latency and high throughput
Autonomy – self service, define quality, and access
Nimble - efficient data movement
Distributed-distributed data processing for cloud native
Agility – quickly respond to change in data
© 2024 REDPANDA DATA

4. © 2024 REDPANDA DATA
An ordinary day of Data Engineer

5. © 2024 REDPANDA DATA© 2024 REDPANDA DATA
Stateless
StreamingPipeline
Transform
format Change, masking, filtering, validating
Dispatch, Wiretap
Spilt, multiple destinationControl
reroute
Normalize/ Denormalize
Enrich
Multiple ingestion
Stateful
StreamingPipeline
Complex event processing
Time-window based processing
Enrich
Multiple ingestion
Micro batch Pipeline
Transform for large output (Dataset)
Partitioning Split workload
Analytics
batch
Pipeline
Analytics large volume (legacy)
Transform large output (Dataset, legacy)
Transport large unstructured data
Better scalability for pipelines

6. Batch
Every 10 mins
CSV
Right away!
CSV
CSV
Stream
Batch pipeline
Batch Processing
Batch
pipeline
The Workshop overview
© 2024 REDPANDA DATA

7. Batch
Every 10 mins
Right away!
CSV
CSV
Stream
Batch pipeline
Batch Processing
Batch
pipeline
The Workshop overview
© 2024 REDPANDA DATA
CSV

8. VM
cassandra:5432
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
The Batch
Setup & Load data
postgres:5432
Table
public.bos_air_traffic
jupyterlab:8888
postgresload.ipynb

9. VM
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
Setup
cassandraload.ipynb
Table
latest_flight_data
cassandra:5432
The Batch

10. VM
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
CSVCSVCSV
spark.ipynb
The Batch

11. VM
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
CSVCSVCSV
Map.ipynb
The Batch

12. VM
© 2024 REDPANDA DATAhttps://bit.ly/osdc-redpanda
Let’s Stream
redpanda-0:9092
console:8080

13. VM
© 2024 REDPANDA DATAhttps://bit.ly/osdc-redpanda
kafka-connect:8083
Let’s Stream

14. VM
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
ConfigTopic:
boston.public.bos_air_trafficTable
public.bos_air_traffic
Let’s Stream

15. VM
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
jobmanager:8081
Topic:
boston.public.bos_air_traffic
Flink Data
Stream
Java JAR
Let’s Stream

16. VM
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
CSVCSVCSV
Topic:
sensor_csv
Topic:
Sensor_csv
Let’s Stream

17. VM
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
Let’s Stream
CSVCSVCSV
Topic:
sensor_csv
rpkredpanda-0:9644
Topic:
filtered_sensor_csv

18. VM
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
CSVCSVCSV
Topic:
sensor_csv
SQL
SQL
Client
Let’s Stream

19. Batch
CSV
Batch pipeline
Batch
pipelineBatch Processing
Every 10 mins
CSV
Right away!
CSV
Stream
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
The Workshop overview

20. © 2024 REDPANDA DATA
Stateless
StreamingPipeline
Transform
format Change, masking, filtering, validating
Dispatch, Wiretap
Spilt, multiple destinationControl
reroute
Normalize/ Denormalize
Enrich
Multiple ingestion
Stateful
StreamingPipeline
Complex event processing
Time-window based processing
Enrich
Multiple ingestion
Better scalability for pipelines

21. © 2024 REDPANDA DATA
Keep Learning
Streaming - Communication
Basics of K8s networking
Connectivity
Performance
Docs
Get a peak under the hood.
https://docs.redpanda.com/
Blogs
Keep up to date with Redpanda.
https://redpanda.com/blog
Slack
Engage with our community.
https://redpanda.com/slack
Code
Check out the source.
https://github.com/redpanda-data
Redpanda University
Free, self-paced online learning
https://university.redpanda.com