ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, PostgreSQL, Redpanda, Debezium, and Benthos to master building advanced real-time data pipelines.

The slide that goes with the workshop.

- Demonstrate Transitioning to Real-Time Data Systems: This track shows how to transition from traditional batch processing to real-time data handling, emphasizing the critical importance of timely updates in scenarios like air traffic control where delays can be hazardous.
- Implement Real-Time Data Solutions: Participants learn to implement and integrate a technology stack featuring Apache Spark, Cassandra, and PostgreSQL to effectively manage and process live data feeds from various sources, suitable for high-stakes environments.
- Employ Streaming and Data Capture Technologies: The workshop demonstrates how to utilize streaming platforms and Change Data Capture (CDC) with tools like Redpanda and Debezium to ensure continuous data flow and immediate update capture, streamlining operations and data accuracy.
- Apply Data Querying Techniques: It concludes by teaching how to apply advanced querying methods using Flink’s SQL client to incorporate real-time data queries into existing systems, ensuring efficient data manipulation and integration into relational and NoSQL databases.

1. BATCH
TO
STREAM
DATA PIPELINE
Christina Lin
Redpanda
Developer Advocate

2. Agenda
•Quick Intro – 10-15 mins
•HANDS-ON – 35-40 mins
•Q & A – 5-10 mins
© 2024 REDPANDA DATA

3. Christina Lin
Developer Advocate, Redpanda
aka. The Redpanda Lady
SOA
WebSphere
DB2
Sybase
Oracle
MQ
J2EE
EJB
DevOps
Microservice
EIP
K8s
Agile
Integration
Data
Mesh
Active MQ
Living data stack
Resilience - handle failures and scale gracefully
Elasticity – infrastructure that can scale dynamically
Decentralization - data ownership, empowering
individual teams
Performance - low latency and high throughput
Autonomy – self service, define quality, and access
Nimble - efficient data movement
Distributed-distributed data processing for cloud native
Agility – quickly respond to change in data
© 2024 REDPANDA DATA

4. © 2024 REDPANDA DATA
An ordinary day of Data Engineer

5. © 2024 REDPANDA DATA© 2024 REDPANDA DATA
Stateless
StreamingPipeline
Transform
format Change, masking, filtering, validating
Dispatch, Wiretap
Spilt, multiple destinationControl
reroute
Normalize/ Denormalize
Enrich
Multiple ingestion
Stateful
StreamingPipeline
Complex event processing
Time-window based processing
Enrich
Multiple ingestion
Micro batch Pipeline
Transform for large output (Dataset)
Partitioning Split workload
Analytics
batch
Pipeline
Analytics large volume (legacy)
Transform large output (Dataset, legacy)
Transport large unstructured data
Better scalability for pipelines

6. Batch
CSV
Every 10 mins
CSV
Right away!
CSV
Stream
Batch pipeline
Batch Processing
Batch
pipeline
The Workshop overview
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda

7. VM
postgres:5432cassandra:5432
postgresload.ipynb
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
The Batch
cassandra:5432

8. VM
postgres:5432cassandra:5432
cassandraload.ipynb
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
The Batch

9. VM
postgres:5432cassandra:5432
spark.ipynb
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
The Batch
CSVCSVCSV

10. VM
postgres:5432cassandra:5432
Map.ipynb
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
Dashboard View
CSVCSVCSV

11. VM
postgres:5432cassandra:5432redpanda-0:9092
© 2024 REDPANDA DATAhttps://bit.ly/osdc-redpanda
Let’s Stream

12. VM
postgres:5432cassandra:5432
© 2024 REDPANDA DATAhttps://bit.ly/osdc-redpanda
Let’s Stream

13. VM
postgres:5432cassandra:5432
Config
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
Let’s Stream

14. VM
postgres:5432cassandra:5432
Job manager
Flink Data
Stream
Java JAR
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
Let’s Stream

15. VM
postgres:5432cassandra:5432
Job manager
CSV
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
Let’s Stream

16. VM
postgres:5432cassandra:5432
Job manager
CSV
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
Let’s Stream
CSVCSV

17. VM
postgres:5432cassandra:5432
SQL Client
SQL
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
The Batch
CSVCSVCSV

18. Batch
CSV
Batch pipeline
Batch
pipelineBatch Processing
Every 10 mins
CSV
Right away!
CSV
Stream
© 2024 REDPANDA DATAhttps://bit.ly/odsc-redpanda
The Workshop overview

19. © 2024 REDPANDA DATA
Stateless
StreamingPipeline
Transform
format Change, masking, filtering, validating
Dispatch, Wiretap
Spilt, multiple destinationControl
reroute
Normalize/ Denormalize
Enrich
Multiple ingestion
Stateful
StreamingPipeline
Complex event processing
Time-window based processing
Enrich
Multiple ingestion
Better scalability for pipelines

20. © 2024 REDPANDA DATA
Keep Learning
Streaming - Communication
Basics of K8s networking
Connectivity
Performance
Docs
Get a peak under the hood.
https://docs.redpanda.com/
Blogs
Keep up to date with Redpanda.
https://redpanda.com/blog
Slack
Engage with our community.
https://redpanda.com/slack
Code
Check out the source.
https://github.com/redpanda-data
Redpanda University
Free, self-paced online learning
https://university.redpanda.com