[DSC DACH 24] Trends in Data Pipeline Orchestration - Igor Kaludjer

Orchestration frameworks and platforms continue to evolve. We'll be talking about transactional semantics, multi-modality, human-in-the-loop and other goodies brought by Prefect 3.0. Declarative approach is eating the world and orchestrators are no exception with Kestra as one of them. And isn't it strange that only recently DWH vendors realized they should natively support transformation pipelines with out-of-the-box orchestration? We'll be taking a look at Snowflake dynamic tables and Databricks delta live tables as examples of declarative transformation pipelines with built-in orchestration.

1. Igor Kaluder, DSC DACH 2024.
Trends in Data Pipeline
Orchestration

2. Service Model -> Data Platform

3. Data Engineer
Data Platform
Engineer
Data Integration
Engineer
Analytics
Engineer

4. Automatic infrastructure
provisioning

5. Serverless and Managed Execution

6. Scheduled -> Event Driven

7. Transactional
Semantics

8. Human In the Loop

9. Declarative
Pipelines

10. Transformation
pipelines with
built-in orchestration

11. AI for Data Pipeline Orchestration?