Fabric Engineering Deep Dive Keynote from Fabric Engineering Roadshow

Fabric Engineering Deep Dive Keynote from Fabric Engineering Roadshow in Munich on 21st of June 2024

1. Microsoft Fabric
Fabric Engineering
Deep Dive
Welcome & Introduction

2. Microsoft Fabric
Fabric Roadshow
Agenda
TimeSessions DurationSpeaker
09:00Welcome & Introduction 15 minGabi
09:15Keynote 45 minGabi & Wolf
10:00Migration pathway – Datawarehouse 60 minArtur
11:00Coffee break 15 mins-
11:15Everything you need to know about DirectLake60 minGabi
12:15What’s new in Fabric Real-Time Intelligence?60 minDevang
13:15Lunch break 60 min-
14:15Roadmap 45 minAll
15:00AMA 60 minAll
16:00Closing

3. Microsoft Fabric
Meet the speakers
Artur Vieira
Fabric CAT Principal PM

4. Microsoft Fabric
Meet the speakers
Devang Shah
Fabric CAT RTI Principal PM

5. Microsoft Fabric
Meet the speakers
Wolf Biber
Practice lead for analytics

6. Microsoft Fabric
Meet the speakers
Gabi Münster
Fabric CAT Senior PM

7. Microsoft Fabric

8. Microsoft Fabric
Fabric Engineering
Deep Dive
Keynote

9. The unified data platform for the era of AI

10. The unified data platform for the era of AI
Data
Engineering
Data
Warehouse
Data
Science
Real-Time
Intelligence
Power BI
Data
Factory

11. Rapid pace
of innovation:

12. Rapid pace
of innovation:

13. The unified data platform for the era of AI
Complete
Analytics
Platform
Everything, unified
SaaS-ified
Secured and governed
Lake Centric and Open
OneLake
One copy
Open at every tier
Empower Every
Business User
Familiar and intuitive
Built into Microsoft 365
Insight to action
AI
Powered
Copilot accelerated
Gen AI on your data
AI
-driven insights

14. Scales to the most demanding projects

15. Scales to the most demanding projects

16. Taskflows

17. The unified data platform for the era of AI
Data
Engineering
Data
Warehouse
Data
Science
Real-Time
Intelligence
Power BI
Data
Factory

18. The unified data platform for the era of AI
Data
Engineering
Data
Warehouse
Data
Science
Real-Time
Intelligence
Power BI
Partner
workloads
Data
Factory

19. The unified data platform for the era of AI
Data
Engineering
Data
Warehouse
Data
Science
Real-Time
Intelligence
Power BIData Factory
and more…

21. Your data application engine
embedded as a native Fabric
experience customers know and love
Rich component framework to for
building native Fabric experience
Single view to see all data processing
operations
Leverage Fabric ingestion and
connectivity to get data into your
workload
Unmatched integration flexibility

22. The unified data platform for the era of AI
Complete
Analytics
Platform
Everything, unified
SaaS-ified
Secured and governed
Lake Centric and Open
OneLake
One copy
Open at every tier
Empower Every
Business User
Familiar and intuitive
Built into Microsoft 365
Insight to action
AI
Powered
Copilot accelerated
Gen AI on your data
AI-driven insights

23. A single SaaS lake for the whole
organization
Provisioned automatically with the tenant
All workloads automatically store their
data in the OneLake workspace folders
All the data is organized in an intuitive
hierarchical namespace
The data in OneLake is automatically
indexed for discovery, MIP labels, lineage,
PII scans, sharing, governance and
compliance
“The OneDrive for Data”

24. Data Factory
Seamlessly connect to
data stores
Data Factory

25. Mirroring of External Databases
linking of external
databases, with full replicas
created with a couple of clicks
Available for both multi- cloud
and on-premises databases
Real time updates of the replicas
using the CDC feeds of the
database
Data is stored in Delta Parquet
tables, with all Fabric services
instantly available
Amazon GoogleAzure

26. OneLake

27. Sharing data in OneLake is as easy as
sharing files in OneDrive, removing the
needs for data duplication
With , data throughout OneLake
can be composed together without any
data movement
Shortcuts also allow instant linking of
data already existing in Azure and in
other clouds, without any data duplication
and movement, making
With support for industry standard APIs, OneLake data can be directly accessed by
any application or service
Multi-Cloud Shortcuts
Customers
360
Finance
Service
Telemetry
Business
KPIs
Data
Factory
Data
Engineering
Data
Warehouse
Data
Science
Real-Time
Intelligence
Power BI Partner
workloads

28. OneLake

29. OneLake

30. OneLake

31. Iceberg support in OneLake
Bi-directional data access
Seamless access from M365
and Copilot

32. Unify your data in OneLake
Bi-directional data access
Seamless access from M365
and Copilot

33. Open and
Governed Lakehouse

35. Developer friendly
to all data in OneLake
OneLake

36. The API for Accessing Fabric Data
Single endpointfor querying data from
On e La ke
GraphQL ensuresconsistent and
predictable results. Clients control the
shape of the response, eliminating
surprises and making it easier to build
stable applications
Request exactly the information needed,
streamlining data access and reducing
unnecessary round trips
Flexibility empowering teams to tailor
data retrieval to their specific
requirements, improving efficiency

37. Reusable Custom Business
Logic in Microsoft Fabric

38. User Data
Functions
Reflex
Triggers T-SQLData FlowsNotebooks
Pipeline
Activities
Event-Driven
Actions
Warehouses Lakehouses Mirrored DBs
Invocable from many Fabric items
Simple programming model
Access to Fabric data sources
Developer-friendly experience
Reusable, discoverable via Hub
Security, governance, capacity
consumption
User Data Functions in Microsoft Fabric

39. The unified data platform for the era of AI
Complete
Analytics
Platform
Everything, unified
SaaS-ified
Secured and governed
Lake Centric and Open
OneLake
One copy
Open at every tier
Empower Every
Business User
Familiar and intuitive
Built into Microsoft 365
Insight to action
AI
Powered
Copilot accelerated
Gen AI on your data
AI
-driven insights

42. The unified data platform for the era of AI
Data
Engineering
Data
Warehouse
Data
Science
Real-Time
Intelligence
Power BI
Partner
workloads
Data
Factory

43. The unified data platform for the era of AI
Complete
Analytics
Platform
Everything, unified
SaaS-ified
Secured and governed
Lake Centric and Open
OneLake
One copy
Open at every tier
Empower Every
Business User
Familiar and intuitive
Built into Microsoft 365
Insight to action
AI
Powered
Copilot accelerated
Gen AI on your data
AI
-driven insights

44. Gen AI accelerates your data journey in Fabric
AI-driven
insights
Copilot accelerated
experiences
Custom
generative AI
for your data

46. AI Powered

47. on your data
Deliver custom
generative AI experiences
for
Enable custom Q&A on your
data in Fabric
Define custom business semantics
and grounding
Scale the custom experiences to
, , and

48. 4 days of
Microsoft Fabric
Learning, Connection
and Inspiration
24-27 September 2024
Stockholm, Sweden
Learn more:
https://aka.ms/FabCon-Europe

49. CSU Migration Factory
Explained
Lakehouse Migration
Offerings
Power BI Migrations
Re al-Time Intelligence
01
02
03
04

50. CSU Migration
Factory Explained

51. A Microsoft CSU delivery model to provide
rehost* migration of Apps, Infra, and Data
workloads at Zero cost to customers.
Modernize your applications and data to
accelerate time to market and deliver new
experiences.

* 7 Options To Modernize Legacy Systems (gartner.com)
Migrate or modernize first? - Cloud Adoption Framework | Microsoft Learn
Data & AI Global Solution Architecture52
What is the CSU
Migration
Factory?

52. Get your first
workloads running
in Azure in weeks
Expert Guidance and Delivery
Zero Cost to Customer
Accelerated Migrations
No Minimum Requirements
53 Data & AI Global Solution Architecture

53. CMF | Workloads & Execution Focus
54 Data & AI Global Solution Architecture
Jumpstart Azure journey for Apps, Infra, and Data workloads through Microsoft- owned delivery at Zero Cost
WS + SQL +
Linux
(including Arc
enabled)
Native
AVD
NoSQL & OSS
Databases
App
Migration
Analytics AI Security
•Rehost/Refactor
migration From On-
prem, AWS, GCP,
Hosters; To: Azure SQL,
Azure VMs
•Upgrade Win OS if
applicable
•Automated scripts for
Arc Enabled
deployment
•Azure VMware Solution:
Factory to migrate
servers, apps, and DBs
into AVS
•Modernize on-
prem RDS
to AVD
•Migrate on-
prem Citrix to
AVD (currently
in incubation)
NoSQL:
•On-prem Cassandra
to Azure MI for
ApacheCassandra
•On-prem MongoDB
to Azure Cosmos DB
for MongoDB (vCore)
OSS Databases:
•MySQL/PostgreSQL
to AzureDB for
MySQL/PostgreSQL
•Single
Server toFlexServer
•.NET, Java Apps
On-Prem to PaaS
•Non .NET
workloads
(Containerized)
workloads On-
Prem to AKS/ACA
•Apps/Self Hosted
K8’s running on
Azure VM’s to
PaaS
•WordPress
migration to
AppService
•Lakehouse
deployment (Data
migration,Build
MVP for initial use
case)
•SQL Server
Reporting Services
to Power BI
•SSAS/Analysis
Services to Power
BI
•P SKU to F SKU
migration
•Real-Time
Analytics
•Deployment of
AOAI use cases:
Conversational
AI/Search,
Virtual Assistant,
Doc Intelligence,
Personalized
content, Image
Analysis
•POC, Landing
Zone for AOAI,
Prod
deployment,
Solution
Optimization
•Defender for Cloud
deployment – cloud
security posture
management
•Deployment of Cloud
workload protection:
Defender for Servers,
Azure SQL, Storage;
•Configuration of
monitoring
components for
automated data
collection
Current localized coverage:
All Time Zones: English
ASIA: Chinese, Japanese
EMEA: Germany, French
LATAM: Spanish, Portuguese •Customer Sponsorship secured
•Scope is confirmed and aligned with CMF scope
•Active MSX Opportunity –
Workload aligned Milestones (for Managed accounts)
•All customers, any size Migration (no minimum size)
•Execution Method (Hands-
on-Keyboard or Screen-Share
guidance)
•Nomination form: https://aka.ms/CMF
Nomination Acceptance Criteria:

54. CSU Migration Factory for Analytics Offerings
SSAS/AAS to PBI Premium
SSRS to PBI Premium
Lakehouse - Fabric
Fabric Real-Time Intelligence
Lakehouse - Databricks
55 Data & AI Global Solution Architecture
P SKU to F SKU

55. Accelerating Adoption through CSU Migration Factory for Analytics
Fabric Databricks
Data & AI Global Solution Architecture56
Lakehouse
•Offerings:
•Fabric Lakehouse
•Fabric Lakehouse + DW
•Scope:
•Lakehouse medallion architecture with bronze, silver, and
gold layer.
•Transformations with Spark notebooks.
•Orchestration of notebooks with Azure Data Factory or
Fabric Data Factory.
•Silver and/or Gold layer can be builtin FabricDW
•One basic Power BI report to demonstrate how to connect
Power BI reports to Gold layer
•How: Leverage repeatable IP to accelerate establishing Lakehouse
environment, migrate data and rewrite scripts leveraging
repeatable components •Offerings:
•Lakehouse
•Unity Catalog
•Scope:
•Lakehouse medallion architecture with bronze, silver, and
gold layer.
•Transformations with Spark notebooks.
•Orchestration of notebooks with Azure Data Factory or
DeltaLiveTables.
•How: Leverage repeatable IP to accelerate establishing Lakehouse
environment, migrate data and rewrite scripts leveraging repeatable
components

56. Accelerating Adoption through CSU Migration Factory for Analytics
Power BI
•Offerings:
•ADX
•Fabric
•Scope: Migrate data to Fabric using scripts, pipelines, streaming
features or agents. Big Data workloads such as Telemetry, IoT,
Cyber/App Logs, Timeseries, Metrics, Geospatial, Graph,
Embedding Vectors, High-granular, Discrete analytics.
•How: Analyze requirements and help you determine the optimal
alignment. Assess business needs, current platform and existing
architecture.
Real-TimeIntelligence
•Offerings:
•SSRStoPBI
•SSAS/AASto PBI
•PSKU toFSKU
•Scope: SQL Server Reporting Services(SSRS) & Analysis
Services (SSAS/AAS) can be migrated easily to Power BI. Migration of P SKU to F SKU workspaces in the same region
or another region with considerations.
•How: Leverage 1
st
party tooling to migrate customers out of
legacy solutions like SSRS & SSAS/AAS into Power BI &
Fabric
Data & AI Global Solution Architecture57
Additional Fabric Offerings

57. Lakehouse
Migrations

58. Lakehouse
Medallion Architecture
The Medallion Architecture describes a
series of data layers that denotes the quality
of data stored in the Lakehouse. This
architecture guarantees atomicity,
consistency, isolation and durability as data
passes through multiple layers of validation
and transformations being stored in a
layout optimized for efficient analytics.
Key Capabilities:
•Ingest raw data to the Bronze layer
•Validate and deduplicate data in the Silver layer
•Power analytics with the Gold layer
Lakehouses are a single location for data engineers, data scientists, and data analysts to
access and use data.
Data & AI Global Solution Architecture60

59. Lakehouse – In Scope
Data Sources Orchestration and Transformation
Azure Data Factory
Fabric Pipelines
FO
Notebooks
PySpark
SparkSQL
Stored Procedures
Delta Live Tables
DO
Unity Catalog
DO
FO
Fabric only
Azure SQL DB a nd SQL MI
PostgreSQL
MySQL
Oracle
SQL Server ( on-premises)
Flat Files
Hadoop
AWS Re d shift
Dedicated SQL Pool
FO *
Google Big Query
FO
Fabric Shortcuts
FO
DO
Databricks only
*
Lift and Shift is not currently supported for Dedicated SQL Pool. See details in Appendix
Data & AI Global Solution Architecture61

60. Lakehouse Project Timeline
1
Requirements: Local CSA, Corp
Factory Team Lead and Customer will
meet to discuss requirements of the
program
2
Design: Local CSA, Corp Fa cto ry
Team Lead and Customer will work
together to design a high value use
case
3
Implementation: Development team
will be doing the work
Key Phases /
Milestones
Start
Date
End
Date
Week 1 Week 2Week 3 Week 4Week 5Week 6Week 7Week 8
Requirements Week 1Week 2
Design Week 2Week 3
ImplementationWeek 3Week 8
Testing Week 5Week 8
Handover Week7Week 8
Requirements
Design
Implementation
Testing
Handover
Te s t in g : Development team and
Customer should be doing iterative
testing
Handover: Development team will do
knowledge transfer sessions
4
5
Data & AI Global Solution Architecture66

61. Power BI
Migrations

62. Migrate .rdl reports and SSRS PBI interactive reports
from SSRS servers to PBI Premium
Migrate SSAS/AAS models to Power BI Semantic
Models
Publish reports that pass checks as PBI Paginated
Reports
Automation and Business Operations, such as .Net
Code, SSIS Packages and Azure Data Factory
Governance and Security such as PBI capacity
governance, workspace config and role membership
Setup and training on Optimization methodology,
tenant management, release management,
monitoring, alerting, post deployment oversight
Out of Scope
In Scope
Migrate P SKU (PBI Premium) to F SKU (Fabric)
PowerBI
Scope
Data & AI Global Solution Architecture68

63. Data & AI Global Solution Architecture72
P SKU to F SKU Migration Scenarios
Migration Scenario Supported
Migrating workspaces having only Power BI items
- within the same region
Ye s
Migrating workspaces having only Power BI items
- To a different region
Ye s
Migrating workspaces having Fabric items
- within the same region
Ye s
Migrating workspaces having Fabric items
- To a different region
No - you must delete all the Fabric items
from the workspace first.
Cross Tenant Migration No

64. Data & AI Global Solution Architecture73
Power BI Project Timeline
1
Requirements and Design: Local CSA,
Corp Factory Team Lead and Customer
will meet to discuss requirements and
design of the program
2
Deployment and Implementation: Development team will be doing the work
3
Testing and Handover: Development team and Customer should be doing iterative
testing and completing the handover
Key Phases /
Milestones
Start
Date
End
Date
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6
Requirements &
Design
Week 1Week 2
Deployment &
Implementation
Week 2Week 5
Testing &
Handover
Week 5Week 6
Requirements & Design
Deployment & Implementation
Testing & Handover

65. Real-Time
Intelligence

66. Real-Time Intelligence
Eventhouse
RT DashboardKQL Queryset
Power BI
Analyze &
Transform
Eventstream
Ingest & Process
Reflex
Act
Real-Time Hub
OneLake
Digital Operations, Observational,(I)IoT+
high-granular,discreteanalytics
Streaming, minimal-latency, data in- motion, predictive analytics
Real-Time Intelligence

67. Real-Time Intelligence – In Scope
IoT
SignalR websockets
REST-APIs
Kafka, Flink, Redpanda, Druid
Splunk can forward to Fabric for analytics
Elasticsearch
Sentinel using continuous export or setup parallel-ingestion
Azure Database Watcher
InfluxDB by leveraging telegraph kusto connector
Aveva OSI-PI
AWS Kinesis, AWS Timestream, Confluent, Google Pubsub, Spark streaming
Azure Time SeriesInsights (retiresJuly 7, 2024)
Azure AI Metrics Advisor -Anomaly Detection (retiresOctober 1, 2026)
Snowflake,Google Big Que ry, IBM DB2 when data is timeseries, logs or telemetry
KSQ L, Singlestore, Clickhouse, Datadog, Newrelic, Dynatrace & Pinot
Grap hDBs such as Neo4j& Tigergraph
VectorstoreDBs such as Weaviate, Qdrant, Chroma, Milvus
CDC scenarios
Azure PostgreSQL
Cosmos DB
Azure MySQL
Azure SQL Database
OracleGoldengate via EH connector
Data Sources Interface Patterns
Data & AI Global Solution Architecture76