OpenMetadata Community Meeting - 5th June 2024

The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.

* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail

Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E

1. OpenMetadata
Community Meeting
June 2024 ??????
The Past, Present, &
Future of Data Quality

2. ●Community Updates & Metrics
●Community Contributions!
●The Past, Present and Future of Data Quality ??????
●Q&A


Agenda

3. 4364
GitHub Stars
Open Source
Developers
253
Community
Members
5838
Community
2313
+5
+150
+224
+61
Community Stats

4. ●500 Qs in the last 4 weeks
●539 active members
Community Metrics
●250 PRs in the last 4 weeks

5. ?????? Siddhant Tripathi
??????Antoine Balliet
??????gpby
??????Maxim Martynov
??????Gaetan Soulas


9 Community Contributions ??????
??????Huanjie Guo
??????Fredrik Möllerstrand
??????Christian Berge
??????Zhang Juntao

6. ?????? Real time Notifications: get your stakeholders informed in case of failures. Reduce alert fatigue
with Test Suite by grouping related tests together

?????? Native Observability Metrics: enable data users to understand the shape and the structure of
their data (SQL, DataLake and NoSQL Connectors)


What we have built (since v0.10)
✅ No Code Data Quality: empower data users to configure quality checks in a few clicks across all
your SQL and DataLake connectors


⚠ Integrated Incident Management: create incidents directly in OpenMetadata to notify data consumers
and producers of ongoing data issues
?????? Rich RBAC: control who can view, edit, and create data quality in your organization.

?????? Root Cause Analysis: view a sample of row failing a test case condition for faster resolution

?????? Data Health Dashboard: track data quality performance across the organization

⚙ 3rd Party Integration and Rich API: natively import your dbt and GX test case results. Centralize
your results from your own DQ framework with the API.

7. Demo

8. Provide a complete observability solution
across your entire data stack through AI powered testing and
automation, impact analysis, and root cause analysis
to drive efficiency and build trust.

9. ?????? Minimize the Effort for User to Start their Quality and Observability Journey
●AI backed dynamic rules assertions [1.5]
●Automation rules using the Automator in Collate
●AI powered test suggestions based on asset characteristics

?????? Lower Time to Resolution
●Visualize upstream failure from the entity page [1.5]
●Set a dimensionality to a test case for more granular checks
●Dimensionality correlation analysis on test case failures
●3rd party incident management integration (PagerDuty, Jira, ServiceNow, etc.)

What we are building (1.5 and beyond)

10. ?????? Provide the Right Insight for Decision-Making
●Incident resolution health dashboard to show the metric that matters [1.5]
●Data Quality Coverage metric to understand the current state of Data Quality [1.5]

?????? Build Trust Across Your Entire Data Stack
●Freshness and Integrity data quality tests [1.5]
●Data Quality Monitoring for your NoSQL Sources
●Monitoring for your pipelines (runtime and status incident) [1.5]

What we are building (1.5 and beyond)

11. Unified Platform for data discovery,
observability and governance

12. Star us on GitHub
https://github.com/open-metadata/OpenMetadata

Join our Slack
https://slack.open-metadata.org/

Follow us on X
@open_metadata

Discover Collate SaaS
https://www.getcollate.io/