Garbage In, Garbage Out: Why poor data curation is killing your AI models (and how to fix it)

Enterprises have traditionally prioritized data quantity, assuming more is better for AI performance. However, a new reality is setting in: high-quality data, not just volume, is the key. This shift exposes a critical gap – many organizations struggle to understand their existing data and lack effective curation strategies and tools. This talk dives into these data challenges and explores the methods of automating data curation.

1. 16th July 2024

2. Agenda
I.Introduction
II.Why quality data matters
III.Data cleaning and curation trends
IV.The steps to producing Quality data
Webinar

3. Speaker

Alexandre Bonnet
ML Solutions Lead

[email protected]

4. *Encord operates on 'Salesforce quarters'—all reported figures align with our fiscal year
Manage and select the right
data for your AI application
USERS
Data Engineer, ML Engineer
Orchestrate human-in-the-loop
workflows for ground truth,
ranking (RLHF), and, validation

USERS
Domain Expert, AI Trainer, Labeler
Find edge cases & debug model
errors, discover high-impact
data to boost performance

USERS
AI Researcher, ML Engineer
CONTINUOUS DATA SELECTION FOR TRAINING, FINE-TUNING & VALIDATION
IDENTIFY & FIX DATA & MODEL ERRORS
DATA SELECTED FOR HUMAN REVIEW VALIDATE QUALITY OF DATA
We are a data development platform that
helps AI teams automate data in a single workflow

5. The Future of AI is in the present

6. The Future of AI is in the present… Sort of

7. The Future of AI is in the present… Sort of

8. encord.com
Build from scratch

Uni-modal

Model Centric

Technical-users

Traditional AI Modern AI
Build on Foundations

Multi-modal

Data Centric

General users

9. Pillars of AI
Data: Foundational information for models, guiding AI
learning and determining task specifications and
accuracy.


Models: The "brains" of AI, using algorithms to process
information and make decisions.


Compute: The hardware and resources required to
train and run models, handling complex calculations
and data processing.

10. While models and compute
are evolving rapidly

Data is still in its infancy
and AI teams’ main
competitive advantage
Models in one config file
Compute in one config file
encord.com

11. Foundation models
●YOLOv8-v10
●RCNNs
●LLAVA
●Segment Anything
●GPT4o
●Claude
●Etc

From Big data to Small data.

Rest assured, you don’t have to
start from scratch -

Presenting, Foundation models.

12. encord.com
Why Data Quality
Maers

13. Bad Data Bad Model
=

14. Great Data Great Model =
… Not (necessarily) Big Data

15. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

16. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

17. Small data, but HIGH quality data…

18. Small data, but HIGH quality data…

https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/
“The use of natural language in training robots is still in its infancy, particularly in autonomous driving.”
“Incorporating language along with vision and action may have an enormous impact…”

19. Domain Specific Fine-tuning - Medical
There are three critical issues
identified that requires your aention
2048x A100s 8x A100s

20. Domain Specific Application - Medical

21. Definitions
Refers to the process of identifying and
correcting errors or inconsistencies in the
data.

This can include:

●Removing noisy and corrupt data
●Handling missing or duplicate data
●Standardizing and preprocessing the
data to ensure quality

Refers to the organization and management
of the data to make it suitable for labeling or
model training.

This can include:

●Selecting data for labeling
●Pre-labeling and enriching data
●Ensuring that the data is error free and
high quality

Data cleaning Data curation

22. encord.com
Data quality &
curation trends

23. Data Collection
Data
Verification
Serving
Infrastructure
Annotation
tool
Model Training
tool
Model
Evaluation
tool
Data
Warehouse
External Tool
Stakeholders involved:
Data science, Annotation
Data volume:
10-100 thousand frames per year
What we used to see in data pipelines

24. Data Collection
Data
Verification Label Quality
tool
Data Curation
tool
Serving
Infrastructure
Versioning
tool
Monitoring
tool
Annotation
tool
Model Training
tool
Model
Evaluation
tool
Data
Warehouse
Data
Visualisation
tool
Workflow
Orchestration
tool
External Tool
Data Cleaning
tool
Stakeholders involved:
Data science, DataOps, MLOps,
Annotation, Leadership
Data volume:
10 million frames per year or more
What we now see in data pipelines

25. Data Collection
Data
Verification Label Quality
tool
Data Curation
tool
Serving
Infrastructure
Versioning
tool
Monitoring
tool
Annotation
tool
Model Training
tool
Model
Evaluation
tool
Data
Warehouse
Data
Visualisation
tool
Workflow
Orchestration
tool
External Tool
Data Cleaning
tool
Today’s focus

26. Manual inspection
One-off Jupyter scripts
Immature OS tools
Cleaning & Curation
is often cumbersome
Manual video scrolling
One-off Python scripts
What we typically see

27. Operational expenses increase due
to unnecessary labeling and MLOps.
Increased Annotation
/ MLOps Cost


Poor data quality and less time spent
on improving ML models decreases
model outcome.
Poor Model Quality

Data quality issues and overhead
slow down product speed.
Delayed
Time-To-Market
There are downsides without cleaning and curation

28. Read the case study >
How Automotus increased
mAP 20% by reducing their
dataset size by 35% with
visual data curation
Achieve beer model performance with less data

29. Model Accuracy
Model with
Clean data
Model with
Clean & Curated data
Model with
noisy unclean data
↘35%
Dataset reduction
↗20%
mAP increase
Customer improvement
↘33%
Labeling cost decrease
Achieve beer model performance with less data

30. encord.com
So… how can we look
at changing this?

31. Learnings of data quality common errors
●Duplicate samples
●Corrupt samples
●Noisy samples
●Data outliers
Cleaning errors
●Random data selection
●Limited task prioritization
●No pre-labeling data
●No label validation
Curation errors

32. Quality Metrics to help you understand your data
DuplicatesBorder
closeness
Occlusion Object
density
Object Area Polygon
shape
Brightness ContrastImage areaRGB valuesAspect ratio Blur








Model performance by metricMetric distribution
Metrics Outliers2D metrics comparison
40+ Metrics out of the box

33. 6 tools out of the box
Embeddings to help you understand your data

34. Data Cleaning

35. Data Curation

36. Data Curation

37. Metadata Validation

38. Metadata Validation

39. Metadata Validation

40. Pre-trained

41. Fine-Tuned

42. ●The data curation process is key to building high performance,
accurate and domain specific Models.
●This curation becomes exponentially more difficult and time
consuming for complex and specific applications without a solid
pipeline in place.
●Building on foundations allows us to move with speed and efficiency.
●We have seen a shift in the scale and complexity of applications and
unstructured data lakes.
Summary

43. encord.com
Thank you!