30B Images and Counting: Scaling Canva's Content-Understanding Pipelines by Kerry Halupka

Scaling content understanding for billions of images is no easy feat. This talk dives into building extreme label classification models, balancing accuracy & speed, and optimizing ML pipelines for scale. You'll learn new ways to tackle real-time performance challenges in massive data environments.

1. A ScyllaDB
Menster SCALE Summit

40B Images and Counting

Scaling Canva's Content-Understanding Pipelines

_ Dr Kerry Halupka
Principal ML Engineer

2. Dr Kerry Halupka (she/her)

+ Principal ML Engineer at Canva

+ Experience in Al-driven content enrichment & large-scale ML
pipelines

+ Passionate about mentoring & sharing knowledge

Canva

Monster SCALE Summit

3. We have

220 Million+

Monthly Active Users

Across 190 countries, in 100 languages |
—

Monster SCALE Summit

5. Humans see nuance

Monster SCALE Summit

6. Computers... don’t

Monster SCALE Summit

7. Q Energetic business meeting © =

Bad Results Good Results

Monster SCALE Summit

8. Concept
Depth

re. Image
a dul Volume

Monster SCALE Summit |

9. Requirements

> if +

Compute Scalability Performance
Model needs Model needs Model needs
tobe cheap to be fast to be accurate

Monster SCALE Summit

10. Solving for compute

Traditional ML-decoder
Transformer-decoder
Computational
t Transformer-
Group Image decoder
Full Image Queries Embeddings
Queries Embeddings : :
5 1 xD (xD
mio (wn) xo
= Cross-Attention

ESS

on

Cross-attention |

) ML-decoder
no
Y
Food-Forward + Cure
as Connected Pooling Number of
y y Concepts
Net Net

ML-Decoder- Scalable and Versatie Classification Head Tal Ridnik’ Gilad Shari" Avi Ben
Cohen Emanuel Ben-Baruch Asaf Noy DAMO Academy, Albaba Group

janster SCALE Summit

11. Solving for scalability

How do we train anew concept?

e.g. Cottagecore

Hint: Cottagecore is an aesthetic
romanticising a simple, rustic, and
idyllic way of life

Monster SCALE Summit

12. Data Labelling

ing to find
rse
representations

Findnew
Unlabelled Data Pre

containing
concept
Define new

ae Labelimages
Checkallother

imagesin

dataset for

occurrences Labelled Data

Previously labelled
images may contain
the new concept

13. Finding Examples: Beyond Simple Search

Q Cottage Core Q Acozy farmhouse Q Ayoung woman Q Apicnic in the
kitchen with wildflowers woods

Monster SCALE Summit

14. Acory
farmhouse
Kitchen

Finding Diverse Examples

Li

Ti

INT,

+ Input: Multiple text descriptions
+ Output: Varied images capturing
different facets of the concept

15. Solving for Performance

Q Acozy farmhouse
kitchen

16. Improving the Training Dataset

“Acozy
farmhouse
kitchen”
Eu High High
Concept _ "ecall _ visualcritic _., "ecall,
Classification En Model high

precision precision

dataset dataset

Unlabelled Data

“Cottagecore”

17. End-to-End Process

New
concept
Description
generation
= 2 y High High
L. a Candidate _> Lowthreshold_ recall _ visual Critic recall
É CLIP =
mas > 7 “images 7? Concept low Model 77 high
UnlabelledData Classification precision precision

dataset dataset

18. Requirements

> if +

Compute Scalability Performance

Model needs Model needs Model needs
to be cheap tobe fast to be accurate

19. Stay in Touch

Dr Kerry Halupka
pa [email protected]

lin] www.linkedin.com/in/kerry-halupka

J