Row vs Columnar format - slides (supplementing the YouTube video)

Presentation uploaded to supplement the youtube video:
https://youtu.be/a38Bj7BCWFg

1. Operational vs Analytical plane
Analytics Operations

2. Analytics Operations
●Operate on a single object
●Smaller data structures
●Very sensitive to latency
●Many concurrent users
●Long & wide datasets
●Not latency sensitive
●Few users
●Queries often target only a subset
of the data
●Batch processing is popular
Disclaimer:
I’m not describing user facing analytics here

3. ColumnRow
VS
Storage orientation

5. Column
orientati
on
Row
orientation

7. Row
orientation

8. Row
orientation
●Efficient writes on a single object

9. Row
orientation
●Efficient writes on a single object
●Efficient sequential reads
●Not efficient for analytical use
cases

10. Column
orientati
on
Avro

11. Avro
●binary row-oriented format

12. Avro
Write schema
Object 1
Object 2
Object N
Avro file
…
Kafka Topic
…
schema id
Object 1
data
Schema registry
…
Object N Schema N
Schema 1
Schema 2

13. Avro
●binary row-oriented format
●contains a schema and data
○in kafka: schema id + data
●supports schema evolution

14. Column
orientati
on
Column
orientation

17. Column
orientation
●Easy to skip columns if you don’t need them
●If you summarize your columns, better analytical query performance
●More efficient compression

18. Column
orientati
on
Parquet

19. Parquet file
…
Row group 1
…
Parquet file structure (high level)
Row group 1
Row group 2
Row group N
Footer (file, row group & column metadata)
Column B chunk 1
Column A chunk 1
Column C chunk 1
Column N chunk 1

20. Parquet
●Binary columnar format
●Widely adopted in Data Lakes and Lakehouses
●Highly performant for analytical use cases
●Very efficient compression
●Performance degrades if too many small files

21. ?
Is Avro a valid choice
for long term
storage?

22. Compression efficiency
●Benchmark with two datasets:
○wide: 194GB data, 103 columns
○narrow: 82.8 million rows, 3 columns
●For narrow comparable space efficiency
●Wide dataset space efficiency:
○Parquet compressed to 4.7 GB
○Avro to 16.9 GB
https://blog.cloudera.com/benchmarking-apache-parquet-the-allstate-experience/

23. Long term storage?
●Data size is a primary driver of your costs
●In Avro, compression efficiency is highly sensitive to batch size
●For use cases where storage cost is not dominant, it might make sense to
keep data in Avro longer for convenience

24. Producer
Application
Analytics
Avro
Kafka
Cluster
S3 (Avro) Parquet
ETL
Kafka Connect
Operations
Data Lake
Data
Pipelines
Consumer
Application

25. Modern table formats
●Iceberg
●Delta Lake
●Time Travel
●ACID