Segment-Based Storage vs. Partition-Based Storage: Which is Better for Real-Time Data Streaming?

Storage is a critical component of any real-time data streaming system, and the choice of storage model can significantly affect the system's performance, scalability, and reliability. Two popular storage models for real-time data streaming systems are segment- and partition-based storage.

In this talk, we will start by explaining what segment-based and partition-based storage means and how they work. We will explore the differences between the two storage models, including how data is organized, how data is stored, and how data is accessed.

We will discuss how a segment-based storage model provides better scalability, performance, and reliability than the partition-based model and how segment-based storage solves some deficiencies of the partition-based model, including the need to re-partition topics just to increase the storage capacity of a topic.

Attendees will leave this talk with a clear understanding of the differences between segment- and partition-based storage and how they affect real-time data streaming systems' performance, scalability, and resiliency.

1. Segment-Based vs Partition-Based Storage: Which is Better for Real-Time Data Streaming? David Kjerrumgaard Developer Advocate at StreamNative

2. David Kjerrumgaard Developer Advocate at StreamNative Author of Pulsar in Action by Manning Press. My perspective on P99s Committer on the Apache Pulsar project. World-traveler, always looking for the next great place to visit.

3. Data Streaming Platforms

4. The Origin of Data Streaming In 2010, LinkedIn needed a way to ingest large amounts of data from its website and internal infrastructure logs and replicate it to multiple databases with a minimal amount of latency.

5. The Origin of Data Streaming But rather than build a custom solution for every point-to-point integration, they envisioned a more generic solution known as Kafka .

6. The Log The key to Kafka’s flexibility is its use of the log data structure. A time-ordered, append-only sequence of data. The records are stored in the log as byte arrays, which makes them easily serialized/deserialized into just about anything.

7. Wait a Minute… Logs aren’t Revolutionary Logs have been used inside databases since 1981. A database uses a log to write out information about the records they will be modifying, before applying the changes, aka changelog .   The log is the record of what happened, and each table or index is a projection of this history into a database table or index. Since the log is immediately persisted it can be used as the authoritative source in restoring all other persistent structures in the event of a crash.

8. Database Logs: Fast, Single Consumer Records are read from the changelog and applied to the table using ACID semantics to guarantee consistency. Once these changes have been applied, the record is deleted from the log. Consequently, logs inside a database are intended to retain the records for only a short period.

9. Data Streaming Logs: Multiple Consumers Unlike database logs, streaming logs have multiple consumers that consume at different rates. To accommodate for lagging consumers, streaming platforms were designed to retain data inside their logs for longer periods of time. Retention periods of days, weeks, or even longer are common. Streaming platforms acted as a buffer for data ingestion.

10. Stream Processing is Born These longer retention periods led to the desire to process and analyze the data inside these streaming platform logs, aka Stream Processing . Stream processing allows users to perform queries that cannot be addressed by traditional, state-based databases; Point-in-time queries such as, “what was the inventory level for this SKU at 8:50 am?” Window analysis such as, “what is the average sensor reading over the five-minute period from 10:41 am to 10:46 am?” This makes data streaming platforms event-based databases .

11. Stream Storage: Partitions vs. Segments

12. Horizontally Scalable Log Storage: Longer data retention periods, coupled with the volume of data being ingested, necessitated a log capacity that exceeded the boundaries of a single physical machine. Therefore, a horizontally scalable storage solution was required to accommodate current, and future data volumes. There are two approaches to horizontally scalable storage: Partition-Based Segment-Based

13. Partition-Based Stream Storage The partition-based storage approach distributes a single logical log across several smaller physically logs known as partitions. These partitions are distributed across several multiple machines, which allows the storage capacity of a single log/topic to exceed that of a single machine. The storage of the logical log can be expanded by adding more partitions.

14. Partition-Based Storage – Data Distribution

15. Partition-Based Storage – Physical Placement The topic partitions are distributed across the brokers in Kafka, and the local disk(s) are used to store the messages. Often, a Kafka broker is used to store multiple partitions at the same time. This makes the broker’s local disk a shared resource across multiple topics.

16. Partition-Based Stream Storage: Pros/Cons The storage model is that it is simple and fast. All reads and writes are sequential. [PRO] A single broker must have enough storage to cope with that replica, so very large replicas can force you to have very large disks. [CON] Scaling out your storage by adding partitions requires a “rebalance”. During this process, the entire topic is offline. [CON]

17. Segment-Based Storage The segment-based storage approach decomposes a single logical log into several smaller pieces known as segments , which contain ~50K records. These segments are then stored on a separate, horizontally scalable storage layer. This allows a single logical log to have the storage capacity of multiple machines. The storage of the logical log can be expanded by adding more nodes to the storage layer.

18. Segment-Based Storage Segments are further broken down into fragments . Fragments contain the individual log entries.

19. Segment-Based Storage – Data Distribution

20. Segment-Based Storage – Physical Placement In Pulsar, segment-based storage is always kept on a separate layer. A metadata store retains an ordered list of the segments in a topic.

21. Segment-Based Stream Storage: Pros/Cons The data of a given topic is spread across multiple machines. Distributing reads/writes across multiple disks/disk controllers. Each bookie need only serve a subset of read/write requests, which can increase total throughput and lower latency. When you need to grow your cluster, just add more bookies and they’ll start getting written to when new segments are created. No rebalancing required.

22. Performance Comparison

23. Segment vs. Partition Read Throughput Segment-Based Stream Storage achieved 2.5x the maximum throughput of a Partition-Based Streaming Platform

24. Segment vs. Partition Latency Segment-Based consistently provides single-digit publish latency that is 100x lower than a Partition-Based Streaming Platform

25. Historical Reads Segment-Based stream storage has a historical read rate that is 1.5x faster than a Partition-Based Streaming Platform

26. David Kjerrumgaard [email protected] @DavidKjerrumga1 https://streamnative.io Thank you! Let’s connect.