Timeseries Storage at Ludicrous Speed by Duarte Nunes

Datadog’s real-time storage system for timeseries data ingests billions of points per second and serves thousands of queries per second with sub-second latencies. This session will describe how the system is designed, how it has evolved over time, and the techniques we use today to meet our performance goals.

1. A ScyllaDB Community
Timeseries storage at
ludicrous speed
Duarte Nunes
Staff Engineer

2. Duarte Nunes (he/him)

Staff Engineer at Datadog
■Background in distributed systems
■Rewrote way too many bespoke databases in Rust
■Into landscape/wildlife photography and
letterpressed books

3. Billions datapoints / second
Millions of hosts

4. High-level overview of the Metrics Platform

5. An RTDB cluster

6. Inside an RTDB node

7. M nocle

8. Data Model: (org, metric, hash(tags)) -> [(timestamp, value)]

9. Thread-per-coreish

10. A Monocle worker instance

11. Memtable

12. A Monocle file set

13. Files & Compaction

14. Cache

15. Throttling queries under load
Gated queries and cost-based scheduling
■Gates allow admission based on:
●Ingestion lag
●Available memory
●Overall concurrency
■Cost-based scheduling
●Uses CoDel to manage latency
●Queries subject to timeout

16. Partitioning
By partitioning on a hash, queries are executed by all workers
■Parallelism is great?
●Amdahl's law
●Head of line blocking

17. Rust
We’ve had lots of success building on Rust and Tokio
■Intermediate schedulers woes (e.g., FuturesUnordered)
●Buffering
●Yielding
■Future cancellation is hard to reason about

18. Looking ahead

19. Smarter Routing
Move to a more dynamic load-balancing system
■Current routing is inspired by Google’s Slicer
●Balances PPS
●Slow to adjust
■Data movement to adapt to query pattern changes and bursts

20. Collocating Points and Tags
Single realtime database, stop indexing everything
■Columnar data
●Read only what’s being queried
●Series-per-row
●Great for SIMD
●Compression techniques that allow random access to the data (e.g.
FSST)
■No more thread-per-core?

21. Thank you! Let’s connect.
Duarte Nunes
[email protected]