Cube Store Design Decisions OSA Analytic

Cube store design decisions

1. Cube Store Design Decisions ‹#› November 2 , 2021

2. Pavel Tiunov @ paveltiunov87 Co-founder & CTO @ Cube Dev Cube.js and Cube Store author github.com/cube-js/cube.js

3. User-facing analytics ‹#›

4. User-facing analytics ‹#›

5. Uber Eats eng.uber.com/restaurant-manager

6. How to build a customer-facing analytics data app ‹#›

7. Pipelines ‹#›

8. Pipelines: pros & cons Criteria Virtually unlimited data size ✅ Sub-second query latency ✅ High query concurrency (1000+ QPS) ✅ Low ingestion latency ✅ Low ingestion pipeline maintenance cost ❌ Cheap data replays ❌ Scalable join support ✅ ‹#›

9. Time series databases ‹#›

10. TSDBs: pros & cons Criteria Virtually unlimited data size ✅ / ❌ Sub-second query latency ✅ High query concurrency (1000+ QPS) ✅ / ❌ Low ingestion latency ✅ Low ingestion pipeline maintenance cost ✅ Cheap data replays ✅ Scalable join support ❌ ‹#›

11. OLAP datastores ‹#›

12. OLAP datastores: pros & cons Criteria Virtually unlimited data size ✅ Sub-second query latency ✅ High query concurrency (1000+ QPS) ✅ Low ingestion latency ✅ Low ingestion pipeline maintenance cost ❌ Cheap data replays ✅ / ❌ Scalable join support ✅ ‹#›

13. Cloud data warehouses ‹#›

14. Cloud DWHs: pros & cons Criteria Virtually unlimited data size ✅ Sub-second query latency ❌ High query concurrency (1000+ QPS) ❌ Low ingestion latency ❌ Low ingestion pipeline maintenance cost ✅ Cheap data replays ✅ Scalable join support ✅ ‹#›

15. Enhance Cloud DWH so it can be as performant as OLAP Store Save original Cloud DWH user experience Solution 🦄 ‹#›

16. Solution 🦄 Criteria Virtually unlimited data size ✅ Sub-second query latency ❌ ⇒ ✅ High query concurrency (1000+ QPS) ❌ ⇒ ✅ Low ingestion latency ❌ ⇒ ✅ Low ingestion pipeline maintenance cost ✅ Cheap data replays ✅ Scalable join support ✅ ‹#›

17. Enter Cube Store ‹#›

18. Billions of rows as an input Sub-second response time High query concurrency Input data always have unique primary key Sources are both batching and real-time streaming Joins between data sources Requirements ‹#›

19. Cube Store Architecture ‹#›

20. High throughput Meta Store Indexes are sorted copies Auto partitioning Parquet as a format Distributed FS as a storage layer Shared-nothing architecture Collocated join indexes Real-time in-memory chunks Tools: Rust, Arrow, DataFusion, RocksDB Design decisions ‹#›

21. Design decisions matrix Billions of rows as an input Sub-second response time High query concurrency Input data always have unique primary key Sources are both batching and real-time streaming Joins between data sources High throughput Meta Store ✅ ✅ Indexes are sorted copies ✅ ✅ ✅ ✅ Auto partitioning ✅ ✅ ✅ ✅ Parquet as a format ✅ Distributed FS as a storage layer ✅ ✅ Shared-nothing architecture ✅ Collocated join indexes ✅ ✅ ✅ ✅ Real-time in-memory chunks ✅ ✅ ✅ ‹#›

22. High Throughput Meta Store

23. High Throughput Meta Store Most accessed data in Cube Store Relatively small to data stored but still can be 10M+ of rows in size Strong read after write consistency Heavy read oriented but with significant write concurrency Should provide RDBMS experience RocksDB is a great tool for a problem

24. Meta Store Tables Schema Table Index Partition Chunk Job Source

25. Meta Store Access Flow

26. Meta Store key-value structure

27. Indexes are sorted copies

28. Indexes are sorted copies Sorting is the most efficient way to compress data for columnar data structures Most optimal for filtering Index per query is the way to go Paying write amplification for the speed of reads is ok Every algorithm can be merge-based: merge sort, merge join, merge aggregate.

29. Indexes are used based on filters

30. Data compression by sorting

31. Data compression by sorting

32. Auto partitioning

33. Auto partitioning Helps to keep all partitions relatively the same size Easy to implement as everything is sorted

34. Partition split at compaction

35. Parquet as a format

36. Parquet as a format De facto standard for storing columnar data Implements Partition Attributes Across (PAX) [1] Allows to leverage min-max statistics for filter push down optimizations [1] https://www.pdl.cmu.edu/PDL-FTP/Database/pax.pdf

37. Parquet file structure

38. Distributed FS as a storage layer

39. Distributed FS as a storage layer Storage-compute separation Compute nodes can be disposable and interchangeable Easy to build replication implementation Warmup everything that will be accessible by queries just before table becomes online to avoid download wait time

40. Distributed FS as a storage layer

41. Partition Warmup

42. Real-time in-memory chunks ‹#›

43. Real-time in-memory chunks Act as a buffer to avoid single row writes to Parquet Streaming rows sent to the partition owner worker In-memory chunks compacted at 1 minute intervals t o a persisted Parquet partition ‹#›

44. Routing for in-memory chunks ‹#›

45. Shared-nothing architecture ‹#›

46. Shared-nothing architecture Worker nodes never communicate with each other Router node receives only aggregated results and just needs to merge them ‹#›

47. Shared-nothing architecture ‹#›

48. Lambda architecture query ‹#›

49. Collocated join indexes ‹#›

50. Collocated join indexes Fastest distributed join algorithm Can be also used by other algorithms like shared-nothing Top-K ‹#›

51. Collocated join indexes ‹#›

52. Collocated join indexes ‹#›

53. Learn more about Cube & Cube Store @paveltiunov87 · c ube.dev Thanks! 🙌 ‹#›