Quantifying the Performance Impact of Shard-per-core Architecture

Most software isn’t architected to take advantage of modern hardware. How does a shard-per-code and shared-nothing architecture help – and exactly what impact can it make? Dor will examine technical opportunities and tradeoffs, as well as disclose the results of a new benchmark study.

1. Quantifying the Performance of Shard Per Core Architecture Dor Laor Co-Founder of ScyllaDB

2. Dor Laor Co-Founder of ScyllaDB Something cool I’ve done: Hello world, Core router KVM, OSv, ScyllaDB My perspective on P99: Less is more Another thing about me: Aspiring MTB rider, orthopedic patient What I do away from work: Fight for democratization

3. Let’s Shard

4. Past, Present & Future Me: router FIB, d evice drivers, kernel load balancer Fault tolerance virtualization using lock step, single VCPU V irtio, no false cache-line sharing PV ticketlocks reduce TLB flush In-kernel apps, no syscalls Van Jacobson networking Shard-per-Core Seastar & ScyllaDB

5. Locking is Evil

6. The LOCK# Signal

7. The ‘Bible’ Documentation

8. Split Locks (detection) Bus Lock of Unaligned buffers still hurts

9. Locking & NUMA & Virtualization

10. Locking & NUMA & Virtualization - Still

11. How much will it take to invalidate the cache?

12. Mutex::lock() No contention In local cache No contention In remote cache ~ 5ns ~ 50ns C ontention Unbounded based on lock holder Time P

13. Locking is Evil Round-Trip is Evil

14. Cost of cross-core RTT

15. RTT measurement; Delayed Branch blog

16. AWS C5.metal, (28-4)*2 cores, Cascade Lake 18 hop RTT worse case Hyper threads, share L1

17. AWS C5.metal, (28-4)*2 cores, Cascade Lake

18. Ice Lake, AWS C6i.metal, 2*32 cores, 128 vcpu

19. Ice Lake, AWS C6i.metal, 2*32 cores, 128 vcpu hyperthread RTT Within die mesh interconnect RTT Cross NUMA node UPI interconnect RTT

20. Single access -> JEmalloc variability Small object From Tcache Time P Tcache miss Arena access No lock contention ~ 25-50ns ~ 200ns ~ 400ns Arena access w/ lock contention Arena miss call mmap( ) ~ 2000ns ……..

21. P99 Latency == Variance Sources of evil: Numa/NUCA Remote cpu RTT Locking TLB flush System call Interrupts Contention

22. “Intellectuals solve problems, geniuses prevent them” LWN Link

23. Birth of Our Shard per Core Locks are evil attitude Van Jacobson's Net channels in OSv Pivot to databases, solve 330 VMs = 1M OPS

24. Shard Per Core Architecture Threads Shards

25. Transitional vs Sharded Kernel Cassandra TCP/IP Scheduler queue queue queue queue queue threads NIC Queues Kernel Traditional Stack Memory Lock contention Cache contention NUMA unfriendly Don’t Do It!

26. Transitional vs Sharded Kernel Cassandra TCP/IP Scheduler queue queue queue queue queue threads NIC Queues Kernel Traditional Stack SeaStar’s Sharded Stack Memory Lock contention Cache contention NUMA unfriendly Application TCP/IP Task Scheduler queue queue queue queue queue smp queue NIC Queue DPDK Kernel (isn’t involved) Userspace Application TCP/IP Task Scheduler queue queue queue queue queue smp queue NIC Queue DPDK Kernel (isn’t involved) Userspace Application TCP/IP Task Scheduler queue queue queue queue queue smp queue NIC Queue DPDK Kernel (isn’t involved) Userspace No contention Linear scaling NUMA friendly Core Database Task Scheduler queue queue queue queue queue smp queue NIC Queue Userspace

27. Traditional stack Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread is a function pointer Stack is a byte array from 64k to megabytes Context switch cost is high. Large stacks pollutes the caches A CPU Scheduler

28. Traditional stack ScyllaDB’s stack Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise is a pointer to eventually computed value Task is a pointer to a lambda function Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread is a function pointer Stack is a byte array from 64k to megabytes Context switch cost is high. Large stacks pollutes the caches No sharing, millions of parallel events A CPU Scheduler

29. An I/O Scheduler Query Commitlog Compaction Queue Queue Queue Userspace I/O Scheduler Disk Max useful disk concurrency I/O queued in FS/device No queues

30. Does it Work?

31. Top

32. Time between failures TBF Ease of maintenance No noisy neighbours No virtualization, container overhead No other moving parts Scale up before out! Small vs Large Machines

33. AWS i4i instance vertical scale 2X 2X 2X 2X 2X 2X Credit: Felipe Cardeneti Mendes

34. Linear Scale Ingestion Constant Time Ingestion 2X 2X 2X 2X 2X

35. Linear Scale Ingestion (2023 vs 2018) Constant Time Ingestion 2X 2X 2X 2X 2X

36. Ingestion: 17.9k OPS per shard * 16

37. Ingestion: <2ms P99 per shard

38. “Nodes must be small, in case they fail” You can scale OPS but can you scale failures?

39. 2X 2X 2X 2X 2X “Nodes must be small, in case they fail” No they don’t! {Replace, Add, remove} Node at constant time

40. Compaction Scale 2X 2X 2X 2X 2X

41. Compaction while Ingestion: 70MB/s per shard 120 * 70MB/s = 8.4GB/s

42. 20GB/s I4i.metal Major Compaction

43. Scale Up before Scale Out

44. You Want More? Tablets - how to distribute data among the shards dynamically

45. You Want More? Tablets - how to distribute data among the shards dynamically Per-Core Challenges - QUIC (many connections, HLB)

46. You Want More? Tablets - how to distribute data among the shards dynamically Per-Core Challenges - QUIC (many connections, HLB) IO_uring - Get rid of interrupts, SI, and IO stalls

47. You Want More? Tablets - how to distribute data among the shards dynamically Per-Core Challenges - QUIC (many connections, HLB) IO_uring - Get rid of interrupts, SI, and IO stalls Client-side shard-per-core

48. Dor Laor [email protected] DorLaor@twitter Thank you! Let’s connect.