Using ScyllaDB for Real-Time Write-Heavy Workloads

Learn about major architectural shifts that enable new levels of elasticity and operational simplicity
ScyllaDB just launched the first release featuring our new “tablets” architecture. Tablets builds upon a multiyear project to re-architect our legacy ring architecture. Our metadata is now fully consistent, thanks to the assistance of the Raft consensus protocol. Together, these changes enable new levels of elasticity, speed, simplicity, and efficiency. Data is dynamically redistributed as the workload and topology evolve. New nodes can be spun up in parallel and start adapting to the load in near real-time.

Join ScyllaDB Co-founder Dor Laor to learn what this new approach means for:

- Rapidly responding to traffic spikes without overprovisioning
- Performing safe, concurrent, fast bootstrapping
- Simplifying cluster administration (e.g., cleanup, repair, tombstone garbage collection)
- Increasing efficiency and eliminating overprovisioning

1. ScyllaDB Fast Forward:
True Elastic Scale
Dor Laor, CEO and Co-Founder, ScyllaDB
Felipe Cardeneti Mendes, Solutions Architect++, ScyllaDB

2. Poll
How often do you scale your database?

3. Presenters

Felipe Cardeneti Mendes, Solutions Architect
+Puppy Lover
+Open Source Enthusiast
+ScyllaDB passionate!

Dor Laor, CEO and Co-Founder
+Puppy Lover
+Open Source, snowboard and MTB Enthusiast
+ScyllaDB passionate!

4. Why ScyllaDB?
Best High Availability in the industry
Best Disaster Recovery in the industry
Best scalability in the industry
Best Price/Performance in the industry Auto-tune - out of the box performance
Compatible with Cassandra & DynamoDB
The power of Cassandra at the speed of Redis with the usability of DynamoDB
No Lock-in
Open Source Software

5. +400 Gamechangers Leverage ScyllaDB

6. Agenda
+Wasn’t ScyllaDB elastic before?
+The value proposition
+Demo time
+Deeper dive
+What's next

7. < 6.0 ScyllaDB Releases Were Good
2X 2X 2X 2X 2X
3 x 4-vcpu VMs
1B keys
3 x 8-vcpu VMs
2B keys
3 x 16-vcpu VMs
4B keys
3 x 32-vcpu VMs
8B keys
3 x 64-vcpu VMs
16B keys
3 x 128-vcpu VMs
32B keys

8. < 6.0 Latency Under Repair/Add/Remove Nodes

9. ScyllaDB < 6.0 Problems
+Eventual Consistency of cluster metadata
+Great architecture for scalable, partition-tolerable, widely distributed DB
+But
+Topology changes are allowed one-at-a-time
+Rely on 30+ second timeouts for poor’s man linearizability
+Node failed/down block scale
+Streaming time is a function of the schema - parsing partitions
+Additional complex operations: Cleanup, rebuild, etc

10. Our journey
from eventual
to immediate
consistency
& tablets

11. ScyllaDB 6.0 Value
Elasticity
+Faster bootstrap/operations
+Incremental bootstrap
+Concurrent node operations


Simplicity
+Decouple topology operations
+Transparent cleanups
+Semi transparent repairs
+Parallel maintenance operations

Speed
+Streaming sstables - 30x faster
+Load balancing of tablets Consistency
+Small tables fit in few (single) nodes
TCO
+Shrink free space
+Reduce static, over provisioned deployments

12. Demo Time!

13. Theory Behind Tablets

14. Raft- Consistent Metadata
Protocol for state machine replication
Total order broadcast of state change commands
X = 0 X += 1
CAS(X, 0, 1)
X = 0

15. node A
bootstrap
bootstrap
Linearizable Token Metadata
node B
node C
system.token_metadata

Read
barrier
Read
barrier

16. +Fencing - each write is signed with topology version
+If there is a version mismatch, the write doesn’t go through
Changes in the Data Plane
Replica
Coordinator
Topology
coordinator
Req(V
1)
Req(V
1
)
Fence(V
1)
Drain(V
1)
Req(V
2
)

17. Linearizable Schema Version
No re-hash of the entire schema on change
10x less CPU with large schemas.
TimeUUID-based Schema versionHash-based schema version
5.x: 6.x:

18. Tablet History: ClayDB > Bigtable > ScyllaDB

19. Consistent Metadata & Tablet Journey
RAFT Safe schema
changes
Safe topology
changes
Dynamic partitioning
Consistent tables
Tablets
5.0
5.2
5.2+
6.0

20. +Introduce a new layer of indirection - the tablets table
+Each table has its own token range to node mapping
+Mapping can change independently of node addition
and removal
+Different tables can have different tablet counts
+Managed by Raft
Implementation - Metadata
System, tablets
Query
Replica
Set
Token

21. +Each tablet replica is isolated into its own
memtable+ SSTables
+Forms its own little Log-Structured Merge Tree
+With compaction and stuff
+Can be migrated as a unit
+Migration: copy the unit
+Cleanup: delete the unit
+Split/merge as the table grows/shrinks
Implementation - Data Path

22. Standard Tables
{R1, R2, R3}
R1
R2
key1key2
Sharding function generates
good load distribution between
CPUs

R3

23. RAFT group
No. 299238
RAFT group
No. 299236 RAFT group
No. 299237
Raft Tables key1key2
tablet
tablet
replica
tablet
replica

24. +Source of truth for the cluster
+How many tablets for each table
+Token boundaries for each tablets
+On which nodes and shards do we have replicas
+What kind of transition the tablet is undergoing
+Which nodes and shards will host tablet replicas
after the transition
+Managed using the Raft protocol
+Replicated on every node
The Tablets Table
CREATE TABLE system.tablets (
table_id uuid,
last_token bigint,
keyspace_name text static,
replicas list<tuple<uuid, int>>,
new_replicas list<tuple<uuid, int>>,
session uuid,
stage text,
transition text,
table_name text static,
tablet_count int static,
PRIMARY KEY (table_id, last_token)
);

25. Tablets - Balancing
Depends on fault-tolerant,
reliable, and fast topology
changes.

26. +Hosted on one node
+But can be migrated freely if the node is down
+Synchronized via Raft
+Collects statistics on tables and tablets
+Migrates to balance space
+Evacuates nodes to decommission
+Migrates to balance CPU load
+Rebuilds and repairs
Implementation - Load Balancer

27. Tablet Scheduler
Scheduler globally controls movement,
maintenance operation on a per tablet basis
repairmigrationtablet 0
tablet 1
schedule schedule
repairBackup

28. Tablets
Resharding is cheap.
SStables split at tablet boundary.
Reassign tablets to shards (logical operation).

29. Tablets
Cleanup after topology change is cheap.
Just delete SSTables. Runs automatically,
under the hood

30. Tablets - Streaming (Enterprise)
Send files over RPC. No per
schema, per row processing.
Up to 30x faster, saturate links.

Sstables
files
Sstables
files
Sstables
files

31. +Drivers are tablet-aware
+But without reading the tablets table
+Driver contacts a random node/shard
+On miss, gets updated routing information for
that tablet
+Ensures fast start-up even with 100,000 entries
in the tablets table
Driver Support

32. Authentication and Service Levels on Raft
ScyllaDB 5.x was Manual
ScyllaDB 6.x:
+Automatically replicated on every node
+Linearizable with CREATE/DROP
+No denial of service if a node is down

33. What’s Coming

34. Tablets - Repair - Fire & Forget
+Continuous, transparent controlled by the load balancer
+Auto GC grace period

35. Serverless (VM Based..)
Typeless Sizeless Limitless

36. Consistent
metadata +
Elasticity =

Much More

37. Poll
How long does it take for you to scale
your existing database?

38. Guilherme Nogueira
[email protected]
Keep Learning
scylladb.com/category/engineering
Register now at p99conf.io
Visit our
blog for more
on ScyllaDB
engineering

39. Thank you
for joining us today.
@scylladb scylladb/
slack.scylladb.com@scylladb company/scylladb/
scylladb/

40. 40
New metrics in keyspace dashboard
●scylla-monitoring provides an updated dashboard that shows tablets over time per instance
and tablet count per instance at the moment

41. 41
Write workload

42. Tablet
Tablet
Tablet
Tablet
Growing a cluster - Tablets
Tablet
Tablet
Tablet
Tablet
Serving
reads/writes

43. Tablet
Tablet
Tablet
Tablet
Growing a cluster - Tablets
Tablet
Tablet
Tablet
Tablet

44. Tablet
Tablet
Tablet
Tablet
Growing a cluster - Tablets
Tablet
Tablet
Tablet
Tablet
Start Serving
reads/writes