Surviving Majority Loss: When a Leader Fails by Konstantin Osipov

In this lightning talk I will present all common combinations of ScyllaDB deployments, single and multi-DC, and how well they play out with Raft based topology management. I'll present ScyllaDB's 2024.2 new feature, zero token nodes, and discuss how they could be used to improve resilience to failure in complex deployments. Another nice feature, which we call "Dynamic voter selection" which is currently available in scylladb.git and our nightly builds helps reduce the load of Raft on large clusters and dynamically adjust Raft to node starts and stops, always maintaining maximal Raft availability.

1. A ScyllaDB Community
Surviving Majority Loss:
When a Leader Fails
Konstantin Osipov
Director of Engineering

2. Konstantin Osipov
■ScyllaDB geek since 2019
■All things consistency
■Aquarist and a father of three

3. Previous Episodes

4. ■ScyllaDB consistent cluster management recap
■Choices for production cluster topology
■… and disaster recovery consequences of those
Presentation Agenda

5. Strong consistency recap

6. Data vs metadata

-metadata


-data
Schema information: table,
view, type definitions
Topology information:
nodes, tokens
Static and regular rows,
counters
Replicated everywhere Partitioned
Not commutative Commutative
Changes rarely Changes frequently
Consistency of Metadata
1
2 3
3
1 2
replication_factor=2
ScyllaDB cluster

7. ■Runs alongside Raft leader
■Highly available
■Drives the progress of topology changes
■Performs linearizable reads and writes of the topology
■Request coordinators still use the local view on topology
■No extra coordination when executing user requests

The Centralized Topology Coordinator

8. All the fun ways to deploy
… and what happens next

9. Let’s start with a 4 node cluster

DC=my_dc

10. Let’s start with a 4 node cluster

DC=my_dc
replication_factor=3 raft_quorum = 3 data_quorum = 2

11. Reaction to a node failure

DC=my_dc
replication_factor=3 raft_quorum = 3 data_quorum = 2

12. Lost majority pre 2025.2

DC=my_dc
replication_factor=3 raft_quorum = 3 data_quorum = 2

13. Lost majority 2025.2

DC=my_dc
replication_factor=3 raft_quorum = 2 data_quorum = 2

14. Two data center deployment - pre 2025.2

DC1=us_east
replication_factor=2 raft_quorum = 3 data_quorum = 4
DC2=eu_london

15. Two data center deployment - 2025.2

DC1=us_east
replication_factor=2 raft_quorum = 2 data_quorum = 4
DC2=eu_london

16. Split brain - 2025.2

replication_factor=2 raft_quorum = 2 data_quorum = 4
DC1=us_east DC2=eu_london

17. Let’s talk about racks

replication_factor=2 raft_quorum = 3 data_quorum = 3
rack 1
rack 2
DC1=us_east DC2=eu_london
rack 1
rack 2

18. Having enough nodes in each rack

replication_factor=2 raft_quorum = 5 data_quorum = 3
rack 1
rack 2
DC1=us_east DC2=eu_london
rack 1
rack 2

19. Limited voter count

replication_factor=1 raft_quorum = 2 data_quorum = 2

20. Limited voter count

replication_factor=1 raft_quorum = 3 data_quorum = 2

21. Zero token nodes

DC1=us_east
replication_factor=2 raft_quorum = 2 data_quorum = 3
DC2=eu_london
DC3=arbiter
scylla.yaml:
join_ring: false

22. Zero token preference

replication_factor=2 raft_quorum = 2 data_quorum = 3
scylla.yaml:
join_ring: false

23. Disaster recovery after a majority loss - 2025.2

scylla.yaml:
recovery_leader: <uuid>

24. 1.Identify alive nodes
2.Choose recovery leader
3.Erase group0 state from system tables
4.Restart the nodes with recovery_leader option
The new recovery procedure - 2025.2

25. ●Adaptive voter selection algorithm makes ScyllaDB clusters even more
resilient to failures

Recap

26. ●Adaptive voter selection algorithm makes ScyllaDB clusters even more
resilient to failures
●Zero token nodes allow improving resilience with low cost

Recap

27. ●Adaptive voter selection algorithm makes ScyllaDB clusters even more
resilient to failures
●Zero token nodes allow improving resilience with low cost
●A new disaster recovery procedure allows to form a new majority
automatically and supports tablets

Recap

28. ●Certain roll-out scenarios, such as:
○Even number of nodes
○Two data centers
○Low number of racks
.. are more error-prone than others and should be avoided.
Recap

29. Stay in Touch
Kostja Osipov
[email protected]