Spotify: Automating Cassandra repairs

Anti-entropy repairs are known to be a very peculiar maintenance operation of Cassandra clusters. They are problematic mostly because of the potential of having negative impact on the cluster's performance. Another problematic aspect is the difficulty of managing the repairs of Cassandra clusters in a careful way that would prevent the negative performance impact.

Based on the long-term pain we have been experiencing with managing repairs of nearly 100 Cassandra clusters, and being unable to find a solution that would meet our needs, we went ahead and developed an open-source tool, named Cassandra Reaper [1], for easy management of Cassandra repairs.

Cassandra Reaper is a tool that automates the management of anti-entropy repairs of Cassandra clusters in a rather smart, efficient and careful manner while requiring minimal Cassandra expertise.

I will have to cover some basics of eventual consistency mechanisms of Cassandra, after which I will be able to focus on the features of Cassandra Reaper and our six months of experience having the tool managing the repairs of our production clusters.

1. Automating Cassandra
Repairs

Radovan Zvoncek
[email protected]

github.com/spotify/cassandra-reaper
#CassandraSummit

2. About zvo

3. About zvo
Likes pancakes

4. About zvo
Likes pancakes

Does this for the 3rd time

5. About zvo
Likes pancakes

Does this for the 3rd time

Works at Spotify

6. Working at Spotify
Is autonomous

Squads responsible for their full stack

Including Cassandra

7. Cassandra

8. Node’s data



Cassandra

9. Replication
Cassandra

10. Running Cassandra
Requires many things

One of them is keeping data consistent

Otherwise it can get lost or reappear

11. Running Cassandra
Requires many things

One of them is keeping data consistent

Eventually

12. Eventual consistency





Cassandra

13. Eventual consistency

Read Repairs





Cassandra
R
W

14. Eventual consistency

Hinted Handoff





Cassandra

15. Eventual consistency

Anti-entropy Repair





Cassandra

16. Coordinated process








Anti-entropy Repair

17. Coordinated process
Four steps:







Anti-entropy Repair

18. Coordinated process
Four steps:
1: Hash







Anti-entropy Repair
#
#
#

19. Coordinated process
Four steps:
1: Hash
2: Compare







Anti-entropy Repair
###

20. Coordinated process
Four steps:
1: Hash
2: Compare
3: Stream







Anti-entropy Repair

21. Coordinated process
Four steps:
1: Hash
2: Compare
3: Stream
4: Merge







Anti-entropy Repair

22. Coordinated process
Four steps:
1: Hash
2: Compare
3: Stream
4: Merge
Can go wild...






Anti-entropy Repair

23. Repair gone wild

24. Repair gone wild
Eats a lot of disk IO
●because of hashing all the data

25. Repair gone wild
Eats a lot of disk IO
Saturates the network
●because of streaming a lot of data around

26. Repair gone wild
Eats a lot of disk IO
Saturates the network
Fills up the disk
●because of receiving all replicas, possibly
from all other data centers

27. Repair gone wild
Eats a lot of disk IO
Saturates the network
Fills up the disk
Causes a ton of compactions
●because of having to merge the received
data

28. Repair gone wild
Eats a lot of disk IO
Saturates the network
Fills up the disk
Causes a ton of compactions

… one better be careful

29. Careful repair

30. nodetool repair


Careful repair
All three intervals

31. Partitioner range
●nodetool repair -pr


Careful repair
This interval only

32. Start & end tokens
●nodetool repair -pr -st -et


Careful repair
A part of interval only

33. Requires splitting the ring into smaller intervals

Smaller intervals mean less data

Less data means fewer repairs gone wild

Careful repair

34. Smaller intervals also mean more intervals

More intervals mean more actual repairs

Repairs need to be babysat :(


Careful repair

35. The Spotify way
Feature teams meant to do features
Not waste time operating their C* clusters

Cron-ing nodetool repair is no good
●mostly due to no feedback loop

This all led to creation of the Reaper

36. The Reaper
REST(ish) service

Does a lot of JMX

Orchestrates repairs for you

37. The reaping
You:
curl http://reaper/cluster --data ‘{“seedHost” : “my.cassandra.host.net”}’

The Reaper:
●Figures out cluster info (e.g. name, partitioner)

38. The reaping
You:
curl http://reaper/repair_run --data ‘{“clusterName”: “myCluster”}’

The Reaper:
●Prepares repair intervals

39. The reaping
You:
curl -X PUT http://reaper/repair_run/42 -d state=RUNNING

The Reaper:
●Starts triggering repairs of repair intervals

40. Reaper’s features

41. Reaper’s features
Carefulness - doesn’t kill a node
●checks for node load
●backs off after repairing an interval

42. Reaper’s features
Carefulness - doesn’t kill a node
Resilience - retries when things break
●because things break all the time

43. Reaper’s features
Carefulness - doesn’t kill a node
Resilience - retries when things break
Parallelism - no idle nodes
●multiple small intervals in parallel

44. Reaper’s features
Carefulness - doesn’t kill a node
Resilience - retries when things break
Parallelism - no idle nodes
Scheduling - setup things only once
●regular full-ring repairs

45. Reaper’s features
Carefulness - doesn’t kill a node
Resilience - retries when things break
Parallelism - no idle nodes
Scheduling - setup things only once
Persistency - state saved somewhere
●a bit of extra resilience

46. What we reaped
First repair done 2015-01-28
1,700 repairs since then, recently 90 per week
176,000 (16%) segments failed at least once
60 repair failures

47. What we reaped

48. What we reaped

49. Reaper’s Future
CASSANDRA-10070

Whatever is needed until then

50. Greatest benefit
Cassandra Reaper automates a very tedious
maintenance operation of Cassandra clusters
in a rather smart, efficient and careful manner
while requiring minimal Cassandra expertise

github.com/spotify/cassandra-reaper

#CassandraSummit

51. Thank you!


github.com/spotify/cassandra-reaper

#CassandraSummit