The Future of Repair: Transparent and Incremental by Botond Dénes

Regularly run repairs are essential to keep clusters healthy, yet having a good repair schedule is more challenging than it should be. Repairs often take a long time, preventing running them often. This has an impact on data consistency and also limits the usefulness of the new repair based tombstone garbage collection. We want to address these challenges by making repairs incremental and allowing for automatic repair scheduling, without relying on external tools.

1. A ScyllaDB Community
The Future of Repair:
Transparent and Incremental
Botond Denes
Software Team Lead

2. ■Consistent data is important
■Consistent tombstones is even more important
■tombstone_gc = {'mode':'timeout'} - at least one repair per
GC_GRACE_SECONDS
■tombstone_gc = {'mode':'repair'} - only tombstones written before last repair
can be garbage-collected

Repair is important

3. ■Slow
■Large dataset
■Mixed-shard cluster
■Need external tools to schedule
■nodetool - manual
■scylla-manager - needs provisioning

Repair is challenging

4. Incremental Repair
Not every repair needs to repair all the data!
■Repair only data written since last repair
■The more frequent repairs are – the less work they have to do
■Repairing often is no longer a problem

5. Move scheduling into ScyllaDB core!
■Operational simplicity - no need for external tools
■External tools can only observe past/current state
■Internal scheduling is also aware of plans

Automatic repair

6. ■Tablet only
■Tablet repair is simpler - 1 peer shard on each peer node
■Tablet repair happens via the tablet scheduler
■User requests tablet(s) to be repaired via /storage_service/tablets/repair
■Request saved in system.tablets
■Repair of individual tablets is scheduled
■Excludes with ongoing migrations
■Consider load on individual nodes and shards

The way to incremental and automatic repair

7. Stay in Touch
Botond Denes
[email protected]
denesb