Database Migration Strategies and Pitfalls by Patrick Bossman

Jump start your migration to ScyllaDB! In this talk, we will discuss how to migrate from Cassandra, DynamoDB, as well as other sources. Review tooling available to assist with Scylla Migrations. Review approaches and considerations for online and offline migrations, and how to plan for a faster migration if necessary.

1. A ScyllaDB Community
Database Migration
Strategies and Pitfalls
Patrick Bossman
Senior Customer Engineer

2. Patrick Bossman (he/him)
■30+ years experience helping enterprise customers
design and deploy highly performant, reliable,
available solutions
■Passion for building new things, solving problems
■2 x girl dad

3. ■Overview of types of migrations
■How to from
■Cassandra (CQL)
■DynamoDB
■Others
■Tuning migrations
Presentation Agenda

4. Migration Overview

5. We can do it! Migrations don’t have to be feared.
■We’re successfully migrating customers all the time
■Frequently with no/minimal application changes
■Rich tooling to assist with the process
■Experienced professionals to lend a hand

Demystifying migration

6. Migration options
■Online - using Scylla and DC to DC migration (way to go!)
■set replication factor (==dual writes)
■start rebuild (== historic data load)
■repair (==validation)
■Online / Live Migration (way to go)
■dual writes
■historic data fork lifting
■validation
■Offline / Downtime Migration (but why?)
■historic data fork lifting
■validation

7. Cassandra (CQL) Migration

8. ■Show example of on-line migration
■Can preserve source writetime and ttl
■Bonus - backfill will not “overwrite” new writes
■Discuss tooling
■Zero Downtime Migration (ZDM) - application transparent dual writes
■Scylla Migrator - Spark based performant backfill and validation

Cassandra migration

9. Write to Cassandra
Time
Read from Cassandra
Live Migration
9
Dual Writes (Application or ZDM)
Enable Dual
Writes
Migrate
Schema

10. Write to Cassandra
Time
Read from Cassandra
Write to ScyllaDB
Live Migration
10
Dual Writes (Application or ZDM)
Enable Dual
Writes

11. Write to Cassandra
Forklifting Existing Data
DBs in Sync
Time
Read from Cassandra
Write to ScyllaDB
Live Migration
11
Dual Writes (Application or ZDM)
Scylla
Migrator

12. Dual Reads
Write to Cassandra
Forklifting Existing Data
Validation
DBs in Sync
Time
Read from ScyllaDB
Read from Cassandra
Write to ScyllaDB
Live Migration
12
Dual Writes (Application or ZDM)

13. Dual Reads
Write to Cassandra
Forklifting Existing Data
Validation
DBs in Sync
Time
Read from ScyllaDB
Fade off
Cassandra
Read from Cassandra
Write to ScyllaDB
Live Migration
13
Dual Writes (Application or ZDM)

14. Alternative to implementing dual writes within application.
■Application connects to zdm-proxy
■Proxy dual writes to source and target
■Optionally dual read to target (validate target performance)
■zdm-proxy

SQL
NoSQL
zdm-proxy - Zero Downtime Migration
14

15. ■Highly resilient to failures
■Access compatible Databases using a native connector
■Scalable, high performance parallelized reads and writes
■Support transformations
■Open Source!

SQL
NoSQL
ScyllaDB Migrator
15

16. ■Slower - but maybe ok for smaller datasets
■DSBulk
■Bulk unload/load tool
■Can also pipe
■CQLSH COPY TO/FROM
■CQLSH copy to/from csv

Other options
16

17. DynamoDB Migration

18. ■ScyllaDB Alternator API compatible with DynamoDB
■Frequently, no application changes required
■Show example of on-line migration using Scylla migrator
■Alternatives

DynamoDB migration

19. Write to DynamoDB
Time
Read from DynamoDB
Live Migration
19
Updates from DynamoDB Stream
Migrate
Schema
Enable
Streams

20. Write to DynamoDB
Forklifting Existing Data
DBs in Sync
Time
Read from DynamoDB
Live Migration
20
Updates from DynamoDB Stream
Apply updates from DynamoDB Streams
Migrate
Schema

21. Dual Reads
Write to DynamoDB
Forklifting Existing Data
Validation
DBs in Sync
Time
Read from ScyllaDB
Fade off
DynamoDB
Read from DynamoDB
Write to ScyllaDB
Live Migration
21
Updates from DynamoDB Stream
Apply updates from DynamoDB Streams

22. ■Highly resilient to failures
■Access compatible Databases using a native connector
■High performance parallelized reads and writes
■Support transformations
■Open Source!
SQL
NoSQL
ScyllaDB Migrator
22

23. ■Existing application fed from queue (kafka)
■Load backup or scylla migrate from source
■Replay queue from offset into target database
■Scylla Migrator now supports DynamoDB S3 backup (Parquet) into
ScyllaDB


Alternatives

24. Migration Tuning

25. ■Spark and ScyllaDB are scalable!
■Scylla migrator breaks work into segments
■Allocate larger/more workers to process faster
■Consider over-provisioning spark/scylla during migration phase
■Configuration (during migration)
■DynamoDB - temporarily disable TTL during backfill
■Compactions: Increase min_threshold, set static_compaction_shares: 100

Migration tuning considerations

26. ■Test migration throughput for a few minutes, then terminate
■Observe impact on source
■Verify throughput meets migration duration requirements
■Practice validation
■Truncate target, and plan/execute real migration

Validate migration throughput

27. Other Migrations

28. ■SQL, MongoDB, …
■Data modeling and application changes likely required
■ETL style migration with application consideration required
■Follow migration dual write and backfill principles discussed earlier

Other migrations

29. Wrap it up

30. Summary
■ScyllaDB scales well horizontally and vertically

■Many migrations successful with no application changes!
■Rich tooling available to perform migration

■You can do it!
■Don’t be afraid to reach out and ask questions!

31. Stay in Touch
Patrick Bossman
[email protected]
linkedin.com/in/bossman