Webinar: Choosing the Right Shard Key for High Performance and Scale

Read these webinar slides to learn how selecting the right shard key can future proof your application.

The shard key that you select can impact the performance, capability, and functionality of your database.

1. Ger Hartnett Director of Technical Services (EMEA), MongoDB @ghartnett #MongoDB Tales from the Field Part three: Choosing the Right Shard Key for High-Performance and Scale

2. Or: Cautionary Tales Don’t solve the wrong problems Bad schemas & shard keys hurt ops too

3. The main talk should take 30-35 minutes You can submit questions via the chat box We’ll answer as many as possible at the end We are recording and will send slides Friday This is the final webinar in a series of 3 Before we start

4. You work in operations You work in development You have a MongoDB system in production You have contacted MongoDB Technical Services ( support ) You attended an earlier webinar in the series ( part1, part2 ) A quick poll - add a word to the chat to let me know your perspective

5. We collect - observations about common mistakes - to share the experience of many Names have been changed to protect the (mostly) innocent No animals were harmed during the making of this presentation (but maybe some DBAs and engineers had light emotional scarring) While you might be new to MongoDB we have deep experience that you can leverage Stories

6. Discovering a DR flaw during a data centre outage Complex documents, memory and an upgrade “surprise” Wild success “uncovers” the wrong shard key The Stories (part three today)

7. Story #1: Quick Review

8. Story #1: Recovering from a disaster Prospect in the process of signing up for a subscription Called us late on Friday, data centre power outage and 30+ (11 shards) servers down When they started bringing up the first shard, the nodes crashed with data corruption 17TB of data, very little free disk space, JOURNALLING DISABLED!

9. Recovering each shard Start secondary read only Mount NFS storage for repair Repair former primary node Iterative rsync to seed a secondary Secondary Primary Secondary

10. Key takeaways for you If you are departing significantly from standard config, check with us (i.e. if you think journalling is a bad idea) Two DC in different buildings on different flood plains, not in the path of the same storm (i.e. secondaries in AWS) DR/backups are useless if you haven’t tested them

11. Story #2: Complex documents, memory and an upgrade “surprise” Well established ecommerce site selling diverse goods in 20+ countries After switching to wired tiger in production, performance dropped, this is the opposite of what they were expecting

12. { _id: 375 en_US : { name : ..., description : ..., <etc... > }, en_GB : { name : ..., description : ..., < etc... > }, fr_FR : { name : ..., description : ..., < etc... > }, de_DE : ..., de_CH : ..., <... and so on for other locales... > inventory: 423 } Product Catalog: Original Schema

13. Key Takeaways When doing a major version/storage-engine upgrade, test in staging with some proportion of production data/workload Sometimes putting everything into one document is counter productive

14. Story #3: Wild success uncovers the wrong shard key Started out as error “[Balancer] caught exception … tag ranges not valid for: db.coll” 11 shards, they had added 2 new shards to keep up traffic - 400+ databases Lots of code changes ahead of the Superbowl Spotted slow 300+s queries, decided to build some indexes without telling us Went production down

15. Adding Shards 2 More Shards….

16. The “Golden Hammer” Tendency

17. Diagnosing the issues #1 The red-herring hunt begins Transparent Huge Pages enabled - production Chaotic call - 20 people talking at once, then in the middle of the call everything started working again Barrage of tickets and calls Connection storms

18. Using mtools to analyse logs - conn churn

19. Diagnosing the issues #2 Got inconsistent and missing log files Discovered repeated scatter-gather queries returning the same results Secondary reads Heavy load on some shards and low disk space

20. Insert load on two shards (from Cloud Manager)

21. Diagnosing the issues #3 Shard key - string with year/month & customer id { _id : ObjectId("4c4ba5e5e8aabf3"), count : 1025, changes: { … } modified : { date : "201 5_02 ", customer Id: 314159 } }

22. Diagnosing the issues #4 First heard about DDOS attack Missing tag ranges on some collections Stopped the balancer which reduced system load from chunk moves Two clusters had a mongos each on the same server

23. Fixing the issues Script to fix the tag ranges Proposed finer granularity shard key - but this was not possible because of 30TB of data Moved mongos to dedicated servers Re-enable the balancer for short windows with waitForDelete and secondaryThrottle Put together scripts to pre-split and move empty chunks to quiet shards based on traffic from month before

24. Monthly pre-split and move chunks { date : "2015_03", customerId: min-500 customerId: 314159 customerId: 501-10000 customerId: 10001-300k customerId: 300k-314158 customerId: 314160-max

25. The diagnosis in retrospect The outage did not appear to have been related to either the invalid tag ranges or the earlier failed moves The step downs did not help resolve the outage but did highlight some queries that need to be fixed The DDoS was the ultimate cause of the outage - lead to diagnosis of deeper issues The deepest issue was the shard key

26. Aftermath and lessons learned Signed up for a Named TSE Now doing pre-split and move before the end of every month Check before making other changes (i.e. building new indexes)

27. Key takeaways for you Choosing a shard key is a pivotal decision - make it carefully Understand current bottleneck Monitor insert distribution and chunk ranges Look for slow queries (logs & mtools) Run mongos, mongod, config server on dedicated server or use containers/cgroups

28. Further Reading Production notes docs.mongodb.org/manual/administration/production-notes Mtools github.com/rueckstiess/mtools Previous Webinars mongodb.com/presentations

29. Ger Hartnett Director Technical Services (EMEA), MongoDB @ghartnett #MongoDB Questions?

30. You can submit questions via the chat box We are recording and will send slides Friday Questions

31. Code GerHartnett gets 25% discount