Weathering the storm, how to manage failure in the cloud

Cloud computing has been a game-changer for businesses striving to enhance digital experiences. However, managing services at scale with modern cloud tech isn't always a cakewalk. Failure is inevitable and the way we prepare, react and learn from it will define the success or failure of our journey.

Join Trent and Javier in this session as they share some stories of different failure modes they’ve encountered during their time at REA Group, Slack, Catch and other tech businesses. These aren't just stories – they're valuable lessons. Get insights, practical tips, and discover strategies to handle the bumps in the road when dealing with cloud tech.

1. How to navigate failure in the cloud
Trent Hornibrook
Javier Turegano

2. Hi! My name is Trent.

3. Hi! My name is Javier.

4. Everything
fails
all the time
Werner Voguels

5. At the begining...
The year is 2011 and we are learning
how to use the Cloud...

6. The zombie apocalypse
That Time I Accidentally Terminated 600 Instances

7. Property photos

8. Myfun.com
New greenfield project using the
AWS Beijing region.

9. Introduce failure to your design

10. Introduce failure to your design

11. Introduce failure to your design

12. Introduce failure to your design

13. Introduce failure to your design
HTTP 500

14. AWS Sydney outage 2016
How REA (Mostly) Survived The Stormy Apocalpse

15. Multi region Active / Active

16. Cattle and not pets
Design to be recoverable
Shift left thinking
Testing for failure
Key lessons

18. New Year - 2021

19. Provision
Service
NETWORK
COMPUTE MONITORING.
Logs
Metrics
Traces
Dashboards
Alerting
View of core services
OKOKOK

20. Provision
Service
NETWORK
COMPUTE MONITORING.
Logs
Metrics
Traces
Dashboards
Alerting
Everything is normal (APAC -> EMEA)
OKOKOK

21. PACKET LOST
Provision
Service
NETWORK
COMPUTE MONITORING
Packet lost commences - First alerts
Logs
Metrics
Traces
Dashboards
Alerting
OKOKOK

22. PACKET LOST
Provision
Service
NETWORK
COMPUTE MONITORING
Logs
Metrics
Traces
Dashboards
Alerting
We lose our visibility
OKOKOK

23. PACKET LOST
Provision
Service
NETWORK
COMPUTE MONITORING
Logs
Metrics
Traces
Dashboards
Alerting
Minor downscale (CPU)
OKOKOK

24. PACKET LOST
Provision
Service
NETWORK
COMPUTE MONITORING
Logs
Metrics
Traces
Dashboards
Alerting
Massive upscale (Apache thread utilization)
SLOWSLOWSLOW

25. PACKET LOST
Provision
Service
NETWORK
COMPUTE MONITORING
Logs
Metrics
Traces
Dashboards
Alerting
Provision Service is saturated (limits)
DOWNDOWNDOWN

26. PACKET LOST
Provision
Service
NETWORK
COMPUTE MONITORING
Logs
Metrics
Traces
Dashboards
Alerting
We fix provision service
SLOW -SLOW -SLOW -
RECOVERINGRECOVERINGRECOVERING

27. Provision
Service
COMPUTE MONITORING
Logs
Metrics
Traces
Dashboards
Alerting
PACKET LOST
NETWORK
Enough web capacity to serve Slack
OKOKOK

28. AWS SCALES TRANSIT GW.
Provision
Service
NETWORK
COMPUTE MONITORING
Logs
Metrics
Traces
Dashboards
Alerting
AWS scales transit gateway
OKOKOK

29. AWS SCALES TRANSIT GW.
Provision
Service
NETWORK
COMPUTE MONITORING
Logs
Metrics
Traces
Everything back to normal
Dashboards
Alerting
Blog post
OKOKOK

30. All Hands on Deck
Blog post

31. Slack outage
January 4th

32. Learning from
incidents
Incident reviews
Understand the timeline
Identify contributing factors
Good, bad and how we got lucky
Follow up items
Share lessons learned and key
patterns with the rest of the org

33. Learning from
incidents
Long term fixes that were put in place
Adjusted scaling policies
Fully re-designed provision service
Worked with AWS to improve
Transit GW auto-scaling
Expedited our migration to the new
network architecture

34. Learn more from other Slack incidents
Slack’s Incident on 2-22-22
A Terrible, Horrible, No-Good,
Very Bad Day at Slack

36. Chaos Engineering Tools

37. Other lessons learned

38. Monzo’s Kube Outage (2017)
Presentation

39. Canva’s Recommender system (2021)
Mayur & Thien’s
Blog post

40. amazon.com Black Friday outage
Article

41. Congestion collapse - when retries are bad
re:Invent 2023 - Surviving overloads (NET402)

43. Multi-Region is a hard problem for writes
AWS Multi-Region Fundamentals

44. Global services, regional services and zonal services
AWS Fault Isolation Boundaries

45. Maybe we just need to know our fault boundaries?
AWS Fault Isolation Boundaries

46. Original Multi-AZ architecture at Slack
Failures within one AZ could affect the
whole system.
Gray failures paper

47. Celular Architecture at Slack
AZs as cells, and cells may be drained
Blog post

48. Things break all the time
Some patterns to build more resilient
systems in the Cloud
A learning culture is key
Popular ways to meitate
Practice your incident management
processes
Share your incident lessons with the
Community
Summary
Some takeaways to
weather the storm

49. Ice breaker
Hey, what’s a
memorable incident
you’ve been part of or
read about?

50. Thank you
for listening!
Feel free to DM us your
questions any time:
@setoide
@mysqldbahelp