Zero Downtime Critical Traffic Migration @Netflix Scale

Picture yourself enthralled by the latest episode of your beloved Netflix series, delighting in an uninterrupted, high-definition streaming experience. Behind these perfect moments of entertainment is a complex mechanism, with numerous gears and cogs working in harmony. But what happens when this machinery needs a transformation? This is where large-scale system migrations come into play.

Come join us to learn about how Netflix does these migrations at scale with replay traffic testing, canary analysis, dual writes/reads, and with NO downtime.

1. Zero Downtime Critical Traffic Migration @Netflix Scale Abhishek Pandey Tech Lead at Meta + Ex Senior Engineer at Netflix

2. Abhishek Pandey ( he/him/his ) Tech Lead at Meta Migrated and modernized bunch of critical Netflix components. Explore nature with my wife and newborn. Travel and Play Tennis 2 truths and a lie: Caused global outage at Uber, Met Roger Federer, Got fired once.

3. Introduction Picture yourself enthralled by the latest episode of your beloved Netflix series, delighting in an uninterrupted, high-definition streaming experience. Behind these perfect moments of entertainment is a complex mechanism, with numerous gears and cogs working in harmony. Large-scale system migrations are necessary when this machinery needs transformation.

4. Introduction Challenges in Transitioning Traffic Netflix's Challenge: Uninterrupted Streaming Backend Systems: Orchestrating Product Experience Evolution and Optimization of Backend Systems Focus of this talk: Migration Strategies

5. Challenges of System Migrations Main Challenge: Transitioning Traffic with No Customer Impact Ensuring confidence in upgraded architecture. Strategies to meet Quality-of-Experience metrics Architecture of Backend Systems Distributed microservices architecture. Migration points across the service call graph. Stateless and stateful APIs involved.

6. Migration Phases Phase 1: Validating functional correctness and performance. Functional Correctness, Scalability, Performance, Resilience Monitoring QoE, SLAs, and KPIs Phase 2: Controlled migration with continuous monitoring. Minimizing Incident Risks Continuous Metrics Monitoring

7. Replay Traffic Testing What is replay traffic testing? Benefits of using replay traffic. Sandboxed testing at scale. Exercise diversity of inputs. Functional correctness, performance validation and load testing.

8. Replay Traffic Testing Components Component 1: Traffic Duplication and Correlation Clone and Fork Production Traffic Record and Correlate Responses Component 2: Comparative Analysis and Reporting Compare and Analyze Responses Generate Comprehensive Reports

9. Approaches for Replay Traffic Generation Device Driven Approach

10. Approaches for Replay Traffic Generation Server Driven Approach

11. Approaches for Replay Traffic Generation Dedicated Service Approach

12. Analyzing Replay Traffic Normalization Preprocessing Responses Addressing Alterations Comparison Diffing Responses Generating Summary Metrics

13. Comparing Live Traffic Replay Traffic Analysis

14. Load Testing with Replay Traffic Stress Testing with Replay Traffic Regulating Load Evaluating Performance Metrics

15. Stateful Systems and Replay Testing Application to Stateful Systems Isolated Data Stores Recording State and Response

16. Case Study: Netflix's Use of Replay Testing Netflix’s Migration Projects Benefits of Replay Testing Building Confidence through Testing

17. Move to Production

18. Canary Deployments Canary Concept Baseline vs. Canary Clusters Monitoring Performance Metrics

19. Sticky Canaries Enhancing Traditional Canary Process The Canary Pool Monitoring Broader System Metrics

20. A/B Testing Verifying Hypotheses Controlled Experiments Divide The Population

21. Dialing Traffic Traffic Control Dial Function Monitoring and Rollback

22. Migrating Persistent Stores Challenges with Stateful APIs ETL-Based Dual-Write Strategy Dual Reads

23. Clean Up Cleanup and Optimization Removing Migration-Related Code Documentation for Future Migrations

24. Conclusion Utilized diverse techniques for various migrations. Achieved success with minimal downtime. Gained valuable insights and refined methods. Customized strategies for unique migration scenarios. Goal: Seamless migrations without disruptions.

25. Abhishek Pandey pandeyabhi1987 @gmail.com pandeyabhi1987 @beinrhythm_abhi Thank you! Let’s connect.