Test Smarter Not Harder - how to design tests for continuous delivery

Imagine: as soon as any developed functionality is submitted into the code repository, it is automatically subjected to the appropriate battery of tests and then released straight into the wild. Setting up the pipeline to do just that has become commonplace, but most organizations hit the same stumbling block: just what IS the appropriate battery of tests? Automated build pipelines don't always lend themselves well to the traditional stages of testing. In this hands-on tutorial, Melissa will introduce testers to the key principles of test case and test suite design that apply to organizations big and small to allow them to take full advantage of the pipeline's capabilities without introducing unnecessary bottlenecks. Testers will learn how to make highly reliable tests that run fast and preserve just enough information to let the team determine exactly what went wrong and how to reproduce the error locally. They will also explore how to reduce overlap while still maintaining adequate test coverage, about what test areas might be most beneficial to combine into a single suite, and what areas might benefit most from being completely broken out.

1. Test smarter, not harder! how to design test suites for continuous delivery STAREAST 2024 Melissa Benua / Head of Platform Engineering, mParticle

2. About Me Melissa Benua Sr Director of Engineering, Platform mParticle Get in touch! [email protected] @ queenofcode Follow my work! https://www.linkedin.com/in/mbenua/ https://www.slideshare.net/MelissaBenua/ https://github.com/queen-of-code/ https://www.queenofcode.net @ queenofcode

3. Imagine: as soon as any developed functionality is submitted into the code repository, it is automatically subjected to the appropriate battery of tests and then released straight into the wild. Setting up the pipeline to do just that has become commonplace, but most organizations hit the same stumbling block: just what IS the appropriate battery of tests? Automated build pipelines don't always lend themselves well to the traditional stages of testing. In this hands-on tutorial, Melissa will introduce testers to the key principles of test case and test suite design that apply to organizations big and small to allow them to take full advantage of the pipeline's capabilities without introducing unnecessary bottlenecks. Testers will learn how to make highly reliable tests that run fast and preserve just enough information to let the team determine exactly what went wrong and how to reproduce the error locally. They will also explore how to reduce overlap while still maintaining adequate test coverage, about what test areas might be most beneficial to combine into a single suite, and what areas might benefit most from being completely broken out. @queenofcode

4. Navigation @ queenofcode

5. continuous delivery what is it and why does it matter? @queenofcode

6. @ queenofcode PRE-MERGE POST-MERGE Code & Test Deploy & Test Deploy & Test Peer Review Monitor & Alert

7. @ queenofcode PRE-MERGE POST-MERGE Code & Test Deploy & Test Deploy & Test Peer Review Monitor & Alert Local Dev Env / Ephemeral QA Env Pre-Production / Production

8. @queenofcode

9. @ queenofcode PRE-MERGE POST-MERGE Code & Test Deploy & Test Deploy & Test Peer Review Monitor & Alert Local Dev Env / Ephemeral QA Env Pre-Production / Production

10. Introducing: MyPhotoApp Gallery Site! @queenofcode

11. Check Categorization what kind of tests do we have here?

12. @queenofcode Functional Tests Non-Functional Tests Automated Tests Manual Tests

13. Definitions @queenofcode

14. Automated vs Manual Tests @queenofcode

15. 01 Unit Tests 02 Automated Tests 03 Manual Tests Functional Test Case Pyramid @queenofcode

16. Unit Test Guidelines @queenofcode Does not cross application boundaries Should support parallelism Textual `Input -> Output` validation Does NOT leave the box

17. Beyond Unit Tests @queenofcode Build Compile change against main Execute unit tests Deploy Deploy change to testing environment Execute integration tests Validate Execute UI + load tests Manual signoff Monitoring signoff Project structure? Moving parts? Functional boundaries? Shared resources? Mocking potential?

18. Automated Test Guidelines (Integration) @queenofcode Crosses application boundaries Does not require manual intervention Simple `Input -> Output` validation Follows user application flow

19. Semi-Automated to Manual Test Guidelines @queenofcode Crosses many boundaries Requires manual intervention or setup Difficult validation (not pass/fail) Validates complex scenarios

20. More Definitions @queenofcode

21. Exercise: Test Cases @queenofcode How should we test our photo gallery? All ideas are good ideas!

22. Smart Test Case Principles what makes a test case ‘smart’?

23. Test Case principles @queenofcode

24. Speed @queenofcode

25. Speed @queenofcode Build Failure? UI Test Failure?

26. Reliability @queenofcode

27. Reliability @queenofcode

28. Specificity @queenofcode

29. Specificity @queenofcode

30. Proceed With Caution @queenofcode

31. Exercise: Smart Test Cases @queenofcode Out of all our test ideas, which ones aren’t specific, speedy, or repeatable? Are there some we just shouldn’t do?

32. Monitoring and reporting what just happened?

33. Code Health @queenofcode

34. If your site crashes on the internet but isn’t monitored, is it really down??? @queenofcode

35. Pillars of Observability @queenofcode

36. Logging vs Metrics @queenofcode Text Log File? Graph?

37. Metric Types @queenofcode Rate Success Requests Per Second Failure Requests Per Second Events Per Second Histogram Request Latency Database Latency 60 75 50 90 Gauge Max items in queue Bytes in use

38. Using Metrics @queenofcode

39. Metrics and Alerts @queenofcode

40. Metrics and Automation @queenofcode void Call(ref int A, ref int B) { A = DoWork (); B = DoOtherWork (); CallsCounter.Increment (); }

41. Exercise: Monitoring @queenofcode How should we monitor our photo gallery? What should we focus on to backstop our testing efforts?

42. Smart Test Pipeline design what kind of tests are successful in cd?

43. Example Service Architecture @queenofcode

44. Test Flow @queenofcode Build Change 01 Compile Code Run Unit Tests Create Code Package 02 Deploy Package 03 Validate Product 04 Package code Create Docker image Deploy app / start Docker container Run Auto Tests Run Manual Tests Validate monitors

45. @queenofcode

46. Example suite breakdown @queenofcode

47. Example Suite Breakdown @queenofcode

48. Pair code changes to appropriate tests! @queenofcode

49. Integration Test Matrix – Login SCENARIO @queenofcode

50. Integration Test Matrix – API Scenario @queenofcode

51. Tips and Tricks @queenofcode

52. Exercise: Mapping Cases to Categorized Suites @queenofcode What should we run? When should we run it? How long should we wait?