Performance Regression Analysis: Accomplishments and Challenges

My invited talk at the The Fifth International Workshop on Large-Scale Testing (LT 2016, co-located with ICPE).

1. Performance Regression AnalysisPerformance Regression Analysis
Cor-Paul Bezemer
Title:QueensLogo_colour.eps
Creator:Adobe Illustrator(R) 12
CreationDate:9/10/08
LanguageLevel:2
Accomplishments and ChallengesAccomplishments and Challenges

2. ““A burnt child dreads fire”A burnt child dreads fire”

3. Regression testing makes sure we do Regression testing makes sure we do
not make the same mistake twicenot make the same mistake twice
Write code
Execute
regression
test
Deploy or fix changes

4. Regression testing also helps us to make Regression testing also helps us to make
sure we are not re-introducing more bugs sure we are not re-introducing more bugs

5. Performance regression testing helps Performance regression testing helps
ensure we are not breaking performanceensure we are not breaking performance
Write code
Execute
performance
test
Deploy if performance is OK or fix changes

6. Performance regression testing Performance regression testing
can/should be done on several metricscan/should be done on several metrics

7. Performance regression testing Performance regression testing
can/should be done on several metricscan/should be done on several metrics

8. Performance regression testing Performance regression testing
can/should be done on several metricscan/should be done on several metrics

9. Performance regression testing Performance regression testing
can/should be done on several metricscan/should be done on several metrics

10. Directions in research on Directions in research on
performance regression analysisperformance regression analysis

11. Directions in performance Directions in performance
regression analysis researchregression analysis research
1. Execution profile comparison
2. Source code analysis
3. Visualization

12. Execution profile comparisonExecution profile comparison

13. Execution profile comparisonExecution profile comparison

14. Disadvantages of execution Disadvantages of execution
profile comparisonprofile comparison
- Actually needs performance tests
- Need to run performance tests
several times because of variation
- Small changes may be difficult to
interpret

15. Source code analysisSource code analysis
Identify risky commits and recommend
those for testing (Huang et al. @ ICSE 2014)

16. Disadvantages of source Disadvantages of source
code analysiscode analysis
- Can not replace 'real' performance
measurements (yet)

17. Visualization of Visualization of
performance regression performance regression
analysis resultsanalysis results

18. Control charts Control charts
(Nguyen et al. @ MSR 2014)(Nguyen et al. @ MSR 2014)

19. Differential flame graphsDifferential flame graphs
(Bezemer et al. @ SANER 2015)(Bezemer et al. @ SANER 2015)

21. Deciding on the baseline Deciding on the baseline
performanceperformance

22. Deciding on the acceptable Deciding on the acceptable
performance deviationperformance deviation

23. Reporting results to the Reporting results to the
developerdeveloper

24. Availability of Availability of
performance testsperformance tests

25. Execution time of testsExecution time of tests

26. Performance regression testing helps Performance regression testing helps
ensure we are not breaking performanceensure we are not breaking performance
Write code
Execute
performance
test
Deploy if performance is OK or fix changes

28. Directions in performance Directions in performance
regression analysis researchregression analysis research
1. Execution profile comparison
2. Source code analysis
3. Visualization

30. Availability of Availability of
performance testsperformance tests

32. Execution time of testsExecution time of tests