Using Change Point Detection to Fight Noisy Benchmark Results by Matt Fleming

Discovering performance regressions in modern systems is tough due to inevitable noise. Change Point Detection (CPD) algorithms are gaining traction for tackling this challenge. This talk covers how CPD works and shares examples of real regressions found in open source projects. #DevOps #CPD

1. A ScyllaDB Community
Matt Fleming
Co-founder at Nyrkiö / Systems Engineer at Cloudflare
Using Change Point Detection to
Fight Noisy Benchmark Results

2. ■Former Linux kernel maintainer
■Focused on performance of OS, DB, and dist sys
■Co-authored papers on Change Point Detection
and testing distributed systems
■Enjoys running
<Matt Fleming> (he/him)

Co-founder at Nyrkiö

3. What is noise?

4. What is noise?
■Noise means “high variability”
■Large range of possible benchmark values without changing the system

5. What is noise?

6. What is noise?
This is noise

7. What is noise?
■Noise comes from indeterminism
■All modern systems have indeterminism
■Usually because of peak performance

8. Examples of noise

9. Examples of noise

10. All Systems Contain Noise

11. Fighting Noise

12. Fighting noise
■Eyeball it
●Possible to spend hours every month staring at graphs
●Laborious and time consuming
●Difficult to onboard new people because of institutional knowledge

13. Fighting noise
■Eyeball it
●Possible to spend hours every month staring at graphs
●Laborious and time consuming
●Difficult to onboard new people because of institutional knowledge
■Thresholds
●Hard to calculate
●Hard to maintain (doesn’t scale)

14. Change Point Detection

15. ■CPD has existed for decades
■Used in many areas
●Climate change
●Fraud detection
●Biomedical research
CPD History

17. CPD Algorithm Steps
search() cmp() filter()

18. ■Signal-processing-library (MongoDB)
■Perfolizer (Used in .NET)
■Ruptures
■Hunter
Open Source CPD libraries

19. Example

20. Linux Kernel syscall

21. Drawbacks
■CPD needs multiple data points
■There’s a “delay” in when change points are identified
■Computational complexity can be prohibitive

22. Resources
■A Select Review of offline change point detection methods
■A Survey of Methods for time series change point detection
■MongoDB paper
■Hunter paper

23. Thank you! Let’s connect.
Matt Fleming
[email protected]
@fleming_matt