"Making .NET Application Even Faster", Sergey Teplyakov.pptx

In this talk we're going to explore performance improvement lifecycle, starting with setting the performance goals, using profilers to figure out the bottle necks, making a fix and validating that the fix works by benchmarking it. The talk will be useful for novice and seasoned .NET developers and architects interested in making their application fast and understanding how things work under the hood.

1. About the speaker Wrote a book on Design Patterns in .NET Blogged for a decade at “ Programming Stuff ” and “ Dissecting the Code ” Author of ErrorProne.NET Analyzers Ex-maintainer of Code Contracts library Twitter: @STeplyakov Github : SergeyTeplyakov 7/16/2024 Making .NET Applications even faster 2

2. Computers are fast! Right? 7/16/2024 Making .NET Applications even faster 3

3. Performance matters System level software Frameworks and libraries Performance is a feature in some contexts High-frequency trading E-commerce Networking Software development tools Reduce the cloud bills 7/16/2024 Making .NET Applications even faster 4

4. How to identify a performance problem? A bug report of some sort A system fails with out of memory Long start time affects system’s availability GC pauses causes sporadic spikes of service unavailability errors User’s studies or surveys Slow responses causes users to leave The devs are unhappy with build time 7/16/2024 Making .NET Applications even faster 5

5. Use the data, Luke! Perf data pyramid High level metrics Profiling data for prod/test Local profiling data for integration tests Benchmark results 7/16/2024 Making .NET Applications even faster 6

6. Performance optimization workflow Start with the problem P90 for request processing is taking > 1ms, causing overall system delays Create a hypothesis of what’s wrong A high time in GC causes sporadic delays affecting high percentiles of request processing Back the hypothesis with data The “Time in GC%” dashboard shows that GC is taking significant portion of execution time Reproduce the issue as locally as possible Run a end-2-end test and measure GC time by the profiler Make a change Switch to server GC Validate the impact in E2E test (or in a benchmark) Validate the impact in dev/ ppe /prod environments 7/16/2024 Making .NET Applications even faster 7

7. Looking into the data Time in GC metric Time in GC profiling 7/16/2024 Making .NET Applications even faster 8

8. The E2E impact P90 request duration dropped by 2x 7/16/2024 Making .NET Applications even faster 9

9. Don’t afraid the setbacks 7/16/2024 Making .NET Applications even faster 10

10. High Performance Application is a journey You can't complete performance work The world is constantly changing Load Use cases Environment Runtimes Performance bugs and regressions will happen Measure often. Look at the metrics. Profile your applications

11. Perf PR best practices State the problem: a chart, a profiling session or something that states the problem. Explain the perf issue. Show the impact of the change. Use as high-level metrics as possible: E2E, not only the benchmarks Show the benchmark results (if possible) 7/16/2024 Making .NET Applications even faster 12

12. Performance vs. Maintainability Focus on readability, maintainability and composability first (*) Identify the hotspots and bottle necks with real load Optimize only when needed Its easier to make a well written code faster than to make a fast but incorrect code work correctly! Don’t overcomplicate the system for no reason!

13. The code can be fast and expressive Span-based code can be complex, but it can be equally or even more expressive New string interpolation impl is faster and more expressive

14. Avoid work if possible How can you beat this? 7/16/2024 Making .NET Applications even faster 15

15. Avoid work if possible (2) Q: What’s the fastest sorting algorithm? A: No sorting at all! 7/16/2024 Making .NET Applications even faster 16

16. Fixing the bottlenecks The problem Service runs out of memory (takes >300Gb of memory) Startup time is very high (>1h) 7/16/2024 Making .NET Applications even faster 17

17. Profiler for the resque 7/16/2024 Making .NET Applications even faster 18

18. What’s the issue? String interning! String interning saves 15% of memory footprint Causes a massive performance penalty A solution? A custom interning cache! More on this: https://sergeyteplyakov.github.io/Blog/benchmarking/2023/12/10/Intern_or_Not_Intern.html 7/16/2024 Making .NET Applications even faster 19

19. The benchmark Benchmarking a global state is tricky! 7/16/2024 Making .NET Applications even faster 20

20. End 2 End Impact 7/16/2024 Making .NET Applications even faster 21 Improved the startup time by ~20%!

21. Small changes with huge impact

22. Performance Optimization Journey Performance fixes are addictive Fixing one perf issue reveals other hot spots At some point you have to stop Set the perf goal Know about diminishing returns 7/16/2024 Making .NET Applications even faster 23

23. Death by a thousand cuts Not all the perf issues are clearly visible in metrics or even under a profiler! Examples: Method call overhead Excessive allocations (and the GC) CPU oversubscription Indexing access Object Pooling Cache unfriendliness 7/16/2024 Making .NET Applications even faster 24

24. InvariantCulture vs. OrdinalIgnoreCase Do you ever use ` StringComparison.InvariantCulture ` in your code? “Culture specific” lookups are expensive (yes, “invariant culture” is a culture specific comparison)

25. The Benchmarks

26. End-2-End results Improved the startup time by ~20%! 7/16/2024 Making .NET Applications even faster 27

27. Memory allocations in performance analysis Look for allocations And Time in GC% Allocation = work. In some cases, redundant work 7/16/2024 Making .NET Applications even faster 28

28. Looking for allocations Scenario: xml processing Step 1: load a file into MemoryStream Step 2: process MemoryStream 7/16/2024 Making .NET Applications even faster 29

29. What can we do? 1. Remove the optimization 2. Pass the size to MemoryStream’s constructor 3. Pool MemoryStream instances 7/16/2024 Making .NET Applications even faster 30

30. Before and After Original version: 34Gb of allocations Constructing MemoryStreams with file sizes: 24Gb Pooling MemoryStreams : 13Gb Overall impact: 20% perf improvements in E2E scenarios 7/16/2024 Making .NET Applications even faster 31

31. Looking at iterator allocations Do you see an issue in this code? 7/16/2024 Making .NET Applications even faster 32

32. Profiling data 7/16/2024 Making .NET Applications even faster 33

33. Yes, we’re using the wrong Contains! 7/16/2024 Making .NET Applications even faster 34

34. Conclusion Performance analysis is VERY data driven Make sure you have top level perf metrics Do not increase complexity without support from the data Use metrics, profiling and benchmarks to back your decisions Focus on simplicity first Performance fixes are addictive Stop when the returns are not worth it End-2-End 7/16/2024 Making .NET Applications even faster 35

35. THANK YOU Sergey Tepliakov