Finding Performance Needles in Haystacks with APerf by Geoffrey Blake

Finding performance issues in modern software is like finding a needle in a haystack and intuition on where to look first is often wrong. APerf is an open source tool we have used many times to help with performance debugging by looking "wide" before going "deep". This session will present the tool along with an example of debugging a difficult P99 latency problem that was recently fixed in upstream Linux.

1. A ScyllaDB Community
Finding Performance Needles in
Haystacks with APerf
Geoffrey Blake
Principal Engineer

2. Geoffrey Blake (he/him)

Principal Engineer at AWS
■Help AWS and customers optimize on Graviton
■Enjoy solving performance puzzles in all domains
■When not at work enjoy flying little airplanes to
weird destinations

3. HELP!
We upgraded and
performance is worse!
Weird, it should
be faster!

4. Groovy web-app P99 latency on AWS Instances
3ms
40ms

5. 13x slower

6. Now what?
flamegraphs
eBPF
strace
ftrace
tracepoints
JFR+JMC
tcpdump
/proc stats netstat
mpstat
sysstat
iostat
sysctl
kernel config
…

7. Wide then deep debugging
■Intuition/instincts unreliable
■Big gains hide in plain sight
■Breadth first search


■APerf tool for wide then deep






github.com/aws/aperf

8. Complementary tool in the toolbox
■APerf to look for signals
●Simple to use
●100’s of system metrics
●High ?????? low level
■Specific tools to go deep
●eBPF
●Wireshark
●ftrace
●…

9. APerf quick start
%> wget
https://github.com/aws/aperf/releases/download/v0.1.16-alpha/aperf-v0.1.16-al
pha-aarch64.tar.gz

%> tar –zxf aperf-v0.1.16-alpha-aarch64.tar.gz
%> export PATH=$PATH:$PWD/aperf-v0.1.16-alpha-aarch64

%> sudo aperf record -r debug_session_1 -i 1 -p 600

%> aperf report -r debug_session_1 -n debug_session_report
<generates debug_session_report.tar.gz>

https://github.com/aws/aperf/blob/main/EXAMPLE.md

10. APerf report UI
10x bugs
2x bugs
<2x bugs

11. APerf CPU utilization view

12. APerf PMU stats view

13. APerf Net Stats view
sysctl-explorer.net
/net/ipv4/
tcp_autocorking/

14. TCPDump deep-dive

15. Why?
tcp_push(...) {
...
if (tcp_should_autocork(...)) {
if (!test_bit(TSQ_THROTTLED, sk)){
set_bit(TSQ_THROTTLED, sk);
}
if (refcount_read(sk->wmem) > n)
return
}
...
__tcp_push_pending_frames(...);
tcp_wfree(...) { ...
refcount_sub_test(sk->wmem);
oval = smp_load_acquire(sk);
do {
if (!(oval & TSQ_THROTTLED))
goto out;
nval = (oval & ~TSQ_THROTTLED);
} while(!try_cmpxchg(sk, oval, nval);
...
tasklet_schedule();
return;
out: sk_free(sk);
/net/ipv4/tcp.c:tcp_push() net/ipv4/tcp_output.c:tcp_wfree()

16. The fix
tcp_push(...) {
...
if (tcp_should_autocork(...)) {
if (!test_bit(TSQ_THROTTLED, sk)){
set_bit(TSQ_THROTTLED, sk);
+ smp_mb__after_atomic();
}
if (refcount_read(sk->wmem) > n)
return
}
...
__tcp_push_pending_frames(...);
tcp_wfree(...) { ...
refcount_sub_test(sk->wmem);
oval = smp_load_acquire(sk);
do {
if (!(oval & TSQ_THROTTLED))
goto out;
nval = (oval & ~TSQ_THROTTLED);
}while(!try_cmpxchg(sk, oval, nval);
...
tasklet_schedule();
return;
out: sk_free(sk);

17. P99 latency after 1-liner kernel fix

18. P99’s hide anywhere

Wide then deep

APerf can help
github.com/aws/aperf

19. Thank you! Let’s connect.
Geoffrey Blake
[email protected]
linkedin – Geoffrey Blake