xCapture v3: Efficient, Always-On Thread Level Observability with eBPF by Tanel Poder

xCapture is an eBPF-based Linux thread activity measurement tool for systematic performance troubleshooting and optimization that can go very deep. Thanks to modern eBPF task iterators, xCapture's passive sampling mode does not have to inject probes to execution paths of other tasks. For itself, xCapture uses at least an order of magnitude less CPU time compared to running "top." In this talk, I'll launch the prod/GA ready version of xCapture v3 + xtop and demonstrate how it helps with real life troubleshooting scenarios.

1. A ScyllaDB Community
xCapture v3:
Efficient, Always-On Thread
Level Observability with eBPF
Tanel Põder
Performance Nerd

2. Tanel Põder

A long-time computer performance nerd
■I made biolatency eBPF code 59x more efficient
■... when tested on 384 CPU server at 37M IOPS ☺
■P99CONF just keeps getting better, I’ll do my part!
■When not working, I hang out with 3..4 of our cats

3. Extended Thread State Sampling
Tracking + Sampling (not tracing)!

Two decoupled layers:

●eBPF populating & maintaining a thread state array
●Keeps only the latest state change for each thread

●Sampling program independent from population
●Python/BCC, C, Rust/libbpf, eBPF iterators, etc...

●Passive sampling without any probes or
tracepoints!

4. xtop demo

9. dimensional performance analysis

10. xcapture architecture

11. xcapture
modular
architecture

12. Where to get it?
Open source, multiple performance tools:
■https://0x.tools
■https://github.com/tanelpoder/0xtools

13. Thank you! Let’s connect.
Tanel Põder
[email protected]
@tanelpoder
tanelpoder.com