Scheduler Tracing With ftrace + eBPF by Jason Rahman

Dive into understanding app latency by exploring the Linux scheduler with ftrace, eBPF, and Perfetto for visualization. Uncover quirks in the CFS scheduler and get a glimpse of the new EEVDF scheduler in recent kernels. #Linux #eBPF #Performance #DevOps

1. A ScyllaDB Community
Linux Scheduler
Tracing With Ftrace
Jason Rahman
Principal Software Engineer at Microsoft

2. Jason Rahman

Principal Software Engineer at Microsoft
■Working On Azure PostgreSQL These Days
■Previously @ Meta Working On MySQL & Distributed File Systems
■Enjoy Learning About The Intricacies Of Operating
Systems + Hardware
■In My Free Time: Adjunct Instructor + Landscape
Photography
■(IG: @jrahman.photography)

3. Linux Scheduler Tracing

4. Linux ftrace subsystem
Linux has a tracing subsystem built into the kernel itself.
■Tracepoints defined across various kernel subsystems
●This includes events like sched/sched_switching
●Can reconstruct full scheduling timeline
■Configured via tracefs
●Enable events with `echo 1 > /sys/kernel/tracing/events/${EVENT}/enable`
●Read trace buffers with ` cp /sys/kernel/tracing/trace ./trace `

5. Lockfree, low overhead per-CPU ring buffers

6. Perfetto Trace Viewer
Data Is Helpful, Visualizations Are Better
■dev.perfetto.ui Has Native Ftrace Support

7. Service Threading Architecture

8. Asynchronous IO Application Architecture

9. Respond Back To Source Thread(s)

10. Key Workload Behaviors
Each group of threads has specific behaviors and performance requirements:
■IO Threads
●Cooperative multi-tasking, avoid long running tasks that block others
●Short bursts of CPU upon IO completion
●Latency sensitive, need to get IO to/from disk/network ASAP
●CPU Threads
●Work-stealing, shared work queue to distribute long running tasks
●Longer running CPU heavy tasks

11. Worker Thread Wake Up To Process Queue Entry

12. IO Thread Wake Up To Send Response

13. Asynchronous IO Application Latency
With at least 2x thread hops per request (likely more in practice including
downstream API calls), scheduler behavior on wakeup has significant impact on
latency
■CPU Tasks Occupy CPU Core For Long Periods Of Time
■IO Threads Need To Wake Up Quickly To Service IO
■The former often interferes with the latter, as we’ll see
■Need high fidelity to debug scheduling problems that occur

14. Let’s Try It Out!

15. Sample Ftrace Usage
EVENTS=“sched/sched_waking sched/sched_switch sched/sched_wakeup_new”
for EVENT in ${EVENTS}; do
sudo bash -c 'echo 1 > /sys/kernel/tracing/events/${EVENT}/enable’
done
sudo bash -c 'echo > /sys/kernel/tracing/trace'
sudo bash -c 'echo 1 > /sys/kernel/tracing/tracing_on'
sleep 1
sudo bash -c 'cp /sys/kernel/tracing/trace ./trace'
sudo bash -c 'echo 0 > /sys/kernel/tracing/tracing_on’
for EVENT in ${EVENTS}; do
sudo bash -c 'echo 0 > /sys/kernel/tracing/events/${EVENT}/enable’
done

16. Load Generator - PWM
let ticks_per_us = ticks_per_us();
loop {
let start = tick();
let target = start + run_us * ticks_per_us;
while tick() < target {
unsafe {
x86_64::_mm_pause();
}
}
std::thread::sleep(Duration::from_micros(period_us - run_us));
}

17. 3 Tasks Consuming 65% Of 2 Cores => Moderate Load

18. Task Runnable + Core Left Idle => Not Work Conserving!

19. Per CPU Runqueue

20. CFS Failing To Load Balance – Overload On Wakeup Bug

21. A Picture Is Worth 1000 Words, A Few Good Traces Are Worth A Research Paper

22. Lessons
What We Learned Today
■Ftrace
●Powerful Tool For Detailed Performance Analysis Accessible via Tracefs
●Built Into Linux, No Additional Tools Needed
●Perfetto
●Quick + Lightweight Trace Viewer In Browser
■Performance Analysis
●Linux Scheduler Has Bugs + Poor Behaviors
●CPU Load Is Average Measurement
■Need Traces To Find Find Grained Latency Issues

23. Thank you! Let’s connect.
Jason Rahman
[email protected]
https://jprahman.substack.com