Debugging realtime application with Ftrace

FOSDEM 2018

Introducing Ftrace and using it for Linux realtime application

Ftrace (Function Tracer) is a very famous tracing/debugging tool. It is built directly into the Linux kernel and most of standard distributions already have various configurations of Ftrace enabled in their most recent releases.

One of the benefits that Ftrace brings to Linux is the ability to see what is happening inside the kernel with a large number of 'tracers' (function or events) .

kernel function calling tree and timestamps
kernel events (such as processus wakeup or switch)

As such, this makes finding problem areas or simply tracking down that strange bug more manageable. Ftrace's ability to show the events that lead up to severe problem gives a better chance of finding exactly what caused it and can help the developer a lot. API to talk with Ftrace is located in a Debugfs file system. typically mounted at /sys/kernel/debug. You can also use 'trace-cmd' utility for easier access.

After a brief history and introduction about tracing tools (SystemTap, eBPF, LTT-ng, ...) we will (briefly) talk about Ftrace internals and describe main features :

installing it on embedded distribution such as Buildroot or Yocto (ARM platform)
tracing kernel functions (function, function_graph)
kernel messages (trace_printk)
markers (trace_marker)
events

After describing a simple use-case for a kernel driver we will focus on events and use trace-cmd on "realtime" application running on standard Linux then PREEMPT_RT (ARM).

We will compare performances of a POSIX based periodic task using SCHEDFIFO or SCHEDOTHER on both kernel and finally generate graphs with GNU-plot.

Speakers: Pierre Ficheux