Getting Started with rtla timerlat: Cross‑Compilation and Usage Guide

Introduction

rtla timerlat is a Linux scheduling‑latency testing and analysis tool. Unlike traditional tools such as cyclictest, which report only overall latency, rtla timerlat splits latency into multiple stages, enabling developers to pinpoint the cause of large delays.

# cyclictest --mlockall --priority=97
policy: fifo: loadavg: 0.33 0.16 0.09 1/51 102
T: 0 (  102) P:97 I:1000 C:  10000 Min:    114 Act:  181 Avg:  185 Max:     417

rtla timerlat provides a detailed view of each latency component:

# rtla timerlat top -T 500 -s 500 -t -k -P f:95
  0 00:00:01   |          IRQ Timer Latency (us)        |          Thread Timer Latency (us)
CPU COUNT      |      cur   min   avg   max |      cur   min   avg   max
  0 #1015      |      236    37    42   236 |      594   213   227   594
---------------|----------------------------------------|---------------------------------------
ALL #1015   e0 |               37    42   236 |                213   227   594
rtla timerlat hit stop tracing
## CPU 0 hit stop tracing, analyzing ##
  IRQ handler delay:                         157.68 us (26.52 %)
  IRQ latency:                               236.24 us
  Timerlat IRQ duration:                     242.08 us (40.72 %)
  Blocking thread:                           101.84 us (17.13 %)
    Blocking thread stack trace
        -> stack_trace_save
        -> timerlat_save_stack.constprop.0
        -> timerlat_irq
        -> trace_event_buffer_commit
        -> __hrtimer_run_queues.constprop.0
        -> hrtimer_interrupt
        -> riscv_timer_interrupt
        -> handle_percpu_devid_irq
        -> ring_buffer_map
        -> handle_irq_desc
        -> riscv_intc_irq
        -> handle_riscv_irq
------------------------------------------------------------------------
  Thread latency:                             594.48 us (100%)

The output shows overall thread latency and the latency contributed by each stage.

Cross‑Compilation

The rtla source resides in tools/tracing/rtla within the Linux kernel tree and depends on libtracefs and libtraceevent. Both libraries must be built first.

$ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git
$ cd libtraceevent/
$ make CROSS_COMPILE=/path/to/cross/compile/riscv64-xx- CC=/path/to/cross/compile/riscv64-xx-gcc LD=/path/to/cross/compile/riscv64-xx-ld
$ sudo make install
$ cd ..
$ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git
$ cd libtracefs/
$ make CROSS_COMPILE=/path/to/cross/compile/riscv64-xx- CC=/path/to/cross/compile/riscv64-xx-gcc LD=/path/to/cross/compile/riscv64-xx-ld

After the libraries are built, compile rtla itself:

$ make CROSS_COMPILE=/path/to/cross/compile/riscv64-xx- CC=/path/to/cross/compile/riscv64-xx-gcc

Copy the resulting binary to the target board’s root filesystem.

Compilation Issues

Kernel sources before version 6.11 contain a bug in the detection of libtracefs and libtraceevent, causing the build to fail even when the libraries are installed. Partial fixes appear in 6.11‑rc1 (see commit1, commit2).

The file tools/build/features/test-libtraceevent.c must include #include <trace-seq.h> instead of #include <traceevent/trace-seq.h> (as shown in commit1).

If the compiler reports an undefined type, e.g.

tracefs.h:55:7: error: unknown type name 'cpu_set_t'

the definition may be missing from the cross‑compiler’s headers. Adding the following line to the rtla source resolves the issue:

#define _GNU_SOURCE

Usage

Kernel Configuration

rtla timerlat relies on the kernel’s tracer infrastructure, so the timerlat tracer must be enabled:

Kernel hacking  -->
  Tracers(FTRACE =y)  -->
    Timerlat tracer(TIMERLAT_TRACER =y)

Common Parameters

rtla timerlat offers two modes:

top – records maximum latency and stack traces.

hist – records a histogram of all latencies.

Frequently used options:

-p, --period us – measurement period.

-i, --irq us – stop test when IRQ latency exceeds this value.

-T, --thread us – stop test when thread latency exceeds this value.

-s, --stack us – capture stack trace when thread latency exceeds this value.

-t, --trace [file] – write results to the specified file (default: timerlat_trace.txt).

-q, --quiet – output only summary after the test.

-d, --duration time[s|m|h|d] – set test duration.

-P, --priority o:prio|r:prio|f:prio|d:runtime:period – set scheduling parameters for the test thread (e.g., -P f:95 uses SCHED_FIFO with priority 95).

Usage Example

A typical command:

# rtla timerlat top -T 500 -s 500 -t -k -P f:95

It runs the test with SCHED_FIFO priority 95, stops when thread latency exceeds 500 µs, and prints the stack trace of the blocking thread. The resulting table contains columns for IRQ Timer Latency (time to enter the timer interrupt handler) and Thread Timer Latency (time to schedule the test thread, which includes the IRQ latency). The columns cur, min, avg, and max represent the current, minimum, average, and maximum values observed. On a single‑core board only one row appears; on multi‑core boards each CPU gets its own row. The detailed breakdown and stack trace help developers identify the stage or thread causing blocking.

References

tools: build: use correct lib name for libtracefs feature detection

tools: Make pkg-config dependency checks usable by other tools

rtla timerlat documentation

Linux scheduling latency debug and analysis