Mastering strace: Practical Tips for Tracing System Calls and Debugging Linux Processes

strace is a lightweight Linux utility that records every system call a program makes, showing the call name, arguments, and return values. It can be used to trace a program from start to finish, filter specific calls, profile time spent in each call, and attach to running processes.

Basic Usage and Filtering

Running strace php 2>&1 | grep php.ini reveals which configuration files PHP attempts to open, showing both successful and failed open calls. Adding the -e open qualifier limits output to only open calls, simplifying the search for configuration paths.

$ strace -e open php 2>&1 | grep php.ini
open("/usr/local/bin/php.ini", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/php.ini", O_RDONLY) = 4

Similar filtering can be applied to other syscalls, such as open,access to investigate permission‑related failures:

$ strace -e open,access 2>&1 | grep your-filename

Inspecting a Running Process

Attach to a process by PID with strace -p <PID>. The -p option shows live system calls; adding -c aggregates time, call count, and errors for each syscall, producing a profiling summary.

# strace -c -p 11084
% time     seconds  usecs/call  calls  errors  syscall
------ ----------- ----------- ------ ------ --------
94.59    0.001014      48      21          select
2.89     0.000031       1      21          getppid
2.52     0.000027       1      21          time
------ ----------- ----------- ------ ------ --------
100.00   0.001072                63          total

The example shows most CPU time spent in select(), indicating the process is largely waiting for I/O.

Network Tracing Example

Tracing a network client such as nc can reveal DNS lookups, socket connections, and data transfer:

$ strace -e poll,select,connect,recvfrom,sendto nc www.news.com 80
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("62.30.112.39")}, 28) = 0
poll([{fd=3, events=POLLOUT, revents=POLLOUT}], 1, 0) = 1
sendto(3, "...", 30, MSG_NOSIGNAL, NULL, 0) = 30
... (additional send/recv calls) ...
connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("216.239.122.102")}, 16) = -1 EINPROGRESS (Operation now in progress)
select(4, NULL, [3], NULL, NULL) = 1 (out [3])

The trace shows an initial attempt to contact the Name Service Cache Daemon (NSCD), a DNS query to resolve the hostname, and a non‑blocking connect to the target IP.

Advanced Options

Combine -T (show elapsed time per call) with -e clone to measure how long a clone() operation takes, useful for identifying expensive process creation:

# strace -T -e clone -p <PID>
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD) = 12345 < 0.312 ms

To capture only read/write activity of a process and its children, use: strace -f -e trace=read,write -p 17151 -o log Finally, the full command‑line syntax is documented in the man page; common qualifiers include trace, abbrev, raw, signal, read, and write. The -e expr option lets you include or exclude specific syscalls, using patterns like -e trace=!open to trace everything except open.

By mastering these options, developers and system administrators can quickly locate configuration problems, performance bottlenecks, and unexpected network behavior without restarting services.