When a Linux Program Crashes: Using Backtrace to Quickly Pinpoint the Fault

Common Reasons for Linux Program Crashes

Memory overflow: allocating more memory than available, e.g., a loop that continuously allocates without freeing.

Null‑pointer dereference: accessing a pointer that has not been assigned a valid address.

File I/O errors: reading a non‑existent file or lacking permissions.

Insufficient system resources: exhausting file descriptors, process slots, network connections, etc.

Logical errors: incorrect condition checks that lead to unexpected execution paths.

System exceptions: hardware faults or kernel crashes that terminate the process.

What Is backtrace?

backtrace ("回溯") generates a function‑call stack when a program crashes or receives a fatal signal. The stack records the sequence of function calls that led to the failure.

Working Principle

During execution the CPU maintains a call stack that stores return addresses, parameters, and local variables. Each function call pushes its return address onto the stack; when the function returns, the address is popped. backtrace walks this stack from the current frame upward, extracting the return address, function name (if available), and offset for each frame, and assembles a readable report.

Example Call Chain

Assume functions A → B → C, where C crashes. backtrace starts at C, records C’s return address (pointing to the call site in B), then moves to B, and finally to A, producing a list of A, B, C with their offsets.

Backtrace‑Related Functions and Usage

int backtrace(void *buffer, int size)

Fills buffer (an array of void*) with up to size return addresses and returns the actual number captured.

#define BT_BUF_SIZE 100
void *buffer[BT_BUF_SIZE];
int nptrs = backtrace(buffer, BT_BUF_SIZE);

char **backtrace_symbols(void const *buffer, int size)

Converts the addresses in buffer to an array of printable strings containing the function name (if resolvable), hexadecimal offset, and address.

char **strings = backtrace_symbols(buffer, nptrs);
for (int i = 0; i < nptrs; i++) {
    printf("%s
", strings[i]);
}
free(strings);

void backtrace_symbols_fd(void const *buffer, int size, int fd)

Writes the same information directly to the file descriptor fd without allocating memory, making it safe in low‑memory situations.

backtrace_symbols_fd(buffer, nptrs, STDOUT_FILENO);

Usage Notes

Compilation options : Avoid optimization levels that remove the frame pointer (e.g., -O2 -fomit-frame-pointer). Compile with debugging symbols and export symbols:

gcc -g -rdynamic -o myprogram myprogram.c

Special cases :

Static functions are not exported, so their names may appear as raw addresses.

Inline functions have no separate stack frame, thus cannot be shown in the backtrace.

Tail‑call optimization reuses the current frame, potentially dropping the caller’s frame.

Capturing System Signals to Obtain the Stack

When a program receives a fatal signal (e.g., SIGSEGV for illegal memory access or SIGFPE for divide‑by‑zero), the kernel terminates the process. By installing a signal handler, backtrace can be invoked before termination.

Using the signal Mechanism

Example registers handlers for SIGSEGV and SIGFPE and prints the backtrace inside the handler:

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <execinfo.h>

void signal_handle(int signal) {
    void *buffer[100];
    char **strings;
    int nptrs;
    printf("
==========> catch signal %d <==========
", signal);
    nptrs = backtrace(buffer, 100);
    strings = backtrace_symbols(buffer, nptrs);
    if (strings == NULL) {
        perror("backtrace_symbols");
        exit(EXIT_FAILURE);
    }
    for (int i = 0; i < nptrs; i++) {
        printf("%s
", strings[i]);
    }
    free(strings);
    exit(EXIT_FAILURE);
}

int main() {
    signal(SIGSEGV, signal_handle);
    signal(SIGFPE, signal_handle);
    int *ptr = NULL;
    *ptr = 10; // trigger null‑pointer dereference
    return 0;
}

Running this program prints a stack trace that includes the signal handler, the offending function, and the entry point.

Case Study: Using backtrace to Locate a Crash

Test Code

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <execinfo.h>

void signal_handle(int signal) { /* same as above */ }

void null_pointer_dereference() {
    int *ptr = NULL;
    *ptr = 10; // triggers SIGSEGV
}

void division_by_zero() {
    int a = 10;
    int b = 0;
    int result = a / b; // triggers SIGFPE
}

int main() {
    signal(SIGSEGV, signal_handle);
    signal(SIGFPE, signal_handle);
    null_pointer_dereference();
    // division_by_zero();
    return 0;
}

Analyzing the Output

Compiled with gcc -g -rdynamic -o test test.c, the program produces:

==========> catch signal 11 <==========
Dump stack start...
./test(signal_handle+0x55) [0x400975]
/lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f0ff12af4b0]
./test(null_pointer_dereference+0x10) [0x400a40]
./test(main+0x2f) [0x400a9f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f0ff129a830]
./test(_start+0x29) [0x400929]
Dump stack end...

The first frame is the signal handler, followed by null_pointer_dereference, indicating the fault location. Using addr2line -e test 0x400a40 -f yields the exact source line ( test.c:19).

Debugging in a Shared Library

When the crash occurs inside a shared object, the raw address refers to a runtime load address, not a file offset. Workflow:

Compile the library with -g -rdynamic:

g++ -g -rdynamic backtrace.cpp -fPIC -shared -o libbacktrace.so

Link the executable with -Wl,-rpath=. so the loader finds the library at runtime:

g++ -g -rdynamic main.cpp -L . -lbacktrace -Wl,-rpath=.

When a signal is caught, note the address (e.g., 0x7f9eddc84ada).

Read /proc/$$/maps to obtain the library’s load base (e.g., 0x7f9eddc84000).

Subtract the base to get the offset ( 0xada), then run addr2line -e libbacktrace.so 0xada to map to the source line ( backtrace.cpp:47).

Alternatively, generate a map file with -Wl,-Map,map.log or use objdump -d libbacktrace.so to locate the function entry address and compute the final offset.

Full Summary

When a Linux program terminates abruptly, first enumerate possible causes such as memory overflow, null‑pointer dereference, I/O errors, resource exhaustion, logical bugs, or system‑level exceptions. Compile with debugging symbols ( -g) and export symbols ( -rdynamic) while avoiding optimizations that remove the frame pointer.

Install signal handlers for fatal signals (e.g., SIGSEGV, SIGFPE) and invoke backtrace inside the handler. Convert the raw addresses with backtrace_symbols or backtrace_symbols_fd for readable output.

Use addr2line (or nm, objdump, map files) to translate addresses to source file lines. For shared libraries, adjust the address by the library’s load base obtained from /proc/<pid>/maps.

These steps provide a systematic, reproducible method to locate the exact line of code that caused a crash, turning a mysterious termination into a solvable debugging problem.