How to Detect and Resolve Linux Thread Deadlocks with pstack and gdb

Deadlock Overview

In computer systems, resources such as printers, tapes, or file inodes can be used by only one process at a time. When multiple processes compete for limited resources and acquire them in an inconsistent order, they may enter a circular wait, resulting in a deadlock.

Deadlocks waste system resources and can cause complete system failure, so they must be taken seriously both theoretically and technically.

Four Necessary Conditions

(1) Mutual exclusion: a resource can be used by only one process at a time. (2) Hold and wait: a process holding resources may request additional ones. (3) No preemption: resources cannot be forcibly taken from a process. (4) Circular wait: processes form a circular chain of resource requests.

Analyzing Deadlocks with pstack and gdb

pstack prints the stack trace of all threads in a Linux process, while gdb is a powerful debugger for C/C++ programs. Together they allow you to locate threads stuck in lock acquisition.

pstack Introduction

pstack is a useful Linux tool that outputs the stack information of a process, showing the call stacks of all threads.

gdb Introduction

GDB (GNU Debugger) lets you inspect running programs, set breakpoints, examine variables, and step through code. To debug with source information, compile with the -g flag.

Test Program

#include <unistd.h>
#include <pthread.h>
#include <string.h>

pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t mutex2 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t mutex3 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t mutex4 = PTHREAD_MUTEX_INITIALIZER;

static int sequence1 = 0;
static int sequence2 = 0;

int func1()
{
    pthread_mutex_lock(&mutex1);
    ++sequence1;
    sleep(1);
    pthread_mutex_lock(&mutex2);
    ++sequence2;
    pthread_mutex_unlock(&mutex2);
    pthread_mutex_unlock(&mutex1);
    return sequence1;
}

int func2()
{
    pthread_mutex_lock(&mutex2);
    ++sequence2;
    sleep(1);
    pthread_mutex_lock(&mutex1);
    ++sequence1;
    pthread_mutex_unlock(&mutex1);
    pthread_mutex_unlock(&mutex2);
    return sequence2;
}

void* thread1(void* arg)
{
    while (1)
    {
        int iRetValue = func1();
        if (iRetValue == 100000)
        {
            pthread_exit(NULL);
        }
    }
}

void* thread2(void* arg)
{
    while (1)
    {
        int iRetValue = func2();
        if (iRetValue == 100000)
        {
            pthread_exit(NULL);
        }
    }
}

void* thread3(void* arg)
{
    while (1)
    {
        sleep(1);
        char szBuf[128];
        memset(szBuf, 0, sizeof(szBuf));
        strcpy(szBuf, "thread3");
    }
}

void* thread4(void* arg)
{
    while (1)
    {
        sleep(1);
        char szBuf[128];
        memset(szBuf, 0, sizeof(szBuf));
        strcpy(szBuf, "thread3");
    }
}

int main()
{
    pthread_t tid[4];
    if (pthread_create(&tid[0], NULL, &thread1, NULL) != 0) { _exit(1); }
    if (pthread_create(&tid[1], NULL, &thread2, NULL) != 0) { _exit(1); }
    if (pthread_create(&tid[2], NULL, &thread3, NULL) != 0) { _exit(1); }
    if (pthread_create(&tid[3], NULL, &thread4, NULL) != 0) { _exit(1); }
    sleep(5);
    pthread_join(tid[0], NULL);
    pthread_join(tid[1], NULL);
    pthread_join(tid[2], NULL);
    pthread_join(tid[3], NULL);
    pthread_mutex_destroy(&mutex1);
    pthread_mutex_destroy(&mutex2);
    pthread_mutex_destroy(&mutex3);
    pthread_mutex_destroy(&mutex4);
    return 0;
}

Compiling the Test Program

g++ -g lock.cpp -o lock -lpthread

Finding the Process ID

ps -ef|grep lock

First pstack Output

[dyu@xilinuxbldsrv purify]$ pstack 6721
Thread 5 (Thread 0x41e37940 (LWP 6722)):
#0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003d1a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x0000003d1a808cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000400a9b in func1() ()
#4  0x0000000000400ad7 in thread1(void*) ()
#5  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0
#6  0x0000003d19cd40cd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x42838940 (LWP 6723)):
#0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003d1a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x0000003d1a808cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000400a17 in func2() ()
#4  0x0000000000400a53 in thread2(void*) ()
#5  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0
#6  0x0000003d19cd40cd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x43239940 (LWP 6724)):
#0  0x0000003d19c9a541 in nanosleep () from /lib64/libc.so.6
#1  0x0000003d19c9a364 in sleep () from /lib64/libc.so.6
#2  0x00000000004009bc in thread3(void*) ()
#3  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d19cd40cd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x43c3a940 (LWP 6725)):
#0  0x0000003d19c9a541 in nanosleep () from /lib64/libc.so.6
#1  0x0000003d19c9a364 in sleep () from /lib64/libc.so.6
#2  0x0000000000400976 in thread4(void*) ()
#3  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d19cd40cd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b984ecabd90 (LWP 6721)):
#0  0x0000003d1a807b35 in pthread_join () from /lib64/libpthread.so.0
#1  0x0000000000400900 in main ()

Second pstack Output

[dyu@xilinuxbldsrv purify]$ pstack 6721
Thread 5 (Thread 0x40bd6940 (LWP 6722)):
#0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003d1a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x0000003d1a808cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000400a87 in func1() ()
#4  0x0000000000400ac3 in thread1(void*) ()
#5  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0
#6  0x0000003d19cd40cd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x415d7940 (LWP 6723)):
#0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003d1a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x0000003d1a808cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000400a03 in func2() ()
#4  0x0000000000400a3f in thread2(void*) ()
#5  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0
#6  0x0000003d19cd40cd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x41fd8940 (LWP 6724)):
#0  0x0000003d19c7aec2 in memset () from /lib64/libc.so.6
#1  0x00000000004009be in thread3(void*) ()
#2  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0
#3  0x0000003d19cd40cd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x429d9940 (LWP 6725)):
#0  0x0000003d19c7ae0d in memset () from /lib64/libc.so.6
#1  0x0000000000400982 in thread4(void*) ()
#2  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0
#3  0x0000003d19cd40cd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2af906fd9d90 (LWP 6721)):
#0  0x0000003d1a807b35 in pthread_join () from /lib64/libpthread.so.0
#1  0x0000000000400900 in main ()

gdb Analysis

(gdb) info thread
5 Thread 0x41e37940 (LWP 6722)  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
4 Thread 0x42838940 (LWP 6723)  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
3 Thread 0x43239940 (LWP 6724)  0x0000003d19c9a541 in nanosleep () from /lib64/libc.so.6
2 Thread 0x43c3a940 (LWP 6725)  0x0000003d19c9a541 in nanosleep () from /lib64/libc.so.6
*1 Thread 0x2b984ecabd90 (LWP 6721)  0x0000003d1a807b35 in pthread_join () from /lib64/libpthread.so.0

(gdb) thread 5
#0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003d1a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x0000003d1a808cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000400a9b in func1 () at lock.cpp:18
#4  0x0000000000400ad7 in thread1 (arg=0x0) at lock.cpp:43

(gdb) thread 4
#0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#3  0x0000000000400a17 in func2 () at lock.cpp:31

Conclusion

The article demonstrates a practical method for analyzing Linux deadlocks using pstack and gdb, showing how to identify threads stuck in lock acquisition and confirming a cross‑lock deadlock between two threads. Understanding the four necessary conditions helps prevent future deadlocks.

How to Prevent Deadlocks

Access tables and rows in a fixed order (e.g., sort IDs before batch updates).

Split large transactions into smaller ones.

Lock all required resources within a single transaction when possible.

Lower isolation levels if business logic permits (e.g., from RR to RC).

Add appropriate indexes to avoid row‑level locking.