How to Detect and Resolve Linux Thread Deadlocks with pstack and gdb

Intermittent "Deadlock found when trying to get lock" errors in an online service

Although the error occurs only about twenty times a day and does not yet impact performance, it must be resolved before it becomes a bottleneck.

What is a deadlock?

A deadlock occurs when multiple processes compete for exclusive resources (e.g., printers, tape drives, file inodes) and each holds one resource while waiting for another, creating a circular wait.

Four necessary conditions for deadlock

Mutual exclusion

Hold and wait

No preemption

Circular wait

Deadlock illustration

Analyzing deadlocks with pstack and gdb

pstack

pstack prints the stack trace of all threads in a Linux process, helping to locate which threads are blocked on mutexes.

gdb

gdb is a powerful debugger for C/C++ programs on Unix/Linux. It can attach to a running process, inspect thread states, and examine variable values.

Sample deadlock program

#include <unistd.h>
#include <pthread.h>
#include <string.h>

pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t mutex2 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t mutex3 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t mutex4 = PTHREAD_MUTEX_INITIALIZER;

static int sequence1 = 0;
static int sequence2 = 0;

int func1()
{
    pthread_mutex_lock(&mutex1);
    ++sequence1;
    sleep(1);
    pthread_mutex_lock(&mutex2);
    ++sequence2;
    pthread_mutex_unlock(&mutex2);
    pthread_mutex_unlock(&mutex1);
    return sequence1;
}

int func2()
{
    pthread_mutex_lock(&mutex2);
    ++sequence2;
    sleep(1);
    pthread_mutex_lock(&mutex1);
    ++sequence1;
    pthread_mutex_unlock(&mutex1);
    pthread_mutex_unlock(&mutex2);
    return sequence2;
}

void* thread1(void* arg)
{
    while (1) {
        int iRetValue = func1();
        if (iRetValue == 100000) {
            pthread_exit(NULL);
        }
    }
}

void* thread2(void* arg)
{
    while (1) {
        int iRetValue = func2();
        if (iRetValue == 100000) {
            pthread_exit(NULL);
        }
    }
}

void* thread3(void* arg)
{
    while (1) {
        sleep(1);
        char szBuf[128];
        memset(szBuf, 0, sizeof(szBuf));
        strcpy(szBuf, "thread3");
    }
}

void* thread4(void* arg)
{
    while (1) {
        sleep(1);
        char szBuf[128];
        memset(szBuf, 0, sizeof(szBuf));
        strcpy(szBuf, "thread3");
    }
}

int main()
{
    pthread_t tid[4];
    if (pthread_create(&tid[0], NULL, &thread1, NULL) != 0) _exit(1);
    if (pthread_create(&tid[1], NULL, &thread2, NULL) != 0) _exit(1);
    if (pthread_create(&tid[2], NULL, &thread3, NULL) != 0) _exit(1);
    if (pthread_create(&tid[3], NULL, &thread4, NULL) != 0) _exit(1);
    sleep(5);
    pthread_join(tid[0], NULL);
    pthread_join(tid[1], NULL);
    pthread_join(tid[2], NULL);
    pthread_join(tid[3], NULL);
    pthread_mutex_destroy(&mutex1);
    pthread_mutex_destroy(&mutex2);
    pthread_mutex_destroy(&mutex3);
    pthread_mutex_destroy(&mutex4);
    return 0;
}

Compiling and running the test program

g++ -g lock.cpp -o lock -lpthread

Finding the process ID and using pstack

ps -ef | grep lock

Example output shows the PID (e.g., 6721). Running pstack 6721 prints stack traces for each thread. The first run shows threads 4 and 5 blocked in pthread_mutex_lock, indicating a deadlock.

Inspecting the deadlocked threads with gdb

(gdb) attach 6721
(gdb) info thread
(gdb) thread 5
(gdb) where
(gdb) thread 4
(gdb) where

The backtraces reveal that thread 5 is waiting for mutex2 while holding mutex1, and thread 4 is waiting for mutex1 while holding mutex2. This cross‑locking confirms the deadlock.

Summary

The article demonstrates a practical method for diagnosing Linux thread deadlocks using pstack and gdb, explains the underlying theory, and provides preventive guidelines such as acquiring locks in a fixed order, splitting large transactions, locking all needed resources at once, lowering isolation levels when possible, and using proper indexing.

How to avoid deadlocks

Access tables/rows in a consistent order.

Break large transactions into smaller ones.

Lock all required resources within a single transaction.

Reduce isolation level if business permits.

Add appropriate indexes to minimize row‑level locking.