Debugging C++ Deadlocks on Linux with Shell Commands and GDB

1. Overview

Multithreaded C++ programs on Linux can become unresponsive when a deadlock occurs, causing all threads to wait indefinitely for resources held by each other. The article demonstrates how to use shell commands and the GDB debugger to locate and fix such deadlocks.

2. Deadlock Basics

A deadlock happens when two or more threads each hold a lock that the other needs, creating a circular wait. The article uses the analogy of two people meeting on a narrow bridge and refusing to yield, illustrating the concept.

3. Common Causes of Deadlocks

Typical reasons include:

Inconsistent lock acquisition order (e.g., thread1 locks mutex1 then mutex2 while thread2 does the opposite).

Repeated locking of a non‑reentrant mutex.

Failing to release a lock due to exceptions or logic errors.

3.1 Inconsistent Lock Order

#include <iostream>
#include <thread>
#include <mutex>

std::mutex mutex1;
std::mutex mutex2;

void thread1Function() {
    mutex1.lock();
    std::cout << "Thread 1: Acquired mutex1" << std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(1));
    mutex2.lock();
    std::cout << "Thread 1: Acquired mutex2" << std::endl;
    mutex2.unlock();
    mutex1.unlock();
}

void thread2Function() {
    mutex2.lock();
    std::cout << "Thread 2: Acquired mutex2" << std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(1));
    mutex1.lock();
    std::cout << "Thread 2: Acquired mutex1" << std::endl;
    mutex1.unlock();
    mutex2.unlock();
}

int main() {
    std::thread thread1(thread1Function);
    std::thread thread2(thread2Function);
    thread1.join();
    thread2.join();
    return 0;
}

The differing lock order creates a deadlock when both threads run concurrently.

3.2 Repeated Locking

#include <iostream>
#include <thread>
#include <mutex>

std::mutex myMutex;

void recursiveFunction(int count) {
    myMutex.lock();
    std::cout << "Entering recursiveFunction, count: " << count << std::endl;
    if (count > 0) {
        recursiveFunction(count - 1);
    }
    myMutex.unlock();
    std::cout << "Exiting recursiveFunction, count: " << count << std::endl;
}

int main() {
    std::thread myThread(recursiveFunction, 3);
    myThread.join();
    return 0;
}

Because std::mutex is not re‑entrant, the second lock attempt deadlocks.

3.3 Missing Unlock

#include <iostream>
#include <thread>
#include <mutex>

std::mutex mutex;

void someFunction() {
    mutex.lock();
    std::cout << "Locked the mutex" << std::endl;
    throw std::runtime_error("Something went wrong");
    mutex.unlock();
    std::cout << "Unlocked the mutex" << std::endl;
}

int main() {
    std::thread thread(someFunction);
    thread.join();
    return 0;
}

Using try‑catch or RAII helpers like std::lock_guard prevents this situation.

4. Simulating a Deadlock Scenario

The article compiles the above code with:

g++ -g -std=c++11 -pthread deadlock_example.cpp -o deadlock

Running ./deadlock shows the program hanging, confirming a deadlock.

5. Preliminary Shell Checks

5.1 View Process Status

Use ps aux | grep deadlock to locate the process and examine fields such as PID, %CPU, %MEM, and STAT. Low CPU usage and a blocked state often indicate a deadlock.

5.2 Inspect Thread Resource Usage

Run top -Hp <PID> to see per‑thread CPU and memory consumption. Threads stuck at ~0% CPU for an extended period suggest they are waiting on a lock.

6. Deep Investigation with GDB

6.1 Attach to the Running Process

Gain root privileges ( su) and attach GDB:

gdb attach 12345

6.2 Show All Thread Stacks

Execute thread apply all bt to dump backtraces for every thread. The output shows which functions each thread is executing and where they are blocked (e.g., inside pthread_mutex_lock).

6.3 Locate the Deadlocked Thread

Use info threads to list thread IDs, then switch to a specific thread with thread <id> and run bt again. The backtrace reveals the exact source line (e.g., line 13 in deadlock_example.cpp) where a thread is waiting for a mutex.

7. Full Summary

The combined use of ps aux and top -Hp provides a high‑level indication of a possible deadlock, while GDB’s attach, thread apply all bt, and per‑thread inspection pinpoint the offending threads and code locations, enabling developers to resolve the deadlock efficiently.