How to Detect and Fix C++ Deadlocks on Linux Using Shell and GDB

When programming C++ on Linux, multithreading provides powerful concurrency, but deadlocks can silently halt a program as threads wait indefinitely for each other to release resources.

Part 1 – Deadlock: The Hidden Killer in Multithreaded Programming

Deadlocks occur when two or more threads each hold a resource the other needs, creating a cycle that prevents any progress, much like two people stuck on a narrow bridge.

In server applications, a deadlock can stop the server from handling new client requests, causing severe service disruption.

Part 2 – Tracing the Root Causes of Deadlocks

2.1 Improper Lock Acquisition Order

If threads acquire multiple mutexes in different orders, they can deadlock. The following example shows two threads locking mutex1 and mutex2 in opposite order.

#include <iostream>
#include <thread>
#include <mutex>

std::mutex mutex1;
std::mutex mutex2;

void thread1Function() {
    mutex1.lock();
    std::cout << "Thread 1: Acquired mutex1" << std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(1));
    mutex2.lock();
    std::cout << "Thread 1: Acquired mutex2" << std::endl;
    mutex2.unlock();
    mutex1.unlock();
}

void thread2Function() {
    mutex2.lock();
    std::cout << "Thread 2: Acquired mutex2" << std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(1));
    mutex1.lock();
    std::cout << "Thread 2: Acquired mutex1" << std::endl;
    mutex1.unlock();
    mutex2.unlock();
}

int main() {
    std::thread thread1(thread1Function);
    std::thread thread2(thread2Function);
    thread1.join();
    thread2.join();
    return 0;
}

Because the lock order differs, the threads can become deadlocked.

2.2 Re‑locking a Non‑Recursive Mutex

Calling lock() on a mutex that the same thread already holds (and that is not recursive) causes a deadlock.

#include <iostream>
#include <thread>
#include <mutex>

std::mutex myMutex;

void recursiveFunction(int count) {
    myMutex.lock();
    std::cout << "Entering recursiveFunction, count: " << count << std::endl;
    if (count > 0) {
        recursiveFunction(count - 1);
    }
    myMutex.unlock();
    std::cout << "Exiting recursiveFunction, count: " << count << std::endl;
}

int main() {
    std::thread myThread(recursiveFunction, 3);
    myThread.join();
    return 0;
}

The recursive call attempts to lock myMutex again, leading to a deadlock.

2.3 Forgetting to Unlock After an Exception

If a thread throws an exception while holding a mutex and never unlocks it, other threads will block forever.

#include <iostream>
#include <thread>
#include <mutex>

std::mutex mutex;

void someFunction() {
    mutex.lock();
    std::cout << "Locked the mutex" << std::endl;
    throw std::runtime_error("Something went wrong");
    mutex.unlock();
    std::cout << "Unlocked the mutex" << std::endl;
}

int main() {
    std::thread thread(someFunction);
    thread.join();
    return 0;
}

The exception prevents mutex.unlock() from executing, causing a deadlock.

Part 3 – Building a Deadlock Test Scenario

The same code from section 2.1 can be compiled with debugging symbols to create an executable that deliberately deadlocks.

g++ -g -o deadlock_example deadlock_example.cpp -lpthread

Running ./deadlock_example prints the acquisition messages and then hangs, demonstrating a deadlock.

Part 4 – Shell Tools: Inspecting Process State

4.1 Using ps aux to View Process Overview

Execute ps aux | grep deadlock_example to see CPU and memory usage. A deadlocked process typically shows near‑zero CPU usage while still holding memory.

4.2 Using top -Hp <pid> for Thread‑Level Analysis

The top -Hp command shows each thread's CPU usage. Threads stuck at 0% CPU may indicate they are blocked in a lock.

Part 5 – GDB: Deep Debugging to Pinpoint the Deadlock

5.1 Attaching GDB to the Running Process

Find the PID with ps aux and attach GDB:

gdb -p 12345

5.2 thread apply all bt – Show Stack Traces for All Threads

The backtrace reveals both threads stopped inside __GI___pthread_mutex_lock, confirming the deadlock location.

5.3 info threads – List Thread States

This command lists each thread’s ID and current frame, allowing you to switch to a specific thread with thread <id> and inspect its stack.

By combining shell inspection and GDB debugging, developers can quickly locate and resolve deadlocks in Linux C++ multithreaded applications.