Mastering POSIX Threads: From User Threads to Thread Pools in Linux

Theoretical Background

Linux Implementation Principles – Processes, Threads, Kernel Threads, Lightweight Processes, Coroutines

Linux Implementation Principles – Process Scheduling and Policy Configuration

Linux Implementation Principles – NUMA Multi‑core Architecture Thread Scheduling Overhead and Optimization

User Threads

User threads are created, scheduled, and destroyed by the user process and run in user space; the kernel does not perceive or manage them.

A user process can contain multiple user threads, including at least one main thread (TID = 1) that starts execution from main().

When developers create several user threads via the pthread library, those threads share the same process resources and only need a program counter, registers, and a stack, making them lightweight and suitable for high‑concurrency scenarios.

Thread Switching

Although the kernel does not schedule user threads directly, they are switched on the CPU allocated to the user process, and the switching is controlled entirely by the user process code (cooperative scheduling), avoiding kernel‑mode/user‑mode transition overhead.

pthread Thread Library

pthread (POSIX Threads) defines a portable API for thread operations. On modern Linux kernels the implementation is NPTL (Native POSIX Thread Library).

pthread uses a Thread Control Block (TCB) to store all thread information, which is much smaller than a Process Control Block (PCB). The relevant struct pthread is:

struct pthread {
    struct pthread *self;               // pointer to itself
    struct __pthread_internal_list *thread_list; // thread list for pool
    void *(*start_routine)(void*);      // entry function
    void *arg;                          // entry argument
    void *result;                       // return value
    pthread_attr_t *attr;               // attributes (stack size, policy, etc.)
    pid_t tid;                          // kernel‑assigned thread ID
    struct timespec *waiters;           // wait timestamps
    size_t guardsize;                   // stack guard size
    int sched_policy;                   // scheduling policy
    struct sched_param sched_params;    // scheduling parameters
    void *specific_1stblock;           // first block of thread‑specific data
    struct __pthread_internal_slist __cleanup_stack; // cleanup stack
    struct __pthread_mutex_s *mutex_list; // held mutexes
    struct __pthread_cond_s *cond_list;   // waited condition variables
    unsigned int detach_state:2;        // detached or joinable
    unsigned int sched_priority:30;     // scheduling priority
    unsigned int errno_val;            // error code
};

The pthread library provides a set of APIs around this structure for managing user threads.

Thread Creation and Destruction

pthread_create()

Purpose: Create a new thread and specify its entry function and argument.

Prototype:

thread : pointer to pthread_t where the kernel‑assigned TID will be stored.

attr : pointer to pthread_attr_t for thread attributes (stack size, scheduling policy, etc.), usually NULL.

start_routine : pointer to the thread’s entry function (must be a static or global function).

arg : argument passed to the entry function.

int pthread_create(pthread_t *thread, const pthread_attr_t *attr,
                   void *(*start_routine)(void *), void *arg);

After pthread_create(), the main/parent thread allocates a TCB, PC, registers, and stack for the child thread, initializes the TCB, and places it into the thread pool awaiting execution.

pthread_join()

Purpose: Wait for a specific child thread to terminate and retrieve its return value; the calling thread blocks until the child exits.

int pthread_join(pthread_t thread, void **retval);

pthread_exit()

Purpose: Terminate the calling thread immediately and return an exit code. void pthread_exit(void *retval); If the main thread calls pthread_exit(), the entire process terminates, so it must be used with care.

pthread_detach()

Purpose: Set a thread to detached state so that its resources are reclaimed automatically upon exit, without needing pthread_join().

int pthread_detach(pthread_t thread);

Multithread Safety and Synchronization

Multithread safety means that concurrent writes to shared data do not cause inconsistency. Synchronization mechanisms ensure data consistency.

pthread provides two primary synchronization tools:

Mutex : protects shared data by allowing only the thread holding the lock to access it.

Condition Variable : lets threads wait for a specific condition to become true before acquiring the associated mutex, preventing deadlocks.

Mutex Functions

pthread_mutex_init()

Purpose: Initialize a mutex object.

int pthread_mutex_init(pthread_mutex_t *mutex, const pthread_mutexattr_t *attr);

pthread_mutex_lock()

Purpose: Acquire a mutex; blocks if the mutex is already held by another thread.

int pthread_mutex_lock(pthread_mutex_t *mutex);

pthread_mutex_unlock()

Purpose: Release a mutex, making it available for other threads.

int pthread_mutex_unlock(pthread_mutex_t *mutex);

Condition Variable Functions

pthread_cond_init()

Purpose: Initialize a condition variable.

int pthread_cond_init(pthread_cond_t *cond, const pthread_condattr_t *attr);

pthread_cond_wait()

Purpose: Atomically release the associated mutex and block the calling thread until the condition variable is signaled.

int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex);

pthread_cond_signal()

Purpose: Wake up one thread waiting on the condition variable.

int pthread_cond_signal(pthread_cond_t *cond);

pthread_cond_broadcast()

Purpose: Wake up all threads waiting on the condition variable.

int pthread_cond_broadcast(pthread_cond_t *cond);

Combining Mutex and Condition Variable

When a thread needs a certain condition (e.g., a shared counter > 0) before accessing data, it first locks the mutex, checks the condition, and if not satisfied, calls pthread_cond_wait() to release the mutex and block until another thread signals the condition.

Thread‑Pool (tiny‑threadpool)

tiny‑threadpool is an open‑source lightweight thread‑pool written in ANSI C and POSIX style. It supports pause, resume, and wait operations and provides a simple API.

Data Structure Design

Task / Job

A job contains a function pointer, its argument, and a pointer to the next job.

/**
 * Job should contain three members:
 *   1. thread entry function;
 *   2. argument for the entry function;
 *   3. pointer to the next Job.
 */
typedef struct job {
    struct job *prev;               // next job in the queue
    void (*function)(void *arg);   // entry function
    void *arg;                     // argument
} job;

Task / Job Queue

The queue is FIFO and protected by a mutex and a binary semaphore.

typedef struct jobqueue {
    job *front;          // queue head
    job *rear;           // queue tail
    int len;             // length
    pthread_mutex_t rwmutex; // lock for concurrent access
    bsem *has_jobs;      // binary semaphore indicating non‑empty queue
} jobqueue;

typedef struct bsem {
    pthread_mutex_t mutex; // semaphore lock
    pthread_cond_t cond;    // semaphore condition
    int v;                 // 0: empty, 1: non‑empty
} bsem;

Worker / Thread

Each worker thread is the consumer that executes jobs.

typedef struct thread {
    int id;                     // friendly ID for debugging
    pthread_t pthread;           // underlying pthread
    struct thpool_ *thpool_p;    // back‑reference to the pool
} thread;

Thread‑Pool Manager

Manages threads, job queue, and synchronization primitives.

typedef struct thpool_ {
    thread **threads;                // array of thread pointers
    volatile int num_threads_alive; // currently alive threads
    volatile int num_threads_working; // threads currently executing jobs
    jobqueue jobqueue;               // associated job queue
    pthread_mutex_t thcount_lock;   // pool lock
    pthread_cond_t threads_all_idle; // condition variable for idle state
} thpool_;

Public APIs

typedef struct thpool_ *threadpool;

// Create a thread pool with a given number of threads
threadpool thpool_init(int num_threads);

// Add a task to the pool
int thpool_add_work(threadpool, void (*function_p)(void *), void *arg_p);

// Wait for all tasks to finish
void thpool_wait(threadpool);

// Pause all tasks (they will sleep)
void thpool_pause(threadpool);

// Resume paused tasks
void thpool_resume(threadpool);

// Destroy the pool, waiting for running tasks to finish
void thpool_destroy(threadpool);

// Get the number of threads currently working
int thpool_num_threads_working(threadpool);

Running Example

$ gcc example.c thpool.c -D THPOOL_DEBUG -pthread -o example

$ ./example
Making threadpool with 4 threads
THPOOL_DEBUG: Created thread 0 in pool
THPOOL_DEBUG: Created thread 1 in pool
THPOOL_DEBUG: Created thread 2 in pool
THPOOL_DEBUG: Created thread 3 in pool
Adding 40 tasks to threadpool
Thread #245600256 working on 0
Thread #246136832 working on 2
Thread #246673408 working on 3
Thread #246673408 working on 6
Thread #246673408 working on 7
...
Killing threadpool