In‑Depth Analysis of Workflow’s 300‑Line C Thread Pool Implementation

🍀 0 - Workflow’s thrdpool

Workflow, a popular asynchronous scheduling engine on GitHub, ships a tiny but powerful thread‑pool module written in C (about 300 lines). The pool is the backbone of Workflow’s executor and is useful for any C/C++ project.

🍀 1 - Prerequisite Knowledge

Why do we need a thread pool? Creating threads with pthread_create or std::thread is cheap, but the number of CPU cores is limited. Excess threads cause overhead and resource contention, so a pool that reuses a fixed number of threads is desirable.

A typical thread pool contains:

Management of a set of worker threads.

A task queue.

Synchronization primitives (mutex, condition variable).

Pool‑specific bookkeeping.

🍀 2 - Code Overview

The implementation can be understood in seven steps.

Step 1: Header file (API)

// Create a thread pool
thrdpool_t *thrdpool_create(size_t nthreads, size_t stacksize);
// Submit a task to the pool
int thrdpool_schedule(const struct thrdpool_task *task, thrdpool_t *pool);
// Destroy the pool
void thrdpool_destroy(void (*pending)(const struct thrdpool_task *), thrdpool_t *pool);

Step 2: Task structure

struct thrdpool_task {
    void (*routine)(void *);   // function pointer
    void *context;             // user data
};

Step 3: Internal pool structure

struct __thrdpool {
    struct list_head task_queue;   // pending tasks
    size_t nthreads;               // number of workers
    size_t stacksize;              // thread stack size
    pthread_t tid;                // a single placeholder ID
    pthread_mutex_t mutex;        // protects the queue
    pthread_cond_t cond;          // notifies workers
    pthread_key_t key;            // thread‑local key
    pthread_cond_t *terminate;     // termination flag/cond
};

Every member has a precise purpose: tid holds the ID of the thread that will join the previous worker, mutex and cond implement the classic producer‑consumer pattern, key marks threads created by the pool, and terminate signals shutdown.

Step 4: Core creation function

thrdpool_t *thrdpool_create(size_t nthreads, size_t stacksize) {
    thrdpool_t *pool;
    ret = pthread_key_create(&pool->key, NULL);
    if (ret == 0) {
        memset(&pool->tid, 0, sizeof(pthread_t));
        pool->terminate = NULL;
        if (__thrdpool_create_threads(nthreads, pool) >= 0)
            return pool;
        ...
    }
    return NULL;
}

The helper __thrdpool_create_threads() loops to spawn nthreads workers.

Step 5: Worker routine

static void *__thrdpool_routine(void *arg) {
    while (1) {
        pthread_mutex_lock(&pool->mutex);
        while (!pool->terminate && list_empty(&pool->task_queue))
            pthread_cond_wait(&pool->cond, &pool->mutex);
        if (pool->terminate)
            break;
        entry = list_entry(*pos, struct __thrdpool_task_entry, list);
        list_del(*pos);
        pthread_mutex_unlock(&pool->mutex);
        task_routine = entry->task.routine;
        task_context = entry->task.context;
        free(entry);
        task_routine(task_context);
        if (pool->nthreads == 0) {
            free(pool);
            return NULL;
        }
    }
    ...
}

The loop fetches a task, releases the lock, executes the routine, and checks whether the pool is being destroyed.

Step 6: Scheduling a task

inline void __thrdpool_schedule(const struct thrdpool_task *task, void *buf, thrdpool_t *pool) {
    struct __thrdpool_task_entry *entry = (struct __thrdpool_task_entry *)buf;
    entry->task = *task;
    pthread_mutex_lock(&pool->mutex);
    list_add_tail(&entry->list, &pool->task_queue);
    pthread_cond_signal(&pool->cond);
    pthread_mutex_unlock(&pool->mutex);
}

This demonstrates “Feature 2”: a task can schedule another task, even while the pool is shutting down.

Step 7: Destroying the pool

void thrdpool_destroy(void (*pending)(const struct thrdpool_task *), thrdpool_t *pool) {
    // 1. set termination flag and wake all workers
    pool->terminate = &term;
    pthread_cond_broadcast(&pool->cond);
    // 2. collect remaining tasks via pending()
    list_for_each_safe(pos, tmp, &pool->task_queue) {
        entry = list_entry(pos, struct __thrdpool_task_entry, list);
        list_del(pos);
        if (pending) pending(&entry->task);
        ...
    }
    // 3. wait for workers to finish (see __thrdpool_terminate())
    ...
    free(pool);
}

The shutdown is “feature 1”: workers exit one‑by‑one without storing every thread ID. Each worker, after noticing pool->terminate , joins the previous worker using the single tid placeholder, forming a chain of joins that finally wakes the thread that called thrdpool_destroy .

🍀 3 - Feature 1: Elegant One‑by‑One Exit

Instead of tracking all thread IDs, the pool uses a single tid field. When a worker sees the termination flag, it records the current tid , replaces it with its own ID, and then joins the previously stored thread. The last worker signals the external destroyer.

🍀 4 - Feature 2: Tasks Can Spawn New Tasks

Because the queue is protected by a mutex/condition variable, a task running inside a worker may safely call thrdpool_schedule to enqueue another task, even during shutdown.

🍀 5 - Feature 3: Tasks May Destroy the Pool

A task can invoke thrdpool_destroy from within the worker. The implementation detects an in‑pool destroyer, detaches the calling thread, decrements the worker count, and lets the final worker free the pool memory after completing its own routine.

🍀 6 - Simple Usage Example

void my_routine(void *context) {
    printf("task-%llu start.\n", (unsigned long long)context);
}

void my_pending(const struct thrdpool_task *task) {
    printf("pending task-%llu.\n", (unsigned long long)task->context);
}

int main() {
    thrdpool_t *pool = thrdpool_create(3, 1024);
    struct thrdpool_task task;
    for (unsigned long long i = 0; i < 5; ++i) {
        task.routine = &my_routine;
        task.context = (void *)i;
        thrdpool_schedule(&task, pool);
    }
    getchar(); // pause
    thrdpool_destroy(&my_pending, pool);
    return 0;
}

Running the program prints task start messages, then pending messages for any tasks that were not executed before shutdown.

🍀 7 - Concurrency and Structural Beauty

The author reflects on how a compact, well‑designed pool can express sophisticated concurrency concepts with minimal code, emphasizing the importance of rigorous design for framework‑level components.

Source repository: https://github.com/sogou/workflow