Comprehensive Guide to Concurrency Optimization in Modern CPUs and Multithreaded Programming

1. Background

Performance of a program depends on algorithm complexity, I/O overhead, and concurrency. Baidu's C++ engineers previously discussed memory‑I/O optimizations; this article focuses on concurrency optimization.

2. Why Concurrency?

Large‑scale services and machine‑learning systems require many instances; improving single‑instance throughput often means exploiting multi‑core parallelism because CPU frequency growth has stalled.

CPU trends show increasing core counts while clock speed plateaus, making effective use of multiple cores essential.

Concurrency acceleration works by splitting a task into independent subtasks that can run simultaneously on parallel execution units.

3. Parallel Execution in a Single Thread

3.1 SIMD

SIMD (Single Instruction Multiple Data) allows a single instruction to operate on multiple data elements in a wide register, providing up to 8× speedup for suitable workloads.

Modern CPUs augment core count with wider registers to increase per‑core throughput without the cost of adding full cores.

SIMD is especially useful for repetitive, data‑parallel loops such as memcpy, vectorizable loops, and specialized libraries like Swiss Table, simdjson, and base64.

3.2 Out‑of‑Order Execution (OoOE)

OoOE reorders instructions to keep execution units busy when a dependency or cache miss would otherwise stall the pipeline.

Modern CPUs combine OoOE with superscalar execution, creating a pipeline that resembles lightweight multithreading.

3.3 Top‑down Microarchitecture Analysis Method (TMAM)

TMAM uses PMU counters to locate bottlenecks by comparing actual instruction‑per‑cycle (IPC) against the theoretical maximum.

int array[1024];
for (size_t i = 0; i < 1024; i += 2) {
    int a = array[i];
    int b = array[i + 1];
    for (size_t j = 0; j < 1024; ++j) {
        a = a + b;
        b = a + b;
    }
    array[i] = a;
    array[i + 1] = b;
}

4. Critical‑Section Protection in Multithreaded Concurrency

4.1 What Is a Critical Section?

Critical sections serialize access to shared data; they become scalability bottlenecks as core count grows.

4.1.1 Mutual Exclusion

Traditional mutexes are pessimistic, blocking all threads when contention occurs.

4.1.2 Lock‑Free

Lock‑free algorithms use optimistic CAS or atomic operations, allowing threads to retry on conflict.

4.1.3 Wait‑Free

Wait‑free guarantees each thread completes in a bounded number of steps, eliminating global stalls but requiring more complex designs.

// Mutual Exclusion
{
    std::lock_guard
lock(mutex);
    sum += calc(sequence, workload);
}

// Lock Free / Atomic CAS
{
    auto current = atomic_sum.load(std::memory_order_relaxed);
    auto next = current;
    do {
        next = current + calc(sequence, workload);
    } while (!atomic_sum.compare_exchange_weak(current, next, std::memory_order_relaxed));
}

// Wait Free / Atomic Modify
{
    atomic_sum.fetch_add(calc(sequence, workload), std::memory_order_relaxed);
}

Experiments show FAA (fetch‑add)‑based wait‑free scaling outperforms both mutex and lock‑free approaches, especially under high contention.

4.3 Counter Optimization Cases

Baidu's bvar counters use per‑thread locks that are cheap because contention is low; newer CPUs allow 128‑bit atomic loads/stores to eliminate locks for combined count‑and‑sum updates.

4.4 Queue Optimization Cases

Lock‑protected queues become bottlenecks under contention. Splitting head and tail locks, lock‑free Michael‑Scott queues, and sharded queues (e.g., tbb::concurrent_queue) reduce latency.

Baidu's babylon::ConcurrentBoundedQueue assigns a unique sequence number to each enqueue/dequeue, mapping it to a slot in a circular buffer, achieving per‑slot synchronization and wait‑free progress for producers.

Overall, effective concurrency optimization requires understanding hardware trends, leveraging SIMD and OoOE, and carefully choosing synchronization primitives based on contention characteristics.