Why Your 128‑Core Server Underperforms: Unlock 300% Gains with CPU Affinity

Why Your 128‑Core Server Performs Worse Than a 32‑Core One: The Power of CPU Affinity

Preface: A Real Case A senior architect discovered that a newly purchased dual‑socket AMD EPYC 7763 (128 cores) server performed worse than a previous 32‑core server under high concurrency. After investigating, the issue was traced to CPU affinity configuration. Proper tuning increased performance by 300%.

Do you encounter similar problems? This article dives into CPU affinity configuration and load‑balancing optimization for multi‑core servers.

Why CPU Affinity Matters

Challenges of Modern Server Architecture

In modern data centers, servers often have dozens or hundreds of CPU cores, but these cores are not identical:

NUMA architecture : Memory access latency can differ by up to 300% between nodes.

Cache hierarchy : L1/L2/L3 cache affinity impacts performance.

Hyper‑threading : Scheduling strategies differ for physical vs logical cores.

The Reality of Performance Loss

Unoptimized systems may suffer:

Frequent process migration between cores causing cache invalidation.

Cross‑NUMA memory accesses increasing latency 2‑3×.

Key processes competing for CPU resources.

CPU Affinity Configuration in Practice

1. System Topology Analysis

First, understand the server’s CPU topology:

# 查看CPU拓扑信息
lscpu
lstopo --of txt

# 查看NUMA节点信息
numactl --hardware

# 查看CPU缓存信息
cat /proc/cpuinfo | grep cache

Sample output:

Available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 ... 63
node 0 size: 131072 MB
node 1 cpus: 64 65 66 67 ... 127
node 1 size: 131072 MB

2. Process CPU Affinity

Method 1: Using taskset

# Bind a process to specific CPU cores
taskset -cp 0-7 <pid>

# Start a program with affinity
taskset -c 0-7 ./your_application

# Bind to a specific NUMA node
numactl --cpunodebind=0 --membind=0 ./your_application

Method 2: Setting affinity inside the program

#include <sched.h>
#include <pthread.h>

void set_cpu_affinity(int cpu_id) {
    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(cpu_id, &cpuset);
    pthread_t current_thread = pthread_self();
    pthread_setaffinity_np(current_thread, sizeof(cpu_set_t), &cpuset);
}

3. Advanced Configuration Strategies

Critical Service Isolation

# Add isolated CPUs to GRUB
echo "isolcpus=8-15" >> /etc/default/grub
update-grub
reboot

# Bind key services to isolated CPUs
taskset -cp 8-15 $(pgrep nginx)
taskset -cp 8-15 $(pgrep mysql)

Dynamic Load‑Balancing Script

#!/bin/bash
# auto_affinity.sh - Intelligent CPU affinity adjustment

get_cpu_usage() {
    top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d% -f1
}

adjust_affinity() {
    local pid=$1
    local current_cpu=$(taskset -cp $pid 2>/dev/null | awk '{print $NF}')
    local cpu_usage=$(get_cpu_usage)

    if (( $(echo "$cpu_usage > 80" | bc -l) )); then
        # High load: spread to more cores
        taskset -cp 0-15 $pid
    else
        # Low load: concentrate on few cores for cache efficiency
        taskset -cp 0-3 $pid
    fi
}

for pid in $(pgrep -f "nginx\|mysql\|redis"); do
    adjust_affinity $pid
done

Load‑Balancing Optimization Strategies

1. Kernel Scheduler Tuning

# Set scheduler policy
echo "mq-deadline" > /sys/block/sda/queue/scheduler

# Adjust CPU scheduler parameters
echo 1 > /proc/sys/kernel/sched_autogroup_enabled
echo 100000 > /proc/sys/kernel/sched_latency_ns
echo 10000 > /proc/sys/kernel/sched_min_granularity_ns

2. Interrupt Affinity Configuration

# View NIC interrupt distribution
cat /proc/interrupts | grep eth0

# Bind NIC interrupts to specific CPUs
echo 2 > /proc/irq/24/smp_affinity   # CPU1
echo 4 > /proc/irq/25/smp_affinity   # CPU2

# Enable automatic balancing
systemctl enable irqbalance
systemctl start irqbalance

3. Application‑Level Load Balancing

Nginx CPU Affinity

# nginx.conf
worker_processes auto;
worker_cpu_affinity auto;

# Manual example
worker_processes 8;
worker_cpu_affinity 0001 0010 0100 1000 10000 100000 1000000 10000000;

Redis Cluster CPU Optimization

# Bind Redis instances to different CPU cores
redis-server redis-6379.conf --cpu-affinity 0-3
redis-server redis-6380.conf --cpu-affinity 4-7
redis-server redis-6381.conf --cpu-affinity 8-11

Performance Monitoring and Tuning

1. Monitoring Metrics

#!/usr/bin/env python3
import psutil, time, json

def collect_cpu_metrics():
    metrics = {
        'timestamp': time.time(),
        'cpu_percent': psutil.cpu_percent(interval=1, percpu=True),
        'load_avg': psutil.getloadavg(),
        'context_switches': psutil.cpu_stats().ctx_switches,
        'interrupts': psutil.cpu_stats().interrupts,
        'numa_stats': {}
    }
    try:
        with open('/proc/numastat', 'r') as f:
            numa_data = f.read()
        metrics['numa_stats'] = parse_numa_stats(numa_data)
    except:
        pass
    return metrics

def parse_numa_stats(numa_data):
    stats = {}
    lines = numa_data.strip().split('
')
    headers = lines[0].split()[1:]
    for line in lines[1:]:
        parts = line.split()
        stat_name = parts[0]
        values = [int(x) for x in parts[1:]]
        stats[stat_name] = dict(zip(headers, values))
    return stats

while True:
    metrics = collect_cpu_metrics()
    print(json.dumps(metrics, indent=2))
    time.sleep(5)

2. Benchmarking CPU Affinity

# benchmark_cpu_affinity.sh
echo "=== CPU Affinity Performance Test ==="

# No affinity constraint
echo "Test 1: No CPU affinity"
time sysbench cpu --cpu-max-prime=20000 --threads=8 run

# Bind to the same NUMA node
echo "Test 2: Bind to NUMA node 0"
numactl --cpunodebind=0 --membind=0 sysbench cpu --cpu-max-prime=20000 --threads=8 run

# Distribute across NUMA nodes
echo "Test 3: Distribute across NUMA nodes"
numactl --interleave=all sysbench cpu --cpu-max-prime=20000 --threads=8 run

Common Pitfalls and Solutions

1. Over‑Binding

Symptoms:

Uneven system load.

Some cores idle while others are overloaded.

Overall performance degradation.

Solution:

# Smart load balancing
#!/bin/bash
balance_cpu_load() {
    local threshold=80
    for cpu in $(seq 0 $(($(nproc)-1))); do
        usage=$(top -bn1 | awk "/Cpu${cpu}/ {print $2}" | cut -d% -f1)
        if (( $(echo "$usage > $threshold" | bc -l) )); then
            migrate_processes $cpu
        fi
    done
}

migrate_processes() {
    local overloaded_cpu=$1
    local target_cpu=$(find_least_loaded_cpu)
    local pids=$(ps -eo pid,psr | awk "\$2==$overloaded_cpu {print \$1}")
    for pid in $pids; do
        taskset -cp $target_cpu $pid 2>/dev/null
        break
    done
}

2. Memory Locality Issues

# Check NUMA memory distribution
numastat -p $(pgrep your_app)

# Optimize memory allocation
echo 1 > /proc/sys/vm/zone_reclaim_mode
echo 1 > /sys/kernel/mm/numa/demotion_enabled

3. Interrupt Handling Optimization

# Automatic interrupt load balancing
#!/bin/bash
optimize_interrupts() {
    local nic_queues=$(ls /sys/class/net/eth0/queues/ | grep rx- | wc -l)
    local cpu_count=$(nproc)
    for ((i=0; i<nic_queues; i++)); do
        local cpu=$((i % cpu_count))
        local irq=$(cat /proc/interrupts | grep "eth0.*-$i" | cut -d: -f1 | tr -d ' ')
        echo $((1 << cpu)) > /proc/irq/${irq}/smp_affinity
    done
}

Future Trends

Hardware Directions

Heterogeneous computing : CPU+GPU+FPGA collaboration.

Deeper NUMA : 8‑level NUMA node architectures.

Intelligent scheduling : Adaptive hardware‑level scheduling.

Software Evolution

eBPF schedulers : User‑space custom scheduling policies.

Machine‑learning‑driven tuning : Intelligent optimization based on workload characteristics.

Container‑native optimization : Kubernetes CPU‑topology‑aware scheduling.

Actionable Recommendations

System Diagnosis : Use lstopo and numactl to understand server topology.

Bind Critical Processes : Pin databases, caches, and other key services to dedicated CPUs.

Interrupt Optimization : Configure NIC interrupt affinity.

Monitoring Setup : Deploy CPU‑affinity monitoring scripts.

Standardize Procedures : Establish SOPs for CPU affinity configuration.

Automation Tools : Develop tools that automatically apply optimal affinity settings.

Team Training : Educate staff on NUMA and CPU affinity concepts.

Continuous Improvement : Implement a feedback loop for ongoing performance tuning.