Unlock 300% Linux Performance: Proven Kernel Tuning Secrets from 10 Years of Ops

Linux Kernel Performance Tuning in Practice: 10‑Year Ops Summary Kernel Optimization Secrets, System Performance Up 300%

1. Introduction

Linux kernel is the core of the OS, bridging applications and hardware. For ops engineers, deep understanding of kernel structure helps tuning and troubleshooting, and is essential to become senior ops experts. This article analyzes kernel architecture, core subsystems, performance optimization, with real code examples.

2. Overall Linux Kernel Architecture

Linux kernel uses a monolithic design, all core services run in kernel space. The kernel is divided into several layers:

┌─────────────────────────────────────────────────────────────┐
│                User Space                                 │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │  Applications│ │ System Tools│ │ Shell Commands│          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
└─────────────────────────────────────────────────────────────┘
               │ System Call Interface
┌─────────────────────────────────────────────────────────────┐
│                Kernel Space                               │
│  ┌─────────────────────────────────────────────────────┐ │
│  │               System Call Layer                     │ │
│  └─────────────────────────────────────────────────────┘ │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │  Process Mgmt│ │ Memory Mgmt │ │ Filesystem │          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │ Network Stack│ │ Device Drivers│ │ Security Modules│   │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
│  ┌─────────────────────────────────────────────────────┐ │
│  │               Hardware Abstraction Layer            │ │
│  └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
               │
┌─────────────────────────────────────────────────────────────┐
│                Hardware Layer                             │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │    CPU      │ │   Memory    │ │   Storage   │          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
└─────────────────────────────────────────────────────────────┘

Kernel modules are important components that can be loaded/unloaded at runtime.

# View basic kernel info
uname -a
cat /proc/version

# View kernel config
cat /boot/config-$(uname -r) | grep -E "CONFIG_(SMP|PREEMPT|RT)"

# View loaded modules
lsmod | head -10

# Load/unload module
modprobe module_name
rmmod module_name

# View module info
modinfo ext4

3. Process Management Subsystem

Process management is a core kernel function, handling creation, scheduling, synchronization, and termination. Linux uses CFS (Completely Fair Scheduler) as the default scheduler.

Process scheduling diagram:
┌─────────────────────────────────────────────────────────────┐
│                Process Scheduler                           │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │   CFS       │ │   RT        │ │   IDLE      │          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
│                     │                     │            │
│  ┌─────────────────────────────────────────────────────┐ │
│  │                Run Queues                           │ │
│  │  ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐           │ │
│  │  │ CPU0  │ │ CPU1  │ │ CPU2  │ │ CPU3  │           │ │
│  │  └───────┘ └───────┘ └───────┘ └───────┘           │ │
│  └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

#!/bin/bash
# Process scheduling analysis script
ps -eo pid,ppid,user,pri,ni,vsz,rss,pcpu,pmem,time --sort=-pcpu | head -20
cat /proc/schedstat | head -5
cat /proc/loadavg
uptime

renice_process() {
  local pid=$1
  local priority=$2
  if [ -z "$pid" ] || [ -z "$priority" ]; then
    echo "Usage: renice_process <PID> <priority(-20 to 19)>"
    return 1
  fi
  renice $priority $pid
  ps -o pid,ppid,user,pri,ni,comm -p $pid
}

set_realtime_process() {
  local pid=$1
  local priority=$2
  chrt -f -p $priority $pid
  echo "Process $pid set to realtime priority $priority"
}

Process state management includes running, sleeping, zombie, etc.

# Process state monitoring script
analyze_process_states() {
  echo "=== Process State Statistics ==="
  echo "Running(R): $(ps -eo stat | grep -c '^R')"
  echo "Sleeping(S): $(ps -eo stat | grep -c '^S')"
  echo "Uninterruptible sleep(D): $(ps -eo stat | grep -c '^D')"
  echo "Zombie(Z): $(ps -eo stat | grep -c '^Z')"
  echo "Stopped(T): $(ps -eo stat | grep -c '^T')"

  # Detailed zombie info
  zombie_count=$(ps -eo stat | grep -c '^Z')
  if [ $zombie_count -gt 0 ]; then
    echo "=== Zombie Process Details ==="
    ps -eo pid,ppid,user,stat,comm | grep ' Z '
  fi
}
analyze_process_states

4. Memory Management Subsystem

The memory management subsystem handles physical and virtual memory, including page allocation, reclamation, swapping, etc.

Memory management diagram:
┌─────────────────────────────────────────────────────────────┐
│                Virtual Memory Management                  │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │  Page Tables│ │   VMA Mgmt  │ │   mmap Mgmt │          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
└─────────────────────────────────────────────────────────────┘
               │
┌─────────────────────────────────────────────────────────────┐
│                Physical Memory Management                 │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │ Page Alloc  │ │ Slab Alloc  │ │ Memory Reclaim│      │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
│  ┌─────────────────────────────────────────────────────┐ │
│  │                Memory Zones                         │ │
│  │  ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐                     │ │
│  │  │ DMA │ │ Normal│ │ HighMem│ │ Movable│                │ │
│  │  └─────┘ └─────┘ └─────┘ └─────┘                     │ │
│  └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

#!/bin/bash
# Memory management analysis script
analyze_memory_layout() {
  echo "=== Memory Layout Analysis ==="
  echo "Physical memory info:"
  cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree"

  echo -e "
Memory zone info:"
  cat /proc/buddyinfo

  echo -e "
Virtual memory statistics:"
  cat /proc/vmstat | grep -E "pgfault|pgmajfault|pgpgin|pgpgout|pswpin|pswpout"
}

# Memory parameter optimization
optimize_memory_parameters() {
  echo "=== Memory Parameter Optimization ==="
  echo 10 > /proc/sys/vm/dirty_ratio
  echo 5 > /proc/sys/vm/dirty_background_ratio
  echo 500 > /proc/sys/vm/dirty_writeback_centisecs
  echo 3000 > /proc/sys/vm/dirty_expire_centisecs
  echo 1 > /proc/sys/vm/swappiness
  echo 100 > /proc/sys/vm/vfs_cache_pressure
  echo 1 > /proc/sys/vm/overcommit_memory
  echo 80 > /proc/sys/vm/overcommit_ratio
  echo "Memory parameters optimized"
}

analyze_memory_layout
optimize_memory_parameters

Huge pages improve memory management efficiency, especially for large‑memory workloads.

# Huge page configuration
configure_hugepages() {
  echo "=== Huge Page Configuration ==="
  cat /proc/meminfo | grep -E "HugePages|Hugepagesize"

  total_mem=$(grep MemTotal /proc/meminfo | awk '{print $2}')
  hugepage_size=$(grep Hugepagesize /proc/meminfo | awk '{print $2}')

  if [ $hugepage_size -gt 0 ]; then
    recommended_hugepages=$((total_mem * 20 / 100 / hugepage_size))
    echo "Recommended huge pages: $recommended_hugepages"
    echo $recommended_hugepages > /proc/sys/vm/nr_hugepages
    actual_hugepages=$(cat /proc/sys/vm/nr_hugepages)
    echo "Actual allocated huge pages: $actual_hugepages"
  fi

  echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
  echo defer > /sys/kernel/mm/transparent_hugepage/defrag
}
configure_hugepages

5. Filesystem Subsystem

The filesystem subsystem provides a unified VFS interface, supporting multiple filesystem types.

VFS architecture diagram:
┌─────────────────────────────────────────────────────────────┐
│                Applications                                 │
└─────────────────────────────────────────────────────────────┘
               │ System Call
┌─────────────────────────────────────────────────────────────┐
│                VFS Layer                                    │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │ inode cache │ │ dentry cache│ │ file cache │          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
└─────────────────────────────────────────────────────────────┘
               │
┌─────────────────────────────────────────────────────────────┐
│                Specific Filesystems                         │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │   ext4      │ │   xfs       │ │   btrfs    │          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
└─────────────────────────────────────────────────────────────┘
               │
┌─────────────────────────────────────────────────────────────┐
│                Block Device Layer                           │
└─────────────────────────────────────────────────────────────┘

#!/bin/bash
# Filesystem analysis script
analyze_filesystem_structure() {
  echo "=== Filesystem Structure Analysis ==="
  echo "Current mount points:"
  mount | column -t

  echo -e "
Filesystem type statistics:"
  mount | awk '{print $5}' | sort | uniq -c | sort -nr

  echo -e "
Filesystem usage:"
  df -h | grep -v tmpfs

  echo -e "
Inode usage:"
  df -i | grep -v tmpfs
}

monitor_filesystem_io() {
  echo "=== Filesystem I/O Monitoring ==="
  iostat -x 1 1
  echo -e "
Filesystem cache statistics:"
  cat /proc/sys/fs/file-nr
  echo "Open files: $(cat /proc/sys/fs/file-nr | awk '{print $1}')"
  echo "Max files: $(cat /proc/sys/fs/file-max)"
  echo -e "
Dentry cache statistics:"
  cat /proc/sys/fs/dentry-state
}

analyze_filesystem_structure
monitor_filesystem_io

Filesystem optimization is crucial for I/O performance; different filesystems have different strategies.

# ext4 optimization
optimize_ext4_filesystem() {
  local device=$1
  local mount_point=$2
  echo "=== ext4 Filesystem Optimization ==="
  mount -o remount,noatime,nodiratime,commit=60,barrier=0 $mount_point
  tune2fs -o journal_data_writeback $device
  blockdev --setra 8192 $device
  echo "ext4 optimization completed"
}

# XFS optimization
optimize_xfs_filesystem() {
  local device=$1
  local mount_point=$2
  echo "=== XFS Filesystem Optimization ==="
  mount -o remount,noatime,nodiratime,logbsize=256k,delaylog $mount_point
  xfs_fsr -v $mount_point
  echo "XFS optimization completed"
}

6. Network Subsystem

The network subsystem implements a full TCP/IP stack, providing network communication.

Network protocol stack diagram:
┌─────────────────────────────────────────────────────────────┐
│                Application Layer                           │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │   HTTP      │ │   FTP       │ │   SMTP      │          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
└─────────────────────────────────────────────────────────────┘
               │ Socket Interface
┌─────────────────────────────────────────────────────────────┐
│                Transport Layer                             │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │   TCP       │ │   UDP       │ │   SCTP      │          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
└─────────────────────────────────────────────────────────────┘
               │
┌─────────────────────────────────────────────────────────────┐
│                Network Layer                              │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │   IP        │ │   ICMP      │ │   ARP       │          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
└─────────────────────────────────────────────────────────────┘
               │
┌─────────────────────────────────────────────────────────────┐
│                Link Layer                                 │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐          │
│  │ Ethernet   │ │   WiFi      │ │   Other IF  │          │
│  └─────────────┘ └─────────────┘ └─────────────┘          │
└─────────────────────────────────────────────────────────────┘

#!/bin/bash
# Network subsystem analysis script
analyze_network_stack() {
  echo "=== Network Protocol Stack Analysis ==="
  echo "Network interface statistics:"
  cat /proc/net/dev | column -t

  echo -e "
TCP connection statistics:"
  ss -s

  echo -e "
Network protocol statistics:"
  cat /proc/net/snmp | grep -E "Tcp:|Udp:|Icmp:"

  echo -e "
Network queue statistics:"
  cat /proc/net/softnet_stat
}

optimize_network_parameters() {
  echo "=== Network Parameter Optimization ==="
  echo 'net.core.rmem_default = 262144' >> /etc/sysctl.conf
  echo 'net.core.rmem_max = 16777216' >> /etc/sysctl.conf
  echo 'net.core.wmem_default = 262144' >> /etc/sysctl.conf
  echo 'net.core.wmem_max = 16777216' >> /etc/sysctl.conf

  echo 'net.ipv4.tcp_fin_timeout = 30' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_keepalive_time = 1200' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_max_syn_backlog = 8192' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_syncookies = 1' >> /etc/sysctl.conf

  echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf
  echo 'net.core.netdev_budget = 600' >> /etc/sysctl.conf

  sysctl -p
  echo "Network parameters optimized"
}

analyze_network_stack
optimize_network_parameters

Network interface optimization is key to improving network performance, involving multi‑queue and interrupt affinity techniques.

# Network interface optimization
optimize_network_interfaces() {
  interfaces=$(ip link show | grep -E "^[0-9]+" | awk '{print $2}' | sed 's/://' | grep -v lo)
  for interface in $interfaces; do
    echo "Optimizing interface: $interface"
    if ethtool -l $interface 2>/dev/null | grep -q "Combined"; then
      max_queues=$(ethtool -l $interface | grep "Combined" | head -1 | awk '{print $2}')
      cpu_cores=$(nproc)
      queues=$((max_queues < cpu_cores ? max_queues : cpu_cores))
      ethtool -L $interface combined $queues
    fi
    ethtool -G $interface rx 4096 tx 4096 2>/dev/null
    ethtool -C $interface adaptive-rx on adaptive-tx on 2>/dev/null
    echo "Interface $interface optimization completed"
  done
}
optimize_network_interfaces

7. Kernel Performance Tuning in Practice

Kernel performance tuning is a crucial part of ops work, requiring holistic consideration of CPU, memory, I/O, and network.

#!/bin/bash
# Comprehensive kernel tuning script
comprehensive_kernel_tuning() {
  echo "=== Comprehensive Kernel Performance Tuning ==="
  tuning_config="/etc/sysctl.d/99-kernel-tuning.conf"
  cat > $tuning_config <<EOF
# Kernel tuning parameters
kernel.sched_migration_cost_ns = 5000000
kernel.sched_min_granularity_ns = 10000000
kernel.sched_wakeup_granularity_ns = 15000000

vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
vm.swappiness = 10
vm.vfs_cache_pressure = 50

net.core.rmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_default = 262144
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 5000

fs.file-max = 1048576
fs.inotify.max_user_watches = 1048576
EOF
  sysctl -p $tuning_config
  echo "Kernel parameters tuned"
}

monitor_kernel_performance() {
  echo "=== Kernel Performance Monitoring ==="
  while true; do
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | sed 's/%us,//')
    mem_info=$(free | grep Mem)
    mem_total=$(echo $mem_info | awk '{print $2}')
    mem_used=$(echo $mem_info | awk '{print $3}')
    mem_usage=$((mem_used * 100 / mem_total))
    load_avg=$(cat /proc/loadavg | awk '{print $1}')
    printf "%-20s CPU: %6s%% MEM: %6s%% LOAD: %6s
" "$timestamp" "$cpu_usage" "$mem_usage" "$load_avg"
    sleep 5
  done
}

comprehensive_kernel_tuning

8. Kernel Fault Diagnosis and Debugging

Kernel fault diagnosis is a must‑have skill for ops engineers, requiring mastery of various debugging tools and methods.

#!/bin/bash
# Kernel fault diagnosis script
debug_kernel_issues() {
  echo "=== Kernel Fault Diagnosis ==="
  echo "Kernel version:"
  uname -a
  echo -e "
Kernel error logs:"
  dmesg | grep -i "error\|panic\|oops\|bug" | tail -10
  echo -e "
System load analysis:"
  uptime
  cat /proc/loadavg
  echo -e "
Memory usage analysis:"
  free -h
  cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable"
  echo -e "
Process status analysis:"
  ps aux | head -10
}

analyze_performance_bottlenecks() {
  echo "=== Performance Bottleneck Analysis ==="
  echo "Top CPU usage:"
  ps -eo pid,user,pcpu,comm --sort=-pcpu | head -11
  echo -e "
Top memory usage:"
  ps -eo pid,user,pmem,comm --sort=-pmem | head -11
  echo -e "
I/O statistics:"
  iostat -x 1 1 | tail -n +4
  echo -e "
Network connection statistics:"
  ss -s
}

debug_kernel_issues
analyze_performance_bottlenecks

9. Conclusion

The Linux kernel is a complex and powerful core of the operating system. By deeply understanding its process management, memory management, filesystem, and network subsystems, ops engineers can more effectively tune systems and troubleshoot faults.

The provided practical scripts and tuning methods help operators quickly locate issues and improve performance. As the kernel evolves, continuous learning is required to keep up with technological advances.