Essential Ops Playbook: Real‑World Linux Tuning & Incident Diagnosis

Introduction: The 3 AM Call

During a Double‑11 sale, a frantic phone woke me at 3 AM: the core service CPU spiked to 95 % and users saw page loads over 10 seconds. Using top I saw a process hogging CPU, but the real culprit was deeper.

Key insight: System tuning isn’t about tweaking a few parameters; it requires understanding Linux kernel internals, just as a doctor must know human anatomy.

Why Tuning Is an Ops "Inner Skill"

In the cloud‑native era many assume Kubernetes auto‑scaling makes tuning irrelevant, yet:

Cost pressure: Proper tuning can cut cloud costs by 30‑50 %.

User experience: Reducing response time from 500 ms to 100 ms can boost conversion by over 15 %.

Failure prevention: 80 % of production incidents stem from resource bottlenecks.

Even high‑end hardware can under‑perform with default settings—like driving a Ferrari in ECO mode.

Typical Scenarios

Scenario 1: E‑commerce flash sale A system that normally handles 100 k daily active users suddenly faces 1 M. Missing connection‑limit tuning triggers Too many open files and 500 errors.

Scenario 2: Log surge A sudden slowdown in log ingestion was caused by an improper vm.dirty_ratio setting, leading to 40 % IO wait.

Scenario 3: Database slow query Even with application‑level optimizations, a default swappiness of 60 caused frequent swapping and severe query latency jitter.

Practical Experience: My Three‑Step Tuning "Sword"

Step 1 – Build a Performance Baseline

Never adjust parameters blindly. Record baseline metrics:

# 1. Record system baseline data
sar -u 1 10   # CPU usage
sar -r 1 10   # Memory usage
sar -d 1 10   # Disk I/O
sar -n DEV 1 10   # Network traffic

# 2. Use flame graphs to find hotspots
perf record -F 99 -p <pid> -g -- sleep 30
perf script | ./flamegraph.pl > flame.svg

Real case: An app appeared slow; flame‑graph analysis showed 80 % of time spent in a JSON serializer. Switching libraries gave a 5× speedup without any hardware upgrade.

Step 2 – Layered Optimization

Think of the system as a barrel; the shortest board limits capacity. Optimize in order:

1. Kernel parameters (quick wins)

# /etc/sysctl.conf – production template
# Network layer
net.core.somaxconn = 65535   # increase listen queue
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_tw_reuse = 1      # fast TIME_WAIT reuse
net.ipv4.tcp_fin_timeout = 30

# Memory management
vm.swappiness = 10            # reduce swap tendency
vm.dirty_ratio = 15           # control dirty page ratio
vm.dirty_background_ratio = 5

# Filesystem
fs.file-max = 2097152        # raise file descriptor limit
fs.inotify.max_user_watches = 524288

Pitfall: Enabling tcp_tw_reuse behind a NAT gateway can cause connection‑reuse errors.

2. Application layer (deep optimization)

# Monitor process resource usage
pidstat -p <pid> 1

# Analyze system‑call bottlenecks
strace -c -p <pid>

# Check thread contention
perf lock record -p <pid> -- sleep 10
perf lock report

Example: A Java service showed 30 % CPU but high latency. jstack revealed many threads blocked on a synchronized hotspot. Replacing it with ConcurrentHashMap dropped CPU to 10 % and doubled throughput.

Step 3 – Build a Fast‑Diagnosis Checklist

When woken at 3 AM, I run a 30‑second script:

#!/bin/bash
# quick-diag.sh – emergency diagnosis

echo "=== CPU ==="
uptime
mpstat -P ALL 1 3

echo "=== Memory ==="
free -h
slabtop -o | head -20

echo "=== Disk IO ==="
iostat -xz 1 3

echo "=== Network ==="
ss -s
iftop -t -s 3

echo "=== Process ==="
ps aux --sort=-%cpu | head -10
ps aux --sort=-%mem | head -10

echo "=== Logs ==="
dmesg -T | tail -20
journalctl -p err -n 20

This script saved me countless times; once it revealed an iostat spike caused by a log‑archiving cron compressing old logs and saturating IO. Adjusting the cron timing resolved the issue.

Personal Summary: Best Practices for Veteran Ops

1. Stay Skeptical – Let Data Speak

Never accept “maybe” or “should” without data; monitoring never lies.

Basic monitoring: Prometheus + Grafana

APM tracing: Jaeger or SkyWalking

Log analysis: ELK or Loki

Alerting: AlertManager with alert‑convergence

2. Avoid Over‑Optimization

Spending two weeks to shave a daily batch job from 10 min to 8 min is a classic “over‑kill”. Apply the 80/20 rule: 80 % of performance problems come from 20 % of code/configuration.

3. Document Every Tuning

## 2024-10-20 Database server memory tuning

**Problem**: OOM during nightly batch
**Cause**: innodb_buffer_pool_size too large, starving OS
**Adjustment**: Reduce from 16 GB to 12 GB
**Effect**: OOM disappeared, query performance unchanged
**Lesson**: Bigger isn’t always better; leave room for OS and other processes

4. Automate Repetitive Work

# check-system-health.sh – hourly health check
CPU_THRESHOLD=80
MEM_THRESHOLD=90
DISK_THRESHOLD=85

cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
if (( $(echo "$cpu_usage > $CPU_THRESHOLD" | bc -l) )); then
    curl -X POST "https://your-alert-webhook" -d "CPU usage high: ${cpu_usage}%"
fi
# ... omitted memory & disk checks

Combine with Ansible or SaltStack for mass configuration rollout and rollback; I once pushed tuning changes to 200 servers in five minutes.

Trends & Extensions: Where Tuning Is Heading

1. eBPF – The Next‑Gen Observability Layer

Traditional tools like strace or tcpdump add noticeable overhead. eBPF enables low‑impact, in‑production tracing.

Example with bpftrace:

# Trace TCP connection attempts
bpftrace -e 'tracepoint:syscalls:sys_enter_connect { printf("%s connecting to %s
", comm, args->uservaddr); }'

2. AIOps – Let AI Assist Tuning

Some platforms now use machine learning to predict bottlenecks, e.g., forecasting a traffic spike on Wednesday night and auto‑scaling ahead of time.

3. New Challenges in Cloud‑Native Environments

Cgroup limits: CPU/memory limits directly affect pod stability.

Network performance: Choice of CNI plugin (Calico vs. Cilium) impacts latency.

Storage I/O: Cloud‑disk IOPS caps must be planned.

When running databases in Kubernetes, set guaranteed QoS to avoid eviction.

Conclusion: Take Action Today

System tuning is an art of practice—no silver bullet, only continuous trial, error, and knowledge accumulation.

Build a monitoring stack today, even on a single test machine.

Write a post‑mortem after each incident to grow your knowledge base.

Join technical communities (e.g., "Efficient Ops", "Ops Help") to learn from real‑world cases.