How to Optimize Linux Server Performance: Monitoring, Tuning, and Resource Management

Linux servers often encounter performance bottlenecks under high load, such as excessive CPU usage, memory pressure, disk I/O saturation, or network latency. Optimizing both hardware and software layers can improve stability and prevent service interruptions.

Software‑level Optimization Areas

Memory tuning

Disk I/O tuning

CPU scheduling and affinity

Process and thread management

Network stack parameters

Monitoring Tools and Their Key Metrics

1. vmstat

Provides a snapshot of memory, process, I/O, and CPU usage. Important fields include:

procs : r (running/ready threads) and b (blocked threads)

memory : swpd, free, buff, cache

swap : si (swap‑in) and so (swap‑out)

io : bi (blocks read) and bo (blocks written)

system : in (interrupts) and cs (context switches)

cpu : us, sy, id, wa, st

vmstat 1 5

2. mpstat

Focuses on per‑CPU statistics. Key columns are %usr, %nice, %sys, %iowait, %irq, %soft, %steal, %guest, and %idle. High %iowait indicates I/O bottlenecks.

mpstat -P ALL 1

3. iostat

Shows device‑level I/O performance, including tps, rkB/s, wkB/s, await, and %util. Values above 80‑90% suggest the device is saturated.

iostat -xz 1

4. sar

Collects historical performance data for later analysis. Output can be redirected to a file for post‑mortem review.

sar -u 1 10 > /tmp/cpu.log

5. Process‑level Tools

top/htop : Real‑time view of CPU, memory, and process states.

ps : Static snapshot of process attributes.

pidstat : Per‑process CPU and memory usage over time.

strace : Traces system calls of a specific process.

Resource‑Limit Mechanisms

ulimit

Limits per‑process resources such as core file size, virtual memory, open files, and maximum user processes. Example to view current limits: ulimit -a To set a limit for the current shell:

ulimit -n 2000   # Max open files
ulimit -u 100    # Max user processes

PAM Limits

Configure /etc/security/limits.conf with entries like:

tom    hard    nproc    300

cgroups

Use control groups to restrict CPU, memory, and I/O for a set of processes or containers. Example to create a memory‑limited cgroup:

mkdir -p /sys/fs/cgroup/memory/limited
echo 500M > /sys/fs/cgroup/memory/limited/memory.limit_in_bytes
echo $$ > /sys/fs/cgroup/memory/limited/tasks

CPU Optimization

CPU affinity with taskset -c 0,1 -p <pid> binds a process to specific cores.

Scheduler selection (CFQ, deadline, noop) can be changed via kernel boot parameters or /sys/block/<dev>/queue/scheduler.

Time‑slice adjustment requires kernel recompilation and is rarely needed.

Migration cost tuning also needs kernel patches and may increase context‑switch overhead.

Memory Optimization

Disable swap when sufficient RAM is available: swapoff -a.

Set vm.swappiness=0 to prefer RAM over swap.

Adjust vm.dirty_bytes (e.g., 1G) to control dirty‑page flushing.

Use tmpfs for fast, memory‑backed storage:

mkdir /mnt/cache
mount -t tmpfs -o size=2G tmpfs /mnt/cache

Enable huge pages for memory‑intensive workloads:

echo 2000 > /proc/sys/vm/nr_hugepages
mount -t hugetlbfs nodev /mnt/huge

Manually flush dirty data with sync or tune vm.dirty_expire_centisecs and vm.dirty_writeback_centisecs.

Clear caches when necessary (e.g., after a crash):

sync && echo 3 > /proc/sys/vm/drop_caches

Disk Optimization

Mount with noatime to avoid updating access timestamps.

Choose filesystem based on workload: ext4 for many small files, XFS for large sequential writes.

Adjust inode count during filesystem creation if the workload contains many tiny files: mkfs.ext4 -N 2000000 /dev/sdb1 Select an appropriate I/O scheduler (CFQ, deadline, noop) via /sys/block/<dev>/queue/scheduler or kernel boot options.

Set read‑ahead size to improve sequential reads:

echo 256 > /sys/block/vda/queue/read_ahead_kb

Defragment ext4 with e4defrag or XFS with xfs_fsr.

Network Optimization

Increase TCP backlog and connection limits:

echo 10000 > /proc/sys/net/ipv4/tcp_max_syn_backlog
echo 10000 > /proc/sys/net/core/somaxconn

Enable TCP Fast Open (value 2 enables both client and server): echo 2 > /proc/sys/net/ipv4/tcp_fastopen Choose a congestion control algorithm (e.g., cubic):

echo cubic > /proc/sys/net/ipv4/tcp_congestion_control
# Persist in /etc/sysctl.conf
net.ipv4.tcp_congestion_control=cubic

Raise socket buffer limits:

echo 16777216 > /proc/sys/net/core/wmem_max
echo 16777216 > /proc/sys/net/core/rmem_max

Disable IPv6 if not used to reduce overhead:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

Enable TCP window scaling and adjust receive/send memory:

net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 131072 6291456
net.ipv4.tcp_wmem = 4096 16384 4194304

Increase the listen queue size (usually twice the max connections):

sysctl -w net.core.somaxconn=2048

Practical Usage Summary

Use vmstat or iostat for real‑time resource checks.

Use sar for historical performance analysis.

Apply ulimit or PAM limits for per‑user or per‑process restrictions.

Leverage cgroups for container‑level isolation.

Fine‑tune CPU affinity, scheduler, and migration costs for compute‑heavy workloads.

Optimize memory usage by disabling swap, adjusting swappiness, and using huge pages where appropriate.

Improve disk throughput with proper mount options, filesystem choice, I/O scheduler, and read‑ahead settings.

Boost network performance by increasing backlog, enabling fast open, selecting suitable congestion control, enlarging socket buffers, and disabling unused protocols.