Mastering Linux Processes: From Basics to Advanced Monitoring and Management

1. Introduction to Processes

A program is a static collection of code and data; when it is executed, the operating system creates a process , which is a dynamic instance that receives memory, an identity (PID), and permissions.

Running a program therefore means running a process.

1.1 Program vs. Process

Program: static binary file (e.g., /bin/ls), can exist indefinitely.

Process: dynamic execution entity with a lifecycle; it disappears when the program terminates.

1.2 Process Lifecycle

A parent process can fork a child process using fork(). The child inherits the parent’s attributes. The parent may wait for the child, the child may exit, or become a zombie if the parent terminates first.

Each process has a parent PID (PPID) and its own PID.

2. Monitoring Process State

2.1 Static View with ps

The ps -aux command shows user, PID, CPU%, MEM%, state, command, etc. Important columns:

USER : owner of the process.

PID : process identifier.

%CPU , %MEM : resource usage.

STAT : current state (see section 2.2).

2.2 Understanding STAT Codes

Common state letters: R: running. S: interruptible sleep. D: uninterruptible sleep (usually I/O). T: stopped. Z: zombie.

Modifiers such as + indicate foreground processes, s marks session leaders, etc.

2.3 Dynamic View with top

top

displays real‑time tasks, CPU usage, memory, and load averages. Key sections:

Tasks: total, running, sleeping, stopped, zombie.

CPU line: percentages for user, system, nice, idle, iowait, etc.

Interactive keys let you sort, filter, and change refresh intervals (e.g., h for help, 1 for per‑CPU view, b to highlight running processes).

3. System‑Level Tools

3.1 Memory Overview – free

Use free -h for human‑readable output of total, used, and free memory.

3.2 Virtual Memory Statistics – vmstat

Command syntax: vmstat [delay] [count]. Columns include processes, memory, swap, I/O, system calls, and CPU usage.

3.3 Disk I/O – iostat

Shows read/write rates per device, e.g., iostat 1 10.

3.4 Comprehensive Monitoring – dstat

Combine multiple metrics: dstat -c -d -m -n -p -r and use --top-cpu, --top-io, --top-mem to highlight the most resource‑hungry processes.

3.5 Network Traffic – iftop

Displays real‑time bandwidth usage per connection.

4. Managing Processes

4.1 Sending Signals with kill

List supported signals: kill -l. Commonly used signals: SIGHUP (1): reload configuration. SIGKILL (9): force termination. SIGTERM (15): graceful termination (default for kill).

Examples:

# Reload vsftpd configuration
kill -1 9160
# Graceful stop
kill 9160
# Force stop
kill -9 9160

4.2 Killing by Name – killall / pkill

These commands combine ps and grep to terminate processes matching a name, e.g., pkill nginx or killall nginx.

4.3 Background Jobs

Traditional shell job control:

# Run in background
sleep 3000 &
# Suspend foreground job
Ctrl+Z
# List jobs
jobs
# Move stopped job to background
bg %2
# Bring job to foreground
fg %1
# Terminate job by job ID
kill %1

More robust solution: screen (or tmux) to detach sessions.

# Install screen
yum install -y screen
# Start a named session
screen -S wget_mysql
# Run a long‑running command inside
wget https://downloads.mysql.com/.../mysql-5.7.30.tar.gz
# Detach without stopping the job
Ctrl+a d
# List sessions
screen -list
# Reattach
screen -r wget_mysql

4.4 Detaching Completely – nohup

nohup ping www.baidu.com >/dev/null 2>&1 &

5. Process Priority (Advanced)

Priority is expressed via the nice value: lower numbers mean higher priority ( -20 highest, +19 lowest).

5.1 Viewing Priority

# top shows NI (nice) and PR (priority)
ps axo pid,command,nice | grep sshd

5.2 Setting Priority at Launch

nice -n -5 vim &

5.3 Changing Priority of a Running Process

renice -n -20 98002   # Change sshd master process to highest priority

6. Load Average and Performance Analysis

The three numbers shown by uptime or top are the average number of runnable or uninterruptible processes over the last 1, 5, and 15 minutes. They are not direct CPU usage percentages.

Interpretation depends on CPU count:

On a 4‑core system, a load of 2 means 50 % of CPU capacity is idle.

On a 2‑core system, a load of 2 means the CPUs are fully utilized.

On a single‑core system, a load of 2 indicates contention.

When the 1‑minute load exceeds the number of CPUs by ~30 % (e.g., load > 0.7 × CPU count), it is a good time to investigate.

6.1 Tools for Root‑Cause Analysis

watch -d uptime

– watch load changes. mpstat -P ALL 5 – per‑CPU usage and iowait. pidstat -u 5 1 – per‑process CPU usage. stress – generate CPU, I/O, or mixed load for testing.

6.2 Example Scenarios

CPU‑bound load : stress --cpu 1 drives load to 100 % CPU, visible as high %usr in mpstat and a matching rise in load average.

I/O‑bound load : stress --io 1 raises %iowait while %usr stays low; load average still climbs because processes are in uninterruptible sleep.

Many processes on few CPUs : stress -c 4 on a single‑core machine produces high load and high %wait in pidstat, showing contention.

7. Security‑Related System Settings (Brief)

Common firewall tools: iptables rules, disabling firewalld with systemctl disable firewalld. SELinux configuration resides in /etc/selinux/config; it can be disabled with SELINUX=disabled or set to permissive temporarily via setenforce 0.