What to Do When a Server Disk Is Full? A Complete Linux Disk Troubleshooting Guide

Background and Problem

Disk space exhaustion is a frequent failure mode on Linux servers. When a filesystem is full, applications cannot write logs, create new files, or flush data; databases cannot write binary logs; SSH sessions may fail to start new commands. Errors appear as “No space left on device”, stopped logging, or service connection failures.

1. Disk Space Fundamentals

1.1 Key Concepts

Filesystem space : The usable space reported by df. This is what applications consume directly.

Logical volume space (LVM) : In LVM setups the logical volume may be smaller than the underlying physical volume, or the filesystem may not have been resized after the LV grew.

Physical disk space : The raw partition size, often tied to RAID layout.

Inode count : Limits the number of files. When inodes are exhausted, new files cannot be created even if free bytes remain.

Block size : The minimum allocation unit (commonly 4 KB). Many tiny files can make usage appear higher than the actual data size.

1.2 Viewing Disk Space

# Show space usage of all filesystems
df -h

# Show inode usage
df -i

# Sort by inode and space usage
df -i | sort -hr

du -sh /var/*
du -h --max-depth=1 /var

# Find large files (>100 MB)
find / -type f -size +100M -exec ls -lh {} \;

1.3 Interpreting df -h Output

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           3.9G     0  3.9G   0% /dev/shm
/dev/sda1       100G   45G   55G  45% /
/dev/mapper/vg-root 200G 180G   20G  90% /data

When Use% exceeds 90 % a warning is needed; above 95 % immediate action is required. Avail shows the total space available to all users.

2. Quickly Locate Space Consumers

2.1 Layered Investigation

# Layer 1: Identify which mount point is full
df -h

# Layer 2: Find the directory consuming the most space
du -sh /*
du -h --max-depth=1 / | sort -hr | head -10

# Layer 3: Drill into the biggest directory
du -h --max-depth=1 /var
du -h --max-depth=1 /var/log

2.2 Find Large Files Quickly

# Files larger than 500 MB
find / -type f -size +500M -exec ls -lh {} \; 2>/dev/null

# Large files in a specific directory (e.g., /var)
find /var -type f -size +100M -exec ls -lh {} \; 2>/dev/null

# Sort directory contents by size
du -ah /var | sort -rh | head -20

2.3 Detect Deleted Files Still Held Open

Deleted files that remain open by a process keep their space allocated.

# Show deleted but still allocated space
lsof +L1

# List all open files marked deleted
lsof | grep deleted

# Inspect a specific process
ps aux | grep <em>process_name</em>
ls -la /proc/<em>PID</em>/fd | grep deleted

Log rotation often leaves the original process writing to the old file descriptor, causing hidden space consumption.

2.4 Resolving Open‑Deleted Files

# Reopen Nginx logs (example for a process that supports log reopening)
nginx -s reopen

# Or send SIGUSR1 to the master process
kill -USR1 $(cat /var/run/nginx.pid)

# Verify no remaining deleted files
lsof +L1 | grep -v "memfd"

3. Log File Investigation and Cleanup

3.1 Log Space Analysis

# Total size of /var/log
du -sh /var/log

# Size of each subdirectory (sorted)
du -h /var/log | sort -rh | head -20

# List individual log files by size
ls -lhS /var/log/*.log

# Count lines in each log file
wc -l /var/log/*.log

3.2 Common Log Files (illustrative list)

/var/log/messages – system kernel logs (usually small)

/var/log/secure – security/authentication logs (can grow large under SSH brute‑force attacks)

/var/log/nginx/access.log – Nginx access logs (can become very large)

/var/log/nginx/error.log – Nginx error logs (usually small unless persistent errors)

/var/log/mysql/error.log – MySQL error log (usually small)

/var/log/httpd/ or /var/log/apache2/ – Apache logs (potentially large)

3.3 Safe Log Cleanup

Never delete a log file that a process is still writing to; truncate it instead.

# For processes that can reopen logs (e.g., Nginx)
nginx -s reopen
# Or send the signal directly
kill -USR1 $(cat /var/run/nginx.pid)
# Then delete the old rotated file
rm /var/log/nginx/access.log.1

# For processes that cannot reopen logs, truncate the file
truncate -s 0 /var/log/nginx/access.log
# Alternative syntax
: > /var/log/nginx/access.log

3.4 Logrotate Configuration Check

# View global logrotate config
cat /etc/logrotate.conf
cat /etc/logrotate.d/nginx

# Example snippet for Nginx logs
/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 nginx nginx
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

4. Database File Space Management

4.1 MySQL/MariaDB Space Analysis

# List database sizes
mysql -e "SELECT table_schema AS 'Database', ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS 'Size (MB)' FROM information_schema.tables GROUP BY table_schema;"

# List top 20 tables by size
mysql -e "SELECT table_schema, table_name, ROUND(data_length / 1024 / 1024, 2) AS data_mb, ROUND(index_length / 1024 / 1024, 2) AS index_mb FROM information_schema.tables ORDER BY data_length DESC LIMIT 20;"

# Show binary logs
mysql -e "SHOW BINARY LOGS;"

# Show InnoDB file sizes
ls -lh /var/lib/mysql/*.ibd

4.2 MySQL Binlog Cleanup

# Show binlog retention settings
mysql -e "SHOW VARIABLES LIKE 'expire_logs_days';"
mysql -e "SHOW VARIABLES LIKE 'max_binlog_size';"

# Purge binlogs up to a specific file
mysql -e "PURGE BINARY LOGS TO 'mysql-bin.000123';"

# Purge binlogs older than a date
mysql -e "PURGE BINARY LOGS BEFORE '2026-04-01 00:00:00';"

Important: Ensure all replicas have received the binlogs before purging, otherwise replication will break.

# Automatic binlog cleanup in my.cnf
expire_logs_days = 7
max_binlog_size = 100M

4.3 PostgreSQL Space Analysis

# List databases by size
psql -c "SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) FROM pg_database ORDER BY pg_database_size(pg_database.datname) DESC;"

# List top 20 tables by size
psql -c "SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC LIMIT 20;"

# Show WAL directory size
ls -lh $PGDATA/pg_wal/
du -sh $PGDATA/pg_wal/

4.4 PostgreSQL WAL Cleanup

# Check WAL retention settings
psql -c "SHOW wal_keep_size;"
psql -c "SHOW max_wal_size;"

# Force a checkpoint and clean WAL
psql -c "CHECKPOINT;"

# Adjust retention in postgresql.conf
wal_keep_size = 1GB
max_wal_size = 2GB

4.5 MongoDB Space Analysis

# List databases and sizes
mongosh --quiet --eval "db.adminCommand('listDatabases')" | grep -E "name|sizeOnDisk"

# List collection sizes
mongosh --quiet --eval "db.getCollectionNames().forEach(function(c){ var s=db[c].stats(); print(c + ': ' + (s.size/1024/1024).toFixed(2) + ' MB'); });"

# Show oplog size
mongosh --quiet --eval "db.oplog.rs.stats().maxSize / 1024 / 1024"

5. Temporary Files and Cache Cleanup

5.1 /tmp Cleanup

# Show /tmp size
du -sh /tmp

# List biggest files in /tmp
find /tmp -type f -exec ls -lh {} \; | sort -k5 -rh | head -20

# Find files not accessed for 30 days
find /tmp -atime +30 -type f -ls

5.2 System Cache Cleanup

# Show memory and cache usage
free -h

# Flush filesystem buffers
sync

# Drop page cache (use with caution in production)
echo 3 > /proc/sys/vm/drop_caches

Warning: Dropping caches can cause performance jitter; use only for troubleshooting.

5.3 Package Manager Caches

# Yarn cache
yarn cache dir
yarn cache clean

# NPM cache
npm cache ls
npm cache clean --force

# Maven cache
du -sh ~/.m2/repository
rm -rf ~/.m2/repository/*

# PIP cache
du -sh ~/.cache/pip
rm -rf ~/.cache/pip

5.4 Docker Resource Cleanup

# Show Docker disk usage
docker system df
docker system df -v

# Prune unused resources
docker system prune -a
docker container prune
docker image prune -a
docker volume prune
docker network prune

# Prune resources older than 7 days
docker system prune --filter "until=168h"

5.5 Old Kernels and Unused Packages

# List installed kernels (Debian/Ubuntu)
dpkg --list | grep linux-image

# Remove old kernels (Debian/Ubuntu)
apt-get autoremove -y
purge-old-kernels --keep 2

# Remove old kernels (RHEL/CentOS)
dnf remove $(dnf repoquery --installonly --latest-limit=-2 -q)
package-cleanup --oldkernels --count=2

6. Inode Exhaustion

6.1 Inode Basics

# Show inode usage
df -i

# Show inode usage for a specific filesystem
df -i /

# Find directories with most inode consumption
find / -xdev -printf '%h
' | sort | uniq -c | sort -rn | head -10

6.2 Small‑File Inode Drain

# Count files in a directory (example: mail queue)
find /var/spool -type f | wc -l

# Show file count per subdirectory under /var
for dir in /var/*; do echo -n "$dir: "; ls -1A "$dir" 2>/dev/null | wc -l; done

# Find directories with the highest file count
find / -xdev -type d -exec sh -c 'echo "$(ls -1A "$1" 2>/dev/null | wc -l) $1"' _ {} \; | sort -rn | head -10

6.3 Resolving Inode Exhaustion

If the mail queue is the cause:

# List mail queue files
ls -lh /var/spool/mail/

# Delete old mail after verification (e.g., older than 30 days)
find /var/spool/mail -type f -mtime +30 -delete

When creating a new filesystem, inode density can be set, e.g.:

# One inode per 8 KB
mkfs.ext4 -i 8192 /dev/sdb1

7. LVM‑Based Issues

7.1 LVM Overview

PV (Physical Volume) – underlying partition or whole disk.

VG (Volume Group) – collection of PVs.

LV (Logical Volume) – allocated from a VG.

7.2 LVM Space Investigation

# Show VG usage
vgdisplay

# Show LV details
lvdisplay
lvs -o lv_name,lv_size,data_percent,metadata_percent

# Show PV usage
pvs
pvdisplay

7.3 Extending a Logical Volume

# Extend LV by 10 GB
lvextend -L +10G /dev/mapper/vg-root

# Resize ext4 filesystem
resize2fs /dev/mapper/vg-root

# Resize XFS filesystem
xfs_growfs /data

7.4 Adding a New Physical Volume

# Create a new partition or disk, then:
pvcreate /dev/sdb1
vgextend vg_name /dev/sdb1

# Extend LV and filesystem as shown above

7.5 Reducing a Logical Volume (Risky)

Only after unmounting and checking the filesystem:

# Unmount
umount /data

# Check filesystem
e2fsck -f /dev/mapper/vg-data

# Shrink filesystem
resize2fs /dev/mapper/vg-data 50G

# Reduce LV
lvreduce -L 50G /dev/mapper/vg-data

# Remount
mount /dev/mapper/vg-data /data

8. Partition Table and Filesystem Repair

8.1 Check Filesystem Errors

# Unmount before checking
umount /data

e2fsck -f /dev/mapper/vg-data   # Force check

e2fsck -p /dev/mapper/vg-data   # Auto‑repair mode

Note: Do not run e2fsck on a mounted filesystem (unless using read‑only mode). In production, use fsck -n to preview issues.

8.2 Repair Corrupted Inodes

# Show inode statistics
dumpe2fs /dev/sda1 | grep -E "Inode size|Inode count|Inode blocks per group|Inodes per group"

# Scan for bad blocks
badblocks /dev/sda1

8.3 Repair LVM Configuration

# Scan physical volumes
pvscan

# Activate all volume groups
vgchange -ay

# List all logical volumes
lvs -a

9. Emergency Handling Procedure

Step 1 – Log Into the Server

# Use SSH key if password login fails
ssh -i ~/.ssh/id_rsa user@server

# If the session is alive but commands fail, try a minimal command
df -h /

Step 2 – Locate the Most Critical Consumer

# Show overall usage
df -h

# Find large files (>100 M)
find / -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null

# Find deleted but still open files
lsof +L1

Step 3 – Immediate Cleanup

# Remove temporary files
rm -rf /tmp/*
rm -rf /var/tmp/*

# Truncate critical logs safely
: > /var/log/messages
: > /var/log/secure

# Clean package manager caches
yum clean all
apt-get clean

# Prune Docker (use with caution)
docker system prune -f

Step 4 – Notify Services

# Restart logging services
systemctl restart rsyslog
systemctl restart systemd-journald

# Restart affected applications
systemctl restart nginx
systemctl restart myapp

10. Preventive Measures

10.1 Disk Space Monitoring and Alerts

# Prometheus alert example
groups:
- name: disk
  rules:
  - alert: DiskSpaceUsageHigh
    expr: (node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_avail_bytes{mountpoint="/"}) / node_filesystem_size_bytes{mountpoint="/"} * 100 > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Disk space usage is above 85%"
  - alert: DiskSpaceUsageCritical
    expr: (node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_avail_bytes{mountpoint="/"}) / node_filesystem_size_bytes{mountpoint="/"} * 100 > 95
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Disk space usage is above 95%, immediate action required"

10.2 Log Space Limits

# Ensure logrotate is configured
logrotate /etc/logrotate.conf

# Example Nginx log buffer limit
access_log /var/log/nginx/access.log combined buffer=32k flush=5s;

10.3 Separate Partitions Strategy

Place critical directories on dedicated partitions to prevent a single directory from filling the root filesystem.

/var/log

/tmp (or tmpfs)

/home

/data or /opt for application data

# Example /etc/fstab entries
tmpfs /tmp tmpfs defaults,noexec,nosuid,size=2G 0 0
tmpfs /var/tmp tmpfs defaults,noexec,nosuid,size=1G 0 0

10.4 Dedicated Database Partitions

# Create a dedicated LV for MySQL
lvcreate -L 100G -n mysql vg
mkfs.xfs /dev/mapper/vg-mysql
mount /dev/mapper/vg-mysql /var/lib/mysql

10.5 Regular Check Scripts

# /etc/cron.daily/disk-check
#!/bin/bash
THRESHOLD=85
df -h | grep -vE '^Filesystem|tmpfs|cdrom' | awk -v t=$THRESHOLD '{if ($5+0 >= t) print $1 " " $5 " is above threshold