Server Disk Full Alert: From Root Cause Analysis to Safe Automated Cleanup

1. Protect the Scene: Confirm Capacity, Inode, or Storage Failure

Start by checking both byte space and inodes simultaneously. A surge of small files in a directory can exhaust inodes even when df -h still shows available space.

df -hT
df -i
findmnt -T <mount_point>

Collect a read-only scene snapshot to a directory with spare space and controlled permissions. The script records timestamp, filesystem usage, inode usage, mount details, and recent kernel logs.

#!/usr/bin/env bash
set -euo pipefail
OUTPUT_DIR="<scene_dir>"
STAMP="$(date +%Y%m%d_%H%M%S)"
mkdir -p "$OUTPUT_DIR"
{
  date --iso-8601=seconds
  df -hT
  df -i
  findmnt -R <mount_point>
  journalctl -k --since '30 min ago' --no-pager | tail -n 200
} > "$OUTPUT_DIR/disk-full-$STAMP.txt"

Check kernel logs and mount options for I/O errors, read-only remounts, or filesystem errors. If found, treat as a storage fault: protect data, stop unrelated writes, and follow storage incident procedures.

journalctl -k --since '2 hours ago' --no-pager \
  | rg -i 'I/O error|read-only|remount|ext4|xfs|nvme|blk_update_request'
findmnt -T <mount_point> -o TARGET,SOURCE,FSTYPE,OPTIONS

Identify the largest top-level directories without crossing mount points using du -xhd1. Verify mount boundaries for /var, /tmp, and Docker paths.

sudo du -xhd1 <mount_point> 2>/dev/null | sort -h
findmnt -T /
findmnt -T /var
findmnt -T /tmp
findmnt -T /var/lib/docker

Check failed services and recent logs. If core write services have failed, drain traffic via existing load-balancer processes before proceeding.

systemctl --failed
systemctl status <service_name> --no-pager
journalctl -u <service_name> --since '30 min ago' --no-pager | tail -n 120

2. Evidence-Based Classification of Space Consumers

Find the largest regular files first, limiting I/O by targeting suspect directories.

sudo find <mount_point> -xdev -type f -size +1G -printf '%s %p
' \
  2>/dev/null | sort -nr | head -n 50

Use recent modification time as a growth clue (not a deletion license). Correlate with deployment records, cron jobs, and application logs.

sudo find <mount_point> -xdev -type f -mtime -1 -size +100M \
  -printf '%TY-%Tm-%Td %TH:%TM %s %p
' 2>/dev/null | sort -k3,3nr | head -n 80

Examine /var and business log directories at depth 2. Logs may live in working directories, cache directories, or container stdout files.

sudo du -xhd2 /var 2>/dev/null | sort -h | tail -n 80
sudo du -xhd2 <log_dir> 2>/dev/null | sort -h | tail -n 80

When du shows less usage than df, check for deleted-but-held files with lsof +L1. Record process, fd, and SIZE/OFF before acting. Prefer service-safe reopen or graceful restart over blind truncation.

sudo lsof -nP +L1
sudo lsof -nP +L1 | awk 'NR==1 || $7 > 1073741824'

Nginx supports log reopen via reload; other services have different signal semantics.

sudo nginx -t
sudo systemctl reload nginx
sudo lsof -nP +L1 | rg 'nginx|<service_name>' || true

Audit logrotate policies and timers. Use -d for dry-run before deploying rules.

sudo logrotate -d /etc/logrotate.conf
sudo logrotate -d /etc/logrotate.d/<service_name>
systemctl status logrotate.timer --no-pager

Example Nginx logrotate config (adjust user, path, retention, and postrotate to match actual service):

/var/log/nginx/*.log {
    daily
    rotate <retention_days>
    missingok
    notifempty
    compress
    delaycompress
    dateext
    create 0640 nginx adm
    sharedscripts
    postrotate
        /usr/bin/systemctl reload nginx > /dev/null 2>&1 || true
    endscript
}

Analyze Docker disk usage before cleaning. docker system prune removes stopped containers, images, and build cache — not a first resort.

docker system df -v
docker ps -a --size
docker image ls
docker volume ls

Check container json-file logs. Container IDs must be verified with the business team; error logs may be incident evidence.

sudo find /var/lib/docker/containers -name '*-json.log' -type f \
  -printf '%s %p
' 2>/dev/null | sort -nr | head -n 30

Long-term prevention: configure log rotation in Docker daemon. Modify daemon.json and restart Docker only after backup, drain, and per-node canary.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "<max_log_file_size>",
    "max-file": "<retained_file_count>"
  }
}

sudo cp -a /etc/docker/daemon.json /etc/docker/daemon.json.bak.$(date +%Y%m%d_%H%M%S)
sudo jq empty /etc/docker/daemon.json
sudo systemctl restart docker
docker info --format '{{.LoggingDriver}}'

If approved, truncate only clearly disposable runtime logs after exporting evidence, draining traffic, and confirming path. Verify container and business health immediately after.

sudo truncate -s 0 /var/lib/docker/containers/<container_id>/<container_id>-json.log
df -hT <mount_point>
docker ps
curl --fail --silent --show-error <health_check_url> >/dev/null

3. Investigate Journal, Coredumps, Temp Files, and Inodes

Verify journal occupancy and integrity before vacuuming. Confirm logs are reliably shipped to central platform and meet audit retention. Test on a single low-risk node first; retention days must come from policy, not guesswork.

# Verify occupancy and integrity
journalctl --disk-usage
journalctl --verify

# After confirming remote retention and retention period
sudo journalctl --vacuum-time=<retention_days>d
journalctl --disk-usage

List coredumps before deciding retention; they may contain sensitive data and are valuable for developer debugging.

coredumpctl list --no-pager
sudo du -sh /var/lib/systemd/coredump 2>/dev/null || true

Configure systemd-coredump limits to prevent future growth (does not reclaim existing space). Check man coredump.conf for version-specific fields.

# /etc/systemd/coredump.conf.d/50-retention.conf
[Coredump]
MaxUse=<max_usage>
KeepFree=<reserved_space>

Generate candidate list for temp files; presence in /tmp does not guarantee no long-running task uses them.

sudo find /tmp /var/tmp -xdev -type f -mtime +<retention_days> \
  -printf '%TY-%Tm-%Td %TH:%TM %s %p
' 2>/dev/null | head -n 200

Save candidates as NUL-delimited list, sample-review, and ensure the list file resides on a writable mount.

sudo find /tmp /var/tmp -xdev -type f -mtime +<retention_days> -print0 \
  > <cleanup_list_file>
sudo xargs -0 -a <cleanup_list_file> -r ls -ld --

After confirming no long tasks, backed-up list, and single-node validation, delete using the list.

sudo xargs -0 -a <cleanup_list_file> -r rm -f --

For inode alerts, find directories with the most small files. Restrict scope to known business directories to avoid unbounded root scans.

sudo find <mount_point> -xdev -type f -printf '%h
' 2>/dev/null \
  | sort | uniq -c | sort -nr | head -n 50

On Debian/Ubuntu, simulate APT cache cleanup first; RHEL derivatives must use dnf / yum. Confirm no offline install, rollback, or air-gap dependencies before actual clean.

# Dry-run and verify cache size
sudo apt-get -s clean
sudo du -sh /var/cache/apt 2>/dev/null || true

# After confirmation, execute and re-check root partition
sudo apt-get clean
df -hT /

4. Automated Cleanup Must Have Guardrails

Proper automation includes path allowlist, file patterns, minimum age, dry-run, audit logging, and fail-fast. Never use rm -rf <dir>/* as a generic cleanup.

The following script handles only .gz logs older than retention days under a specific log directory. It defaults to dry-run; fails if variables empty, target is root, or directory missing.

#!/usr/bin/env bash
set -euo pipefail

TARGET_DIR="<log_dir>"
RETENTION_DAYS="<retention_days>"
AUDIT_LOG="/var/log/ops-log-cleanup.log"
DRY_RUN="1"

test -n "$TARGET_DIR"
test "$TARGET_DIR" != "/"
test -d "$TARGET_DIR"

find "$TARGET_DIR" -xdev -type f -name '*.gz' -mtime +"$RETENTION_DAYS" -print0 \
  | while IFS= read -r -d '' file; do
    if [ "$DRY_RUN" = "1" ]; then
      printf 'would delete: %q
' "$file"
    else
      sudo rm -f -- "$file"
      printf '%s deleted=%q
' "$(date --iso-8601=seconds)" "$file" \
        | sudo tee -a "$AUDIT_LOG" >/dev/null
    fi
done

Run dry-run first and manually spot-check candidates. High candidate count is not a reason to skip review. sudo /usr/local/sbin/ops-log-cleanup After confirmation, set DRY_RUN=0 in the script, run on a low-risk node, then verify space, service status, and API health.

sudo /usr/local/sbin/ops-log-cleanup
df -hT <mount_point>
systemctl is-active <service_name>

Use a systemd timer for scheduling; avoid backup, compression, and batch-processing peaks.

# /etc/systemd/system/ops-log-cleanup.service
[Unit]
Description=Clean expired compressed logs

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/ops-log-cleanup

# /etc/systemd/system/ops-log-cleanup.timer
[Unit]
Description=Run log cleanup daily

[Timer]
OnCalendar=*-*-* 03:20:00
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now ops-log-cleanup.timer
systemctl list-timers ops-log-cleanup.timer --all
journalctl -u ops-log-cleanup.service -n 80 --no-pager

5. Verification, Monitoring, and Rollback

Prometheus metrics must match actual node exporter output. Alert on both capacity and inode percentages separately.

100 * (
  1 - node_filesystem_avail_bytes{mountpoint="<mount_point>",fstype!~"tmpfs|overlay"}
  / node_filesystem_size_bytes{mountpoint="<mount_point>",fstype!~"tmpfs|overlay"}
)

100 * (
  1 - node_filesystem_files_free{mountpoint="<mount_point>",fstype!~"tmpfs|overlay"}
  / node_filesystem_files{mountpoint="<mount_point>",fstype!~"tmpfs|overlay"}
)

Trend prediction helps catch sustained growth, but absolute available-space thresholds are still needed for burst protection.

predict_linear(
  node_filesystem_avail_bytes{mountpoint="<mount_point>",fstype!~"tmpfs|overlay"}[6h],
  4 * 3600
) < 0

Post-remediation verification: capacity, inodes, mount writability, service status, error logs, and business API. Rapid re-growth indicates root cause not resolved.

df -hT <mount_point>
df -i <mount_point>
findmnt -T <mount_point> -o TARGET,SOURCE,FSTYPE,OPTIONS
systemctl is-active <service_name>
journalctl -u <service_name> --since '10 min ago' --no-pager | tail -n 100
curl --fail --silent --show-error <health_check_url> >/dev/null

Rollback config by restoring backups; disable timer for automation rollback. rm and truncate deletions are irreversible — recover only from validated backups, log platform, or object storage.

sudo systemctl disable --now ops-log-cleanup.timer
sudo systemctl status ops-log-cleanup.timer --no-pager

Docker config rollback: restore backup, validate JSON, restart Docker per node after drain/migration.

sudo cp -a <backup_dir>/daemon.json /etc/docker/daemon.json
sudo jq empty /etc/docker/daemon.json
sudo systemctl restart docker
docker ps

Every incident must leave a record: growth evidence, root cause, freed objects, freed amount, impact scope, verification results, and long-term measures.

6. Turn Alerts from "Full" into Actionable Events

Disk alerts must carry sufficient context: instance, mount point, remaining bytes, inodes, 1-hour delta, recent deployments, owning service. Alert severity should weigh both absolute headroom and projected exhaustion time — a 2 TB data disk at 5% free differs vastly from a 20 GB root disk at 5%.

Do not close the incident immediately after space recovery. Observe at least one full business peak or cleanup cycle: linear growth? journal/container logs rotating per new policy? "no space left on device" errors? backup tasks harmed by cleanup? If growth recurs, compare against the initial snapshot to identify new releases, traffic anomalies, retry storms, coredump loops, or ineffective retention.

For log-related root causes, define clear ownership boundaries: producer (rate-limiting, structured output), collector (delivery, retry caps, disk-backpressure protection), platform (central retention, search), host-side cleanup (only compressed, expired, centrally-searchable replicas). Any gap turns the local disk into an unbounded buffer.

For data, backup, and cache root causes, classify before deletion: "regeneratable", "restorable from backup", "sole copy". Build caches are usually regeneratable; database physical backups generally not deletable; business caches require cache-penetration and origin-capacity checks. Document these classifications in runbooks — more effective than aggressive cron cleanup.

Monthly fire-drill: in staging, create a controlled large log or deleted-held file; require on-call to execute scene protection, root-cause classification, intervention, verification, and rollback. Drill value lies in validating alert context, permissions, backups, cleanup scripts, and cross-team communication — not in space freed.

Expansion is not the only answer after cleanup failure, nor is it risk-free. First confirm growth legitimacy: an accidental log storm means expansion merely masks missing rotation; legitimate business growth with sound retention means cleanup would harm business. Cloud disk or LVM expansion involves partitioning, filesystem, snapshots, backup windows, and cost governance; root-disk expansion may be constrained by boot disk, image, and rescue modes. Runbooks should separate "expansion request" and "emergency space release" with distinct triggers and approvers.

After space recovery, check components impacted by the full disk: failed cron jobs without auto-retry, log agents stopped on write failure, container runtime left in abnormal state, application retry queues backed up, database temp-file errors causing connection-pool exhaustion. Read each component's failure records and state, then recover per its documentation — do not assume df dropping means the fault self-healed.

Finally, bake "minimum available space" into capacity planning, not just percentage alerts. Root partitions need headroom for upgrade packages, log bursts, and crash dumps; data partitions need room for compaction, snapshots, temporary sorts, and recovery ops; container nodes need space for image pulls and layer extraction. Minimum free space per purpose should derive from peak write rates and recovery time objectives, and be re-validated during scale events.

Capacity retrospectives must also audit permissions and ownership. Manual sudo operations during a full-disk incident can alter application directory ownership or SELinux contexts; "permission denied" errors often surface only after space returns. Log every manual command, path, and executing account; verify with ls -l, namei, and security audit logs before restoring service — far more reliable than post-hoc permission guessing.