How to Diagnose and Automate Cleanup for Server Disk‑Full Alerts

1. Preserve the scene and confirm whether the problem is capacity, inode exhaustion, or storage failure

Check both byte space and inode usage because a surge of small files can exhaust inodes even when df -h shows free space.

df -hT
df -i
findmnt -T <mount_point>

Collect a read‑only snapshot of the system state; the script writes to a directory with sufficient free space and controlled permissions.

#!/usr/bin/env bash
set -euo pipefail
OUTPUT_DIR="<snapshot_dir>"
STAMP="$(date +%Y%m%d_%H%M%S)"
mkdir -p "$OUTPUT_DIR"
{
  date --iso-8601=seconds
  df -hT
  df -i
  findmnt -R <mount_point>
  journalctl -k --since '30 min ago' --no-pager | tail -n 200
} > "$OUTPUT_DIR/disk-full-$STAMP.txt"

Inspect kernel logs and mount options; if I/O errors, read‑only remounts, or filesystem errors appear, treat the issue as a storage fault rather than a simple cleanup.

journalctl -k --since '2 hours ago' --no-pager \
  | rg -i 'I/O error|read-only|remount|ext4|xfs|nvme|blk_update_request'
findmnt -T <mount_point> -o TARGET,SOURCE,FSTYPE,OPTIONS

Identify the top‑level directory to scan, avoiding cross‑mount scans with -x.

sudo du -xhd1 <mount_point> 2>/dev/null | sort -h

When du reports low usage but df is still full, look for deleted files still held open by processes using lsof +L1.

sudo lsof -nP +L1

Confirm large deleted files before removal; avoid truncating unknown file descriptors.

sudo lsof -nP +L1 | awk 'NR==1 || $7 > 1073741824'

Reload Nginx logs safely; other services may have different signal semantics.

sudo nginx -t
sudo systemctl reload nginx
sudo lsof -nP +L1 | rg 'nginx|<service_name>' || true

Audit logrotate policies with a dry‑run.

sudo logrotate -d /etc/logrotate.conf
sudo logrotate -d /etc/logrotate.d/<service_name>
systemctl status logrotate.timer --no-pager

For Docker, list usage before any docker system prune and examine container logs.

docker system df -v
docker ps -a --size
docker image ls
docker volume ls
sudo find /var/lib/docker/containers -name '*-json.log' -type f -printf '%s %p
' | sort -nr | head -n 30

To truncate a log file that has no retention value, export evidence first, then run truncate and verify health.

sudo truncate -s 0 /var/lib/docker/containers/<container_id>/<container_id>-json.log

2. Classify evidence to locate the space source

Start with the largest regular files as primary candidates; scanning the whole disk during an alert is costly.

sudo find <mount_point> -xdev -type f -size +1G -printf '%s %p
' 2>/dev/null | sort -nr | head -n 50

Use recent modification time as a clue, but still cross‑check with release records and application logs.

sudo find <mount_point> -xdev -type f -mtime -1 -size +100M -printf '%TY-%Tm-%Td %TH:%TM %s %p
' 2>/dev/null | sort -k3,3nr | head -n 80

Inspect /var and business log directories; logs may reside in work, cache, or container stdout locations.

sudo du -xhd2 /var 2>/dev/null | sort -h | tail -n 80
sudo du -xhd2 <log_dir> 2>/dev/null | sort -h | tail -n 80

When du is small but df reports full, focus on files deleted but still open.

3. Investigate journal, dumps, temporary files and inode usage

Validate journal size with journalctl --disk-usage and --verify. Only vacuum after confirming remote retention.

# Verify usage and integrity
journalctl --disk-usage
journalctl --verify
# After confirming remote storage and retention policy
sudo journalctl --vacuum-time=<retain_days>d
journalctl --disk-usage

Core dumps may contain sensitive data; list them first and keep only needed ones.

coredumpctl list --no-pager
sudo du -sh /var/lib/systemd/coredump 2>/dev/null || true

Configure systemd coredump retention (version‑specific).

# /etc/systemd/coredump.conf.d/50-retention.conf
[Coredump]
MaxUse=<max_usage>
KeepFree=<reserved_space>

Generate a candidate list for temporary files; do not assume /tmp files are unused.

sudo find /tmp /var/tmp -xdev -type f -mtime +<retain_days> -printf '%TY-%Tm-%Td %TH:%TM %s %p
' 2>/dev/null | head -n 200

Save the list as a NUL‑separated file and sample before deletion.

sudo find /tmp /var/tmp -xdev -type f -mtime +<retain_days> -print0 > <cleanup_list>
sudo xargs -0 -a <cleanup_list> -r ls -ld --
sudo xargs -0 -a <cleanup_list> -r rm -f --

Identify inode hot spots by counting files per directory, limiting the scan to known business paths.

sudo find <mount_point> -xdev -type f -printf '%h
' 2>/dev/null | sort | uniq -c | sort -nr | head -n 50

For Debian/Ubuntu, simulate APT cache cleanup before actual removal.

# Dry‑run and check cache size
sudo apt-get -s clean
sudo du -sh /var/cache/apt 2>/dev/null || true
# After verification
sudo apt-get clean
df -hT /

4. Automation cleanup must have guardrails

A robust script includes whitelist paths, file patterns, minimum age, dry‑run mode, audit logging and failure exit. Never use a blunt rm -rf <dir>/*.

#!/usr/bin/env bash
set -euo pipefail
TARGET_DIR="<log_dir>"
RETENTION_DAYS="<retain_days>"
AUDIT_LOG="/var/log/ops-log-cleanup.log"
DRY_RUN="1"

# Validate inputs
test -n "$TARGET_DIR"
test "$TARGET_DIR" != "/"
test -d "$TARGET_DIR"

find "$TARGET_DIR" -xdev -type f -name '*.gz' -mtime +$RETENTION_DAYS -print0 |
  while IFS= read -r -d '' file; do
    if [ "$DRY_RUN" = "1" ]; then
      printf 'would delete: %q
' "$file"
    else
      sudo rm -f -- "$file"
      printf '%s deleted=%q
' "$(date --iso-8601=seconds)" "$file" | sudo tee -a "$AUDIT_LOG" >/dev/null
    fi
  done

Run the script first with DRY_RUN=1, sample the output, then set DRY_RUN=0 on a low‑risk node.

sudo /usr/local/sbin/ops-log-cleanup
# After verification
sudo /usr/local/sbin/ops-log-cleanup
df -hT <mount_point>
systemctl is-active <service_name>

Deploy as a systemd timer to track failures and next run.

# /etc/systemd/system/ops-log-cleanup.service
[Unit]
Description=Clean expired compressed logs

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/ops-log-cleanup

# /etc/systemd/system/ops-log-cleanup.timer
[Unit]
Description=Run log cleanup daily

[Timer]
OnCalendar=*-*-* 03:20:00
Persistent=true

[Install]
WantedBy=timers.target

# Enable timer
sudo systemctl daemon-reload
sudo systemctl enable --now ops-log-cleanup.timer
systemctl list-timers ops-log-cleanup.timer --all
journalctl -u ops-log-cleanup.service -n 80 --no-pager

5. Acceptance, monitoring and rollback

Prometheus alerts should use separate metrics for capacity and inode; do not rely on a single percentage.

100 * (1 - node_filesystem_avail_bytes{mountpoint="<mount_point>",fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{mountpoint="<mount_point>",fstype!~"tmpfs|overlay"})

100 * (1 - node_filesystem_files_free{mountpoint="<mount_point>",fstype!~"tmpfs|overlay"} / node_filesystem_files{mountpoint="<mount_point>",fstype!~"tmpfs|overlay"})

Predictive trends can warn of continuous growth, but absolute free space thresholds are still required.

predict_linear(node_filesystem_avail_bytes{mountpoint="<mount_point>",fstype!~"tmpfs|overlay"}[6h], 4*3600) < 0

After remediation, verify space, inode, mount writability, service status, logs and health‑check endpoints. If space quickly rises again, the root cause was not fully addressed.

df -hT <mount_point>
df -i <mount_point>
findmnt -T <mount_point> -o TARGET,SOURCE,FSTYPE,OPTIONS
systemctl is-active <service_name>
journalctl -u <service_name> --since '10 min ago' --no-pager | tail -n 100
curl --fail --silent --show-error <health_check_url> >/dev/null

Rollback configuration by restoring backups and disabling timers; note that rm and truncate are irreversible.

sudo systemctl disable --now ops-log-cleanup.timer
sudo systemctl status ops-log-cleanup.timer --no-pager
sudo cp -a <backup_dir>/daemon.json /etc/docker/daemon.json
sudo jq empty /etc/docker/daemon.json
sudo systemctl restart docker
docker ps

6. Turn "disk full" alerts into actionable events

Enrich alerts with instance, mount point, remaining bytes, inode count, recent change rate, latest deployment and responsible service so responders can jump straight to verification.

After capacity is restored, keep the incident open for at least one full business peak or cleanup cycle to confirm that growth does not resume, journal rotation works, services stay healthy, and backup jobs are not impacted.

Define clear responsibility boundaries for log generation, collection, retention and cleanup; missing any side turns the host into an uncontrolled buffer.

Classify data, backup and cache root causes into "regenerable", "recoverable from backup" and "single‑instance" before deletion, and document the classification in runbooks.

Conduct monthly tabletop drills that simulate a large log burst or a deleted‑but‑held‑open file, requiring the on‑call engineer to follow the full preservation‑root‑cause‑intervention‑validation‑rollback workflow.

Capacity planning should record absolute free‑space lower bounds for root disks (upgrade packages, burst logs, dump files) and data disks (compaction, snapshots, temporary sorting), derived from peak write rates and recovery‑time objectives.