When Logs Fill the Disk: How I Cleared Three Days of Log Files

Incident Overview

Monitoring alerts showed disk usage climbing to 98 %, 99 % and finally 100 %. df -h displayed the root filesystem /dev/vda1 at 100 % capacity, causing system slowdown and service failures.

Investigation Process

Step 1 – Identify large directories

du -sh /* 2>/dev/null | sort -hr | head -10

Result: /data ≈ 23 GB and /var/log ≈ 12 GB, together accounting for most space.

Step 2 – Locate large files in /var/log

du -sh /var/log/* | sort -hr | head -10

Top files: /var/log/syslog ≈ 8.5 GB, /var/log/app.log ≈ 4.2 GB, /var/log/nginx/access.log ≈ 2.1 GB, /var/log/auth.log ≈ 1.8 GB.

Step 3 – Inspect file contents

tail -500 /var/log/app.log

The tail shows hundreds of identical stack‑trace lines per second, indicating a rapidly growing log caused by a repeatedly thrown exception.

Root‑Cause Analysis

Log rotation not configured – no size‑ or time‑based rotation, so files grew without bound.

Log level too high – DEBUG enabled in production.

Exception loop – an infinite catch block kept logging the same error.

No disk usage alerts – monitoring thresholds were absent, so the problem was discovered only after the disk filled.

Emergency Cleanup

# 1. Truncate the biggest files (keep the file, clear contents)
> /var/log/app.log
> /var/log/syslog

# 2. Delete rotated logs older than 7 days
find /var/log -name "*.log.*" -type f -mtime +7 -delete

# 3. If still full, locate other large files
find / -type f -size +500M 2>/dev/null

Long‑Term Solutions

Solution 1 – Configure logrotate

# /etc/logrotate.d/app
/var/log/app.log {
    daily
    rotate 7
    size 500M
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root root
    postrotate
        systemctl restart app
    endscript
}

Solution 2 – Application‑level logging configuration (logback)

<!-- logback.xml: production should use INFO level -->
<root level="INFO">
    <appender-ref ref="ASYNC"/>
</root>

<!-- Size‑ and time‑based rolling -->
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>logs/app.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
        <fileNamePattern>logs/app.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
        <maxFileSize>500MB</maxFileSize>
        <maxHistory>7</maxHistory>
        <totalSizeCap>10GB</totalSizeCap>
    </rollingPolicy>
</appender>

Solution 3 – Add Prometheus disk‑usage alerts

# Prometheus alert rules
groups:
- name: disk_alerts
  rules:
  - alert: DiskUsageHigh
    expr: (1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100 > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Disk usage exceeds 80%"
  - alert: DiskUsageCritical
    expr: (1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100 > 90
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Disk usage exceeds 90%"

Command Reference

df -h

– view filesystem usage. du -sh * | sort -hr – list directories by size. find / -type f -size +500M – locate files larger than 500 MiB. > /var/log/app.log – truncate a file without deleting it. find /var/log -name "*.log.*" -mtime +7 -delete – delete rotated logs older than seven days. journalctl --vacuum-size=500M – clean up systemd journal.