Zero‑Downtime Nginx Log Rotation: Full Logrotate Automation & Compression Guide

Applicable Scenarios & Prerequisites

Applicable Business : Daily traffic > 1M PV, log growth > 500 MB/day, requires automatic archiving and cleanup.

Prerequisites :

Nginx ≥ 1.18 (supports USR1 signal)

Logrotate ≥ 3.8 (supports su command)

OS: RHEL 7/8, Ubuntu 18.04/20.04/22.04

Disk space: reserve at least 30 days of logs (log size × 30 × 1.2)

Permissions: root or sudo, writable /var/log/nginx/ and

/etc/logrotate.d/

Environment & Version Matrix

Nginx : 1.18+ (recommended 1.24+)

Logrotate : 3.8+ (recommended 3.18+)

Compression tools : gzip, bzip2, xz (CPU ≥ 2 cores)

Cron : built‑in

Quick Checklist

Check Nginx log paths and permissions

Verify Logrotate is installed and running

Create Nginx‑specific Logrotate config (daily/weekly rotation, compression)

Configure Nginx post‑rotate script (USR1 signal)

Test Logrotate manually ( --debug --force)

Validate rotation and compression

Configure automatic cleanup of expired logs

Monitor Logrotate task status

Optimize compression performance (parallel, algorithm)

Optionally archive logs to remote storage (rsync/OSS)

Implementation Steps

Step 1: Verify Nginx Log Path & Permissions

View current log configuration:

# grep -r "access_log\|error_log" /etc/nginx/nginx.conf /etc/nginx/conf.d/*.conf

Check file permissions: ls -lh /var/log/nginx/ Expected ownership: nginx (or www-data) and group adm, mode 640.

Step 2: Verify Logrotate Installation

# RHEL/CentOS
rpm -qa | grep logrotate
# Ubuntu/Debian
dpkg -l | grep logrotate

If missing, install:

# RHEL/CentOS
sudo yum install -y logrotate
# Ubuntu/Debian
sudo apt update && sudo apt install -y logrotate

Check cron job:

cat /etc/cron.daily/logrotate

Step 3: Create Nginx Logrotate Configuration

/var/log/nginx/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 $(cat /var/run/nginx.pid)
        fi
    endscript
}

For Ubuntu/Debian adjust the create line to use www-data.

Step 4: Test Manual Rotation

sudo logrotate -f /etc/logrotate.d/nginx

Verify new log file and rotated files:

ls -lht /var/log/nginx/ | head -n 5

Step 5: Verify Compression

ls -lh /var/log/nginx/*.gz

Calculate compression ratio if needed.

Step 6: Advanced Rotation Strategies

Size‑based rotation (trigger when a single log exceeds 500 MB):

/var/log/nginx/*.log {
    size 500M
    rotate 50
    compress
    delaycompress
    missingok
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        kill -USR1 $(cat /var/run/nginx.pid) 2>/dev/null || true
    endscript
}

Per‑project configuration (API logs, static logs) with dateext and dateformat.

High‑compression algorithm using xz:

/var/log/nginx/*.log {
    daily
    rotate 30
    compresscmd /usr/bin/xz
    compressoptions -9
    compressext .xz
    uncompresscmd /usr/bin/unxz
    delaycompress
    missingok
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

Step 7: Automatic Cleanup of Expired Logs

Method 1 – Logrotate maxage:

/var/log/nginx/*.log {
    daily
    rotate 30
    maxage 30
    compress
    delaycompress
    missingok
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

Method 2 – Independent cron script:

# /usr/local/bin/clean-old-nginx-logs.sh
#!/bin/bash
find /var/log/nginx/ -name "*.log.*.gz" -mtime +30 -delete
find /var/log/nginx/ -name "*.log.*" ! -name "*.gz" -mtime +7 -delete
chmod +x /usr/local/bin/clean-old-nginx-logs.sh
0 4 * * * /usr/local/bin/clean-old-nginx-logs.sh

Step 8: Monitoring Logrotate Execution

Check cron logs: grep logrotate /var/log/cron | tail -n 20 Prometheus metrics examples:

# Log file size
node_filefd_allocated{path="/var/log/nginx/access.log"} > 5e9
# Rotation timestamp older than 24 h
time() - node_textfile_mtime_seconds{file="/var/lib/node_exporter/textfile_collector/logrotate_status.prom"} > 86400
# Disk usage > 80%
(node_filesystem_size_bytes{mountpoint="/var/log"} - node_filesystem_avail_bytes) / node_filesystem_size_bytes > 0.8

Step 9: Compression Performance Tuning

Parallel gzip with pigz:

# Install pigz
sudo yum install -y pigz   # or apt install -y pigz
# Logrotate snippet
compresscmd /usr/bin/pigz
compressoptions -p 4 -6
compressext .gz

CPU‑nice and I/O‑nice:

compresscmd /usr/bin/nice
compressoptions -n 19 gzip -6
# or combine with ionice
compresscmd /usr/bin/nice
compressoptions -n 19 ionice -c 3 gzip -6

Step 10: Remote Archiving (Optional)

Rsync to backup server:

# /usr/local/bin/nginx-log-archive.sh
#!/bin/bash
SOURCE_DIR="/var/log/nginx/"
BACKUP_SERVER="backup.example.com"
BACKUP_DIR="/data/nginx-logs/$(hostname)/"
rsync -avz --include="*.log.*.gz" --exclude="*.log" $SOURCE_DIR root@$BACKUP_SERVER:$BACKUP_DIR
find $SOURCE_DIR -name "*.log.*.gz" -mtime +30 -delete

OSS upload example omitted for brevity.

Security & Compliance

File Permissions : set chmod 640 on log files, owner nginx (or www-data) and group adm; directory chmod 750.

Encryption for Sensitive Logs using GPG:

# Generate GPG key
gpg --full-generate-key
# Encrypt rotated logs
for file in /var/log/nginx/*.log.*.gz; do
    gpg --encrypt --recipient [email protected] -o ${file}.gpg $file
    rsync -avz ${file}.gpg backup@secure-server:/encrypted-logs/
    rm -f $file ${file}.gpg
done

Audit Access with auditd:

sudo auditctl -w /var/log/nginx/ -p r -k nginx_log_access
sudo ausearch -k nginx_log_access -i

Compliance Checklist (ensure permissions 640, retention days, encryption, audit, automatic cleanup, encrypted transfer, integrity verification).

Common Issues & Troubleshooting

Logs still written to old file after rotation – check lsof | grep deleted and ensure postrotate sends USR1 signal.

Logrotate not running – verify cron service and executable /etc/cron.daily/logrotate.

Compressed files not created – ensure compress directive and proper permissions.

Disk space exhaustion – clean old logs or adjust rotate count.

Permission errors – correct create user/group.

Nginx log loss – debug postrotate script.

Incorrect filenames – enable dateext and dateformat.

Change & Rollback Playbook

Maintenance Window

Recommended time: 03:00‑04:00 AM. Pre‑checks: backup /etc/logrotate.d/nginx, verify disk space, ensure Nginx is running.

Gray‑Deployment Strategy

Stage 1 – Deploy to a single server, test with logrotate -f, monitor for 24 h.

Stage 2 – Batch deploy via Ansible:

ansible webservers -m copy -a "src=/etc/logrotate.d/nginx dest=/etc/logrotate.d/"
ansible webservers -m command -a "logrotate -d /etc/logrotate.d/nginx"

Health Checks

# Syntax check
logrotate -d /etc/logrotate.d/nginx
# Nginx file descriptors
lsof -p $(cat /var/run/nginx.pid) | grep -E '\.log.*deleted'
# Disk usage
df -h /var/log | awk 'NR==2 {if (int($5) > 85) exit 1}'
# Last rotation timestamp
stat -c %y /var/log/nginx/access.log.1

Rollback Conditions

Nginx stops writing after rotation

Logrotate causes high I/O or CPU load

Rollback steps:

# Restore original config
cp /etc/logrotate.d/nginx.backup /etc/logrotate.d/nginx
# Force Nginx to reopen logs
kill -USR1 $(cat /var/run/nginx.pid)
# Verify logging
tail -f /var/log/nginx/access.log
# If still failing, restart Nginx as last resort
systemctl restart nginx

Best Practices

Set rotation frequency (daily for >1 GB/day) and retention according to compliance.

Always enable delaycompress for the most recent rotated file.

Use minimal permissions (640 for files, 750 for directory) and owner Nginx user.

Make postrotate idempotent – use kill -USR1 instead of full reload.

Choose compression algorithm based on workload (gzip‑6 for daily, xz‑9 for archival).

Keep only 7‑30 days locally; archive older logs to remote storage.

Monitor rotation timestamps and disk usage via Prometheus alerts.

Test new configuration in a staging environment with logrotate -f before production rollout.

Version‑control Logrotate files (Git) and review changes via PR.

Enable dateext for date‑based filenames to simplify searching.

Appendix

Full Production Configuration (/etc/logrotate.d/nginx)

/var/log/nginx/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    dateext
    dateformat -%Y%m%d
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid) 2>/dev/null || true
    endscript
    lastaction
        find /var/log/nginx/ -name "*.log-*.gz" -mtime +90 -delete
    endscript
}

# API logs (90‑day retention)
/var/log/nginx/api-access.log {
    daily
    rotate 90
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    dateext
    dateformat -%Y%m%d
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

Ansible Playbook for Bulk Deployment

---
- name: Deploy Nginx Logrotate configuration
  hosts: webservers
  become: yes
  tasks:
    - name: Backup existing config
      copy:
        src: /etc/logrotate.d/nginx
        dest: /etc/logrotate.d/nginx.backup
        remote_src: yes
        ignore_errors: yes

    - name: Deploy new config
      copy:
        src: files/logrotate.d/nginx
        dest: /etc/logrotate.d/nginx
        owner: root
        group: root
        mode: '0644'

    - name: Verify syntax
      command: logrotate -d /etc/logrotate.d/nginx
      register: logrotate_check
      failed_when: logrotate_check.rc != 0

    - name: Test rotation
      command: logrotate -f /etc/logrotate.d/nginx
      when: not ansible_check_mode

    - name: Verify Nginx log writing
      shell: |
        sleep 5
        tail -n 1 /var/log/nginx/access.log
      register: log_check
      failed_when: log_check.stdout == ""

Logrotate Status Check Script (/usr/local/bin/check-logrotate-status.sh)

#!/bin/bash
# Check last rotation time
LAST_ROTATE=$(stat -c %Y /var/log/nginx/access.log.1 2>/dev/null || echo 0)
CURRENT_TIME=$(date +%s)
AGE=$(( (CURRENT_TIME - LAST_ROTATE) / 3600 ))
if [ $AGE -gt 25 ]; then
    echo "CRITICAL: Log rotation delayed by $AGE hours"
    exit 2
fi
# Check current log size (>5GB)
LOG_SIZE=$(stat -c%s /var/log/nginx/access.log 2>/dev/null || echo 0)
if [ $LOG_SIZE -gt 5000000000 ]; then
    echo "WARNING: access.log size $((LOG_SIZE/1024/1024))MB exceeds 5GB"
    exit 1
fi
# Disk usage (>85%)
DISK_USAGE=$(df /var/log | awk 'NR==2 {print int($5)}')
if [ $DISK_USAGE -gt 85 ]; then
    echo "CRITICAL: /var/log disk usage ${DISK_USAGE}%"
    exit 2
fi
echo "OK: Log rotation working properly"
exit 0