Practical Guide to Linux Log Management with rsyslog, journald, and logrotate

Overview

The article explains why a reliable log management system is essential for rapid incident response, citing a real‑world case where uncoordinated logs across dozens of servers caused a four‑hour outage that was later reduced to under 15 minutes after redesign.

Log Architecture

Three components work together:

journald – binary, indexed storage with fast queries (e.g., journalctl -u nginx.service -p err --since "1 hour ago").

rsyslog – classic syslog daemon that reads from journald (via imjournal) and writes structured text files.

logrotate – rotates, compresses and archives old log files.

The default CentOS 7/8 setup uses imjournal for seamless cooperation; Ubuntu defaults to imuxsock which loses structured fields.

Installation & System Check

# Verify OS version
cat /etc/os-release
# Verify systemd version
systemctl --version
# Install rsyslog and required modules
sudo yum install -y rsyslog rsyslog-relp rsyslog-mmjsonparse   # CentOS/RHEL
sudo apt install -y rsyslog rsyslog-relp rsyslog-mmjsonparse   # Ubuntu/Debian
# Ensure logrotate is present
which logrotate
logrotate --version

Core rsyslog Configuration

The main file /etc/rsyslog.conf is divided into three sections: global directives, module loading, and rule definitions.

# Global directives
global(
    workDirectory="/var/lib/rsyslog"
    maxMessageSize="64k"
    preserveFQDN="on"
)

# Load modules
module(load="imuxsock" SysSock.Use="on" SysSock.RateLimit.Interval="5" SysSock.RateLimit.Burst="2000")
module(load="imjournal" StateFile="imjournal.state" IgnorePreviousMessages="off")
module(load="imklog")
module(load="immark" interval="600")
# TCP/RELP modules are enabled only on log‑center servers
#module(load="imtcp" MaxSessions="1000")
#input(type="imtcp" port="514" ruleset="remote")
#module(load="imrelp")
#input(type="imrelp" port="2514" ruleset="remote")

Key filtering rules (facility.severity syntax, property‑based selectors, and RainerScript expressions) are demonstrated, e.g.:

# Log all info and higher, except mail, authpriv, cron
*.info;mail.none;authpriv.none;cron.none    /var/log/messages
# Log all errors to a separate file
*.err    /var/log/error-all.log
# Program‑specific rule for sshd
:programname, isequal, "sshd"    /var/log/sshd.log

Template Definitions

Two templates are provided: a precise timestamp format and a JSON format for downstream ELK ingestion.

template(name="precise" type="string" string="%timegenerated:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%
")

template(name="json-syslog" type="list") {
    constant(value="{")
    constant(value="\"@timestamp\":\"") property(name="timegenerated" dateFormat="rfc3339")
    constant(value="\",\"host\":\"") property(name="hostname")
    constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
    constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
    constant(value="\",\"program\":\"") property(name="programname")
    constant(value="\",\"pid\":\"") property(name="procid")
    constant(value="\",\"message\":\"") property(name="msg" format="jsonf")
    constant(value="}
")
}

Remote Log Forwarding

Both TCP (reliable) and RELP (guaranteed delivery) examples are included. The configuration uses a disk‑backed queue to avoid loss during network interruptions.

# TCP forwarding with persistent queue
action(type="omfwd" target="192.168.1.100" port="514" protocol="tcp" \
       action.resumeRetryCount="-1" queue.type="LinkedList" queue.filename="fwd_to_logserver" \
       queue.maxDiskSpace="1g" queue.saveOnShutdown="on" queue.size="50000" queue.discardMark="48000" queue.discardSeverity="7" action.resumeInterval="10")

# RELP forwarding (recommended for audit logs)
module(load="omrelp")
action(type="omrelp" target="192.168.1.100" port="2514" \
       action.resumeRetryCount="-1" queue.type="LinkedList" queue.filename="relp_to_logserver" \
       queue.maxDiskSpace="2g" queue.saveOnShutdown="on")

logrotate Configuration

The global /etc/logrotate.conf sets weekly rotation, 4 retained copies, compression, and dateext for readable filenames. Individual service files (e.g., /etc/logrotate.d/nginx) use postrotate to send SIGUSR1 to nginx.

# /etc/logrotate.conf
weekly
rotate 4
create
dateext
dateformat -%Y%m%d
compress
delaycompress
include /etc/logrotate.d

# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    rotate 30
    missingok
    notifempty
    compress
    delaycompress
    dateext
    dateformat -%Y%m%d
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 $(cat /var/run/nginx.pid)
        fi
    endscript
}

Performance Optimisation

Enabling asynchronous writes and a larger main queue reduces CPU from 12 % to 3 % at 5 k msg/s. Example /etc/rsyslog.d/01-performance.conf shows a LinkedList queue with 100 k entries and four worker threads.

# Main queue optimisation
main_queue(
    queue.type="LinkedList"
    queue.filename="main_q"
    queue.maxDiskSpace="1g"
    queue.saveOnShutdown="on"
    queue.size="100000"
    queue.discardMark="90000"
    queue.discardSeverity="7"
    queue.workerThreads="4"
    queue.dequeueBatchSize="1000"
    queue.timeoutEnqueue="0"
)

Security Hardening

Set Storage=persistent in /etc/systemd/journald.conf and create /var/log/journal to avoid loss on reboot.

Restrict log file permissions (0640 for most logs, 0600 for /var/log/secure).

Use TLS for remote forwarding (gtls module) and enable RELP for audit‑grade reliability.

Mark critical logs with the immutable attribute ( chattr +a) and manage the attribute in prerotate/postrotate scripts.

Troubleshooting & Monitoring

Common failure scenarios (e.g., /var/log/messages not updating, logrotate not running, journald missing entries) are listed with step‑by‑step diagnostics, such as checking rsyslogd -N1, SELinux denials, imjournal.state corruption, and disk usage.

# Verify rsyslog syntax
rsyslogd -N1
# Check journald storage mode
grep -i storage /etc/systemd/journald.conf
# Show disk usage of journal
journalctl --disk-usage
# Quick log‑write test
logger -p local0.info "test message"

Performance metrics to monitor (CPU, memory, queue size, /var/log usage, journald disk usage) are provided, along with Prometheus alert rules for high CPU, disk saturation, and log loss.

Backup & Recovery

A Bash script ( /usr/local/bin/log-backup.sh) creates daily compressed archives of rotated logs, journald snapshots, and configuration files, stores them under /data/log-backup/YYYYMMDD, and optionally syncs to a remote rsync server. Retention is configurable (default 90 days).

# Example backup script excerpt
BACKUP_DIR="/data/log-backup"
DATE=$(date +%Y%m%d)
mkdir -p "${BACKUP_DIR}/${DATE}"
# Copy rotated gzip files
find /var/log -name "*.gz" -newer "${BACKUP_DIR}/.last_backup" -exec cp {} "${BACKUP_DIR}/${DATE}/" \;
# Export recent journald entries
journalctl --since "24 hours ago" > "${BACKUP_DIR}/${DATE}/journald-${HOSTNAME}-${DATE}.log"
# Archive configuration
tar czf "${BACKUP_DIR}/${DATE}/log-config-${DATE}.tar.gz" /etc/rsyslog.conf /etc/rsyslog.d /etc/systemd/journald.conf /etc/logrotate.conf /etc/logrotate.d

Restoration steps include stopping rsyslog, extracting the configuration archive, validating with rsyslogd -N1, and optionally restoring compressed log files.

Conclusion

The guide consolidates best practices for a production‑grade Linux logging stack: use journald for fast, structured queries; let rsyslog handle classification, remote forwarding, and flexible formatting; rely on logrotate for safe rotation and retention; and enforce security, performance, and observability throughout.