Master Linux Log Management: rsyslog, journald & logrotate Hands‑On Guide

Overview

This guide explains why a well‑designed log management stack (rsyslog + systemd‑journald + logrotate) is essential for fast incident response on CentOS 7/8 and Ubuntu 20.04/22.04. It shows the complete log flow from kernel messages to remote aggregation and provides production‑grade configuration examples.

Preparation

Understand the Linux logging pipeline: kernel → rsyslog (imklog) → /var/log, systemd services → journald (binary) → imjournal → rsyslog → text files.

Verify OS version, systemd version, and that rsyslog is installed.

Ensure a dedicated /var/log partition (≥10 GB) and enough RAM for rsyslog (30‑80 MB).

Core Configuration

rsyslog

The main configuration file is /etc/rsyslog.conf. It consists of three sections: module loading, global directives, and rules.

# /etc/rsyslog.conf – main configuration
module(load="imuxsock" SysSock.Use="on" SysSock.RateLimit.Interval="5" SysSock.RateLimit.Burst="2000")
module(load="imjournal" StateFile="imjournal.state" IgnorePreviousMessages="off" DefaultSeverity="5" DefaultFacility="user")
module(load="imklog")
module(load="immark" interval="600")

# Template for precise timestamps (RFC‑3339 with milliseconds)
template(name="precise" type="string" string="%timegenerated:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%
")

# Store most messages in /var/log/messages (exclude mail, authpriv, cron)
*.info;mail.none;authpriv.none;cron.none    action(type="omfile" file="/var/log/messages" template="precise")

# Application‑specific facilities
local0.*    /var/log/nginx/syslog.log
local1.*    /var/log/app/service.log

# Remote forwarding with a persistent disk queue
action(type="omfwd" target="log-center.internal.com" port="514" protocol="tcp" template="precise" queue.type="LinkedList" queue.filename="fwd_to_logcenter" queue.maxDiskSpace="1g" queue.saveOnShutdown="on" queue.size="50000" action.resumeRetryCount="-1" action.resumeInterval="10")

Key rsyslog parameters workDirectory="/var/lib/rsyslog" – location for queue and state files. maxMessageSize="64k" – maximum size of a single log entry.

Queue settings (type, size, disk space, worker threads) control buffering and loss‑tolerance.

Use the leading - on a file name (e.g. - /var/log/maillog) to enable async writing.

systemd‑journald

Configuration lives in /etc/systemd/journald.conf. The most important options are:

# /etc/systemd/journald.conf
[Journal]
Storage=persistent               # store logs under /var/log/journal
SystemMaxUse=1G                 # total disk usage limit
SystemMaxFileSize=100M          # per‑file size limit
MaxRetentionSec=30day           # keep logs for 30 days (adjust for compliance)
RateLimitIntervalSec=30s
RateLimitBurst=10000
ForwardToSyslog=yes            # forward to rsyslog for text files
Compress=yes

After editing, run systemctl restart systemd-journald and verify with journalctl --disk-usage.

logrotate

Global settings are in /etc/logrotate.conf; per‑application rules live under /etc/logrotate.d/. A typical global file:

# /etc/logrotate.conf – global settings
weekly
rotate 4
create
dateext
dateformat -%Y%m%d
compress
delaycompress
include /etc/logrotate.d

Example for nginx logs (rotate daily, keep 30 days, signal nginx after rotation):

# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    rotate 30
    missingok
    notifempty
    compress
    delaycompress
    dateext
    dateformat -%Y%m%d
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 $(cat /var/run/nginx.pid)
        fi
    endscript
}

Use dateext to avoid filename collisions and create (or copytruncate when the application cannot reopen its log file).

Best Practices & Pitfalls

Performance Optimization

Enable async writing and a multi‑threaded main queue in rsyslog to reduce CPU usage under high log rates.

Adjust journald RateLimitBurst for bursty applications.

Prefer dateext in logrotate to make rotated filenames self‑describing.

Security Hardening

Set log file permissions to 0640 (or 0600 for highly sensitive logs) and owner root:adm.

Use TLS (or RELP) for remote rsyslog forwarding: set streamDriver="gtls", provide CA and certificate files, and listen on port 6514.

Mark critical log files with chattr +a to make them append‑only, and protect the attribute with a prerotate / postrotate hook in logrotate.

High Availability

Configure dual log‑center servers and forward to both using separate omfwd actions (different queue.filename values).

Back up /var/log and journald archives to NFS or object storage on a daily schedule.

Troubleshooting & Monitoring

Common Issues

/var/log/messages not updating : check rsyslog status, run rsyslogd -N1 for syntax errors, verify SELinux denials, and delete a corrupted /var/lib/rsyslog/imjournal.state file.

journalctl missing history : ensure Storage=persistent and that /var/log/journal exists; otherwise create it and restart journald.

Disk fills up : confirm logrotate is running (cron or logrotate.timer), check dateext usage, and verify /var/log partition size.

Remote forwarding loss : avoid UDP; use TCP or RELP with a persistent disk queue and increase queue.maxDiskSpace.

Debug Commands

# Verify rsyslog syntax
rsyslogd -N1
# Run rsyslog in foreground for debugging
rsyslogd -dn
# Check journald disk usage
journalctl --disk-usage
# Test remote forwarding
logger -p local0.info "test remote"

Performance Metrics (Prometheus example)

# rsyslog CPU usage
rate(namedprocess_namegroup_cpu_seconds_total{groupname="rsyslogd"}[5m]) * 100
# /var/log partition usage (node_exporter metric)
(1 - node_filesystem_avail_bytes{mountpoint="/var/log"} / node_filesystem_size_bytes{mountpoint="/var/log"}) * 100
# journald size (custom exporter)
log_journald_size_bytes

Export these metrics via a custom script that writes to /var/lib/node_exporter/textfile_collector/ or use a dedicated rsyslog exporter.

Backup & Recovery

Backup Script (example)

#!/bin/bash
set -euo pipefail
BACKUP_DIR="/data/log-backup"
DATE=$(date +%Y%m%d)
HOST=$(hostname -s)
mkdir -p "$BACKUP_DIR/$DATE"
# Archive rotated gzip logs
find /var/log -name "*.gz" -newer "$BACKUP_DIR/.last_backup" -exec cp {} "$BACKUP_DIR/$DATE/" \;
# Export recent journald entries
journalctl --since "24 hours ago" > "$BACKUP_DIR/$DATE/journald-${HOST}-${DATE}.log"
# Archive configuration files
tar czf "$BACKUP_DIR/$DATE/log-config-${DATE}.tar.gz" /etc/rsyslog.conf /etc/rsyslog.d/ /etc/systemd/journald.conf /etc/logrotate.conf /etc/logrotate.d/
# Optional: rsync to remote storage
# rsync -az "$BACKUP_DIR/$DATE/" remote:/backup/logs/$HOST/$DATE/
# Cleanup old backups (keep 90 days)
find "$BACKUP_DIR" -maxdepth 1 -type d -mtime +90 -exec rm -rf {} \;
# Update timestamp marker
touch "$BACKUP_DIR/.last_backup"

Recovery Steps

Stop rsyslog: systemctl stop rsyslog.

Restore the configuration archive:

tar xzf /data/log-backup/20240115/log-config-20240115.tar.gz -C /

.

Optionally copy archived log files back to /var/log and decompress them.

Start rsyslog and verify:

systemctl start rsyslog && logger "recovery test" && tail -1 /var/log/messages

.

Conclusion

Combining rsyslog, systemd‑journald, and logrotate provides a complete, scalable logging solution for Linux servers. Proper configuration, performance tuning, security hardening, and regular backup ensure reliable log collection, fast query, and compliance with audit requirements.