Master Linux Log Management: rsyslog, systemd‑journal, and Logrotate Explained

rsyslog Log Management

rsyslog is the traditional syslog daemon on Linux. It writes structured log messages to files under /var/log. The newer systemd‑journal captures kernel, boot, and service output in a binary journal.

Common Log Files

/var/log/wtmp – login/logout events

/var/log/boot.log – boot‑time messages

/var/log/messages – general system and service logs

/var/log/secure – security‑related logs

/var/log/lastlog – successful login timestamps

/var/log/btmp – failed login attempts

/var/log/xfer.log – FTP activity

/var/log/httpd/access_log – HTTP access logs

/var/log/httpd/error_log – HTTP error logs

/var/log/yum.log – yum package manager logs

Log Levels

emerg – system crash or kernel panic

alert – immediate‑fix issues (e.g., database corruption)

crit – high‑severity failures (e.g., disk failure)

err – general errors (service start/stop failures)

warning – misconfigurations or non‑critical warnings

notice – normal but noteworthy events

info – informational messages

debug – debugging output

Facilities (Log Types)

auth – authentication events

authpriv – privileged authentication

mail – mail subsystem

cron – scheduled jobs

kern – kernel messages

user – user‑level messages (default)

local0‑local7 – custom local facilities

Configure Log Paths (rsyslog.conf)

vim /etc/rsyslog.conf
*.info;mail.none;authpriv.none;cron.none    /var/log/messages
authpriv.*                                 /var/log/secure
mail.*                                     /var/log/maillog
cron.*                                     /var/log/cron
uucp,news.crit                             /var/log/spooler
local7.*                                   /var/log/boot.log
.emerg                                      :omusrmsg:*

Remote Log Server – Client Configuration

vim /etc/rsyslog.recieve
*.info;mail.none;authpriv.none;cron.none    @your-log-server
systemctl restart rsyslog.recieve

Test the forwarding with:

logger -p local7.notice "Log entry created on host"
# On the server monitor the incoming logs
journalctl -f -u rsyslog

Configure Machine as Log Server

vim /etc/rsyslog.conf
module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp")
input(type="imtcp" port="514")
systemctl restart rsyslog.recieve

Open firewall ports for UDP/TCP 514:

firewall-cmd --add-port=514/tcp --permanent
firewall-cmd --add-port=514/udp --permanent
firewall-cmd --reload

Verify listening sockets:

lsof -i:514

Secure rsyslog (append‑only)

chattr +a /var/log/messages   # make file append‑only
lsattr /var/log/messages       # view attributes

systemd‑journal Log Management

systemd‑journal stores logs in a binary format. By default logs are kept in /run/log/journal (volatile) and disappear after a reboot unless persistence is enabled.

journald.conf Options

/etc/systemd/journald.conf
Storage=auto          # auto, volatile, persistent, none
SystemMaxUse=        # total disk space limit
SystemMaxFileSize=   # per‑file size limit
SystemMaxFiles=100   # max number of files
RuntimeMaxUse=       # memory usage limit
MaxFileSec=          # rotation interval
MaxRetentionSec=     # retention period (e.g., 1month, 7day)
ForwardToSyslog=no
ForwardToKMsg=no
ForwardToConsole=no
ForwardToWall=yes
TTYPath=/dev/console
MaxLevelStore=debug
MaxLevelSyslog=debug
MaxLevelKMsg=notice
MaxLevelConsole=info
MaxLevelWall=emerg

Analyzing Journal Logs

journalctl -n 10                     # last 10 entries
journalctl --since "2023-04-20 19:40"   # from a specific time
journalctl -u httpd                # logs for a specific service
journalctl -f                      # follow live output
journalctl --disk-usage            # total journal size
journalctl --vacuum-size=1G        # keep size under 1 GB
journalctl --vacuum-time=1h        # keep logs for the last hour
journalctl -p err                  # filter by error level
journalctl -o json                 # output as JSON
journalctl -b -1                    # logs from previous boot (requires persistence)

Persisting Journal Logs

Edit /etc/systemd/journald.conf and set Storage=persistent, then restart the service.

Leave Storage=auto, create /var/log/journal, set group ownership to systemd-journal, apply permissions chmod 2775 /var/log/journal, and restart the service.

Log Rotation with logrotate

logrotate prevents log files from growing indefinitely, protecting system stability and disk space.

Identify log files to rotate.

Check if they exceed size or age limits.

Compress old logs if needed.

Delete or archive older generations.

Create a fresh log file for continued logging.

/etc/logrotate.conf
weekly                # rotate weekly
rotate 4              # keep 4 weeks of archives
create                # create new empty log files
dateext               # append date to rotated filenames
compress              # gzip old logs
include /etc/logrotate.d   # include per‑service configs