Master Linux Logging: rsyslog, journald, and logrotate Explained

rsyslog Service

rsyslog is the default log management daemon on most Linux distributions, responsible for collecting and handling system and application logs.

Service file:

/lib/systemd/system/rsyslog.service

# /lib/systemd/system/rsyslog.service
[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8) https://www.rsyslog.com/doc/

[Service]
Type=notify
ExecStart=/usr/sbin/rsyslogd -n -iNONE
StandardOutput=null
Restart=on-failure
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
Alias=syslog.service

rsyslog can collect logs in two ways:

Directly from the system via the UNIX syslog interface.

Through systemd‑journald, where journald stores logs and rsyslog pulls them.

Typical workflow:

Collect logs locally or receive them from the network.

Parse the messages and process them according to rules defined in configuration files.

Configuration Management

Main configuration file: /etc/rsyslog.conf Additional files are included via $IncludeConfig, usually matching /etc/rsyslog.d/*.conf.

Configuration consists of three blocks:

Module configuration – load modules for needed functionality.

Global configuration – set basic parameters for the daemon.

Rule configuration – define how collected messages are processed.

Rule Management

Each rule has a selector and an action.

Selector

Selectors combine a facility (origin) and a priority (severity). Common facilities include auth, kern, mail, daemon, user, and * (all). Common priorities range from emerg (highest) to debug (lowest), plus none and * (all).

Action

Actions define what to do with matching messages, such as writing to a file, forwarding to a remote server, or stopping further processing.

Write to file: *.* /var/log/messages Write info‑level messages: *.info /var/log/info.log Write auth messages: auth,authpriv.* /var/log/auth.log Asynchronous write (dash prefix): *.*;auth,authpriv.none -/var/log/syslog Forward to remote server: *.* @remote‑server:514 Stop processing:

:msg, contains, "some text" ~

Custom Facility Example – sshd

Change sshd to use local7 facility and store its logs in /var/log/sshd.log:

# Edit /etc/ssh/sshd_config
SyslogFacility local7

# Create rsyslog rule
local7.* -/var/log/sshd.log

# Restart services
sudo systemctl restart sshd.service
sudo systemctl restart rsyslog.service

# Test
logger -p local7.info "hello sshd"

Network Log Management

Multiple hosts can send logs to a central rsyslog server.

Server configuration

Load input modules:

# UDP module
module(load="imudp")
input(type="imudp" port="514")

# TCP module
module(load="imtcp")
input(type="imtcp" port="514")

Client configuration

Send logs to the server using a selector followed by @host:port (UDP) or @@host:port (TCP).

*.info;mail.none;authpriv.none;cron.none @10.0.0.11:514

journald Service

systemd‑journald collects logs for the whole system and provides the journalctl tool for querying.

Primary configuration file: /etc/systemd/journald.conf (additional snippets in /etc/systemd/journald.conf.d/).

Storage Options

volatile

– keep logs only in memory. persistent – store logs on disk. auto (default) – use persistent if /var/log/journal exists, otherwise volatile. none – do not store logs.

Retention and Size Limits

Examples:

Storage=auto
MaxRetentionSec=30day
MaxFileSec=1month
SystemMaxUse=50M

Forward logs to traditional syslog daemon: ForwardToSyslog=yes Sync interval to flush memory to disk:

SyncIntervalSec=5m

journalctl Usage

Show all logs: journalctl Time range: journalctl --since "2023-10-01" --until "2023-10-08" Follow live output: journalctl -f Service logs: journalctl -u service_name (add -xe for extra details).

Process logs: journalctl _PID=1234 Kernel logs: journalctl -k Boot logs:

journalctl -b

logrotate

logrotate rotates and compresses log files to prevent uncontrolled growth.

Configuration Files

Main file: /etc/logrotate.conf Per‑service snippets:

/etc/logrotate.d/

Typical Rule Example

/var/log/syslog {
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

nginx Log Rotation Example

/var/log/nginx/*.log {
    monthly
    rotate 6
    compress
    delaycompress
    missingok
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 `cat /var/run/nginx.pid`
        fi
    endscript
}

Operation Workflow

Read configuration files.

Check each log file against rotation criteria (size, age, etc.).

Rename the current file and create a new empty one.

Optionally run post‑rotate scripts.

Compress old files and delete those exceeding the retention count.

Systemd Service and Timer

logrotate is executed by logrotate.service (a oneshot unit) triggered by logrotate.timer, which runs daily.

# /lib/systemd/system/logrotate.service
[Unit]
Description=Rotate log files
Documentation=man:logrotate(8) man:logrotate.conf(5)

[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf
Nice=19
PrivateTmp=true
ProtectSystem=full

# /lib/systemd/system/logrotate.timer
[Unit]
Description=Daily rotation of log files

[Timer]
OnCalendar=daily
AccuracySec=12h
Persistent=true

[Install]
WantedBy=timers.target