Linux Logging Deep Dive: Kernel, journald, rsyslog & 5 Real Fault Cases

1. Linux Logging System Overview

The article opens with a data-flow diagram showing how kernel logs (printk) flow into rsyslog via syslog protocol to /var/log/* text files, while simultaneously feeding journald (binary indexed storage at /var/log/journal/) which also forwards to rsyslog. Modern distributions (CentOS 7+, RHEL 7+, Ubuntu 16+) run a dual stack: journald (binary) + rsyslog (text).

1.1 Component Responsibilities

Kernel : printk output → ring buffer (text)

journald : collects all service, kernel, user-process logs → /var/log/journal/ (binary)

rsyslog : receives syslog protocol, filters, forwards → /var/log/*.log (text)

auditd : audit logs → /var/log/audit/audit.log (text)

Services : nginx, tomcat, mysql custom logs → /var/log/<service>/ (custom)

2. Kernel Logs (dmesg)

2.1 Viewing Commands

# All kernel logs
dmesg
# Recent errors
dmesg --level=err,crit,alert,emerg
# OOM events
dmesg | grep -i 'out of memory'
# Network interface
dmesg | grep -i eth
# Real-time follow
dmesg -w
# Human-readable timestamps
dmesg -T

Output format:

[ 5.123456] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

— bracket shows seconds since boot, not wall-clock time.

2.2 Common Keywords

out of memory

: OOM Killer triggered segfault: process segmentation fault TCP: out of memory: TCP memory exhausted net_ratelimit: network logs rate-limited NIC Link is Up/Down: NIC state changes I/O error: disk I/O errors EXT4-fs error: filesystem errors BUG: scheduling while atomic: kernel scheduling anomaly hardware name: hardware identifier for kernel bug reports

2.3 Clearing dmesg

dmesg -C  # requires root

Risk: clears ring buffer history but does not affect journald's persistent copy.

3. journald & journalctl

3.1 Configuration ( /etc/systemd/journald.conf )

[Journal]
Storage=persistent
SystemMaxUse=4G
SystemKeepFree=1G
SystemMaxFileSize=200M
MaxRetentionSec=2month
ForwardToSyslog=yes
RateLimitIntervalSec=30s
RateLimitBurst=1000

Key parameters: Storage=persistent (default /var/log/journal/), volatile (memory only), auto (try persistent, fallback to memory), SystemMaxUse (disk cap), ForwardToSyslog=yes (forward to rsyslog), MaxRetentionSec (retention time).

3.2 Common journalctl Commands

journalctl                          # all logs
journalctl -k                       # kernel logs
journalctl -u nginx                 # specific service
journalctl -u nginx --since today
journalctl --since "1 hour ago"
journalctl -p err                   # by priority (emerg=0..debug=7)
journalctl _PID=1234                # by PID
journalctl _UID=1000                # by UID
journalctl -f                       # follow
journalctl -b                       # current boot
journalctl -b -1                    # previous boot
journalctl --list-boots             # list boots
journalctl /usr/sbin/nginx          # by executable
journalctl /dev/sda                 # by device
journalctl -n 100                   # last 100 lines
journalctl -o json                  # JSON output
journalctl -o short                 # short format

3.3 Advanced Usage

# Top erroring services today
journalctl -p err --since today | awk '{print $5}' | sort | uniq -c | sort -rn | head
# OOM events
journalctl -k | grep -i oom
# SSH logins in time window
journalctl -u sshd --since "2026-06-12 09:00" --until "2026-06-12 10:00"
# Merge kernel + service logs
journalctl -k -u nginx
# Verify integrity
journalctl --verify
# Disk usage
journalctl --disk-usage

3.4 Cleanup

journalctl --vacuum-size=1G
journalctl --vacuum-time=1month
journalctl --vacuum-time=1week

Warning: --vacuum-size deletes immediately; ensure logs not needed for analysis.

4. rsyslog & /var/log/*

4.1 Configuration ( /etc/rsyslog.conf )

module(load="imuxsock")
module(load="imjournal")
module(load="imklog")

template(name="TraditionalFormat" type="string" string="%timegenerated% %syslogtag% %msg% %inputname%
")

*.info;mail.none;authpriv.none;cron.none    /var/log/messages
authpriv.*                                   /var/log/secure
mail.*                                       -/var/log/maillog
cron.*                                       /var/log/cron
*.emerg                                      :omusrmsg:*
uucp,news.crit                               /var/log/spooler
local7.*                                     /var/log/boot.log

*.* @10.0.10.10:514  # remote UDP

Fields: * all priorities; authpriv.* auth; mail.* mail; cron.* cron; *.emerg emergencies; leading - means async write (no immediate fsync).

4.2 /var/log/* Reference Table

Key files: /var/log/messages (general), /var/log/secure (SSH/sudo/login), /var/log/cron, /var/log/boot.log, /var/log/dmesg (post-boot kernel), /var/log/audit/audit.log, /var/log/yum.log, /var/log/wtmp (logins, binary), /var/log/btmp (failed logins), /var/log/lastlog (last login per user), /var/log/tallylog / faillock (pam counters), plus service directories for nginx, httpd, mysqld, redis, tomcat.

4.3 Binary Login Logs

last          # wtmp
last -F       # full times
last -i       # show IPs
lastb         # btmp failed logins
lastlog       # per-user last login

4.4 Text Log Analysis Examples

grep 'Failed password' /var/log/secure | tail -20
grep sudo /var/log/secure
grep -i 'cmd' /var/log/cron | tail
grep 'Installed' /var/log/yum.log
grep 'Accepted' /var/log/secure
grep -i 'disconnect\|error\|timeout' /var/log/secure

5. Service Log Locations

Web: nginx ( access.log, error.log), Apache ( access_log, error_log), Tomcat ( catalina.out, dated logs), PHP-FPM ( error.log). Databases: MySQL ( mysqld.log, slow query via slow_query_log_file), Redis ( redis.log), PostgreSQL ( /var/log/postgresql/), MongoDB ( mongod.log), Elasticsearch ( /var/log/elasticsearch/). Cache/MQ: Memcached (no default log, -vv for verbose), Kafka ( server.log), RabbitMQ ( /var/log/rabbitmq/), Nginx stream ( stream.log). K8s/Docker: kubelet ( kubelet.log or journalctl -u kubelet), kube-apiserver, containerd, Docker, kubectl logs. Monitoring: Prometheus, Grafana, Node Exporter, AlertManager under /var/log/.

6. Log Rotation (logrotate)

6.1 Global Config ( /etc/logrotate.conf )

daily
rotate 30
create 0640 root root
dateext
dateformat -%Y%m%d
compress
notifempty
missingok
sharedscripts
include /etc/logrotate.d

6.2 Nginx Example ( /etc/logrotate.d/nginx )

/var/log/nginx/*.log {
    daily
    rotate 30
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

6.3 Tomcat Example (copytruncate)

/var/log/tomcat/*.log {
    daily
    rotate 30
    missingok
    notifempty
    compress
    delaycompress
    copytruncate
}

6.4 Key Parameters

daily/weekly/monthly/yearly

: frequency rotate N: keep N archives size 100M: rotate at size compress/delaycompress: gzip, delay one cycle notifempty: skip empty files create 0640 root root: new file perms missingok: no error if missing copytruncate: copy then truncate (for apps that don't reopen) postrotate/endscript: commands after rotation sharedscripts: run once for all matched files dateext/dateformat: date suffix

6.5 copytruncate vs postrotate

copytruncate : copy + truncate; simple but risks lost writes between copy and truncate.

postrotate + signal : rename + create new + notify service (e.g., Nginx USR1); preferred.

Risk: broken postrotate command can stop logging. Guard with /dev/null 2>&1.

6.6 Testing

logrotate -vf /etc/logrotate.d/nginx  # force
logrotate -d /etc/logrotate.d/nginx   # dry-run

6.7 Disk Not Released After Rotation

Symptom: process holds deleted file descriptor. Diagnose: lsof | grep deleted. Fix: use postrotate to signal reopen (preferred), or copytruncate (risk of loss), or restart service (last resort).

7. Troubleshooting Cases (Phenomenon → Root Cause → Fix → Verify → Retrospective)

7.1 SSH Brute Force

Phenomenon : alert "SSH failed logins > 100/min". Commands : extract attacker IPs ( awk '{print $11}'), usernames ( awk '{print $9}'), time distribution. Root cause : public SSH port exposed to automated scanners. Fix : immediate iptables drop, enable fail2ban ( systemctl enable --now fail2ban), move SSH behind bastion. Verify : fail2ban-client status sshd, iptables -L INPUT -n | wc -l.

7.2 Disk Full

Phenomenon : "No space left on device". Commands : df -h, df -i, du -sh /* | sort -h | tail, find / -type f -size +1G, lsof | grep deleted. Key finding : /var/log/messages 50 GB due to missing logrotate. Fix : > /var/log/messages (safe truncate) or truncate -s 0; add logrotate config with daily, size 100M, postrotate HUP rsyslog. Verify : df -h, ls -la /var/log/messages*. Retrospective : all logs need rotation; disk alerts at 70/85/95%.

7.3 OOM Killer

Phenomenon : process disappears, monitoring shows "process not exist". Commands : dmesg | grep -i 'out of memory', journalctl -k | grep -i oom. Output example :

Killed process 1234 (java) total-vm:8388608kB, anon-rss:4194304kB, oom_score_adj:0

. Root cause : Java process exceeded cgroup limit or system memory. Fix : set oom_score_adj=-100 for critical processes, adjust cgroup MemoryHigh/MemoryMax, add RAM, tune JVM heap/GC. Verify : monitor journalctl -k | grep -i oom, check /proc/<pid>/oom_score.

7.4 NIC Packet Loss

Phenomenon : high P99 latency. Commands : ip -s link show eth0 (RX errors/dropped/overruns), ethtool -S eth0 | grep -i 'drop\|err\|miss', ethtool -g eth0 (ring buffer), ethtool eth0 (speed/duplex), dmesg | grep -i eth. Key indicators : dropped counter rising, ring buffer full, NIC negotiated at lower speed. Root cause : ring buffer too small for burst traffic. Fix : ethtool -G eth0 rx 4096 tx 4096, sysctl -w net.core.netdev_max_backlog=300000, disable TSO/GSO, upgrade to 10GbE if saturated. Verify : iperf3 -c 10.0.0.1 -t 60, re-check ip -s link.

7.5 systemd Service Restart Loop

Phenomenon : service starts then fails, status failed. Commands : systemctl status myapp -l, journalctl -u myapp -n 100 --no-pager, systemd-analyze blame | grep myapp. Key indicators : Active: failed, Main PID: ... (code=exited, status=...), restart counter is at N. Root cause : app cannot connect to DB/Redis on startup; systemd default restart masks issue. Fix : (1) fix app to wait for dependencies; (2) adjust systemd: Restart=on-failure, RestartSec=10, StartLimitBurst=5, StartLimitIntervalSec=300; (3) add After=network.target mysql.service redis.service, Requires=mysql.service redis.service. Verify : systemctl restart myapp, journalctl -u myapp -f.

8. Log Analysis Toolchain

8.1 grep/awk/sed

grep -c 'Failed password' /var/log/secure
grep -E 'Failed password|Invalid user' /var/log/secure
grep -A 5 -B 5 'Oops' /var/log/messages
awk '{print $11}' /var/log/secure | sort | uniq -c
sed -n '100,200p' /var/log/messages

8.2 goaccess

goaccess -f /var/log/nginx/access.log -c          # terminal
 goaccess /var/log/nginx/access.log -o /var/www/report.html -c  # HTML

8.3 ELK/PLG

Referenced as previously covered.

8.4 logwatch

yum install -y logwatch
# /etc/logwatch/conf/logwatch.conf
MailTo = [email protected]
Detail = High
Service = All
Range = yesterday

8.5 Custom Monitoring Scripts

Bash scripts for OOM alerting (parsing dmesg) and SSH failure counting (parsing /var/log/secure with date filter), emailed via cron.

9. Log Security & Integrity

9.1 Tamper Protection

chattr +a /var/log/messages          # append-only
chattr -R +a /var/log/               # recursive (breaks logrotate)

Warning: +a interferes with logrotate rename/create; apply only to critical logs.

9.2 Audit Access

auditctl -w /var/log/ -p wa -k log-access
ausearch -k log-access

9.3 Centralized Collection

Remote syslog (rsyslog client → central rsyslog)

ELK/PLG (Filebeat/Promtail)

Splunk (commercial)

Cloud log services (Aliyun, Tencent)

*.* @10.0.10.10:514          # UDP
*.* @@(o)10.0.10.10:6514     # TLS TCP

9.4 Detecting Post-Intrusion Log Wiping

sort -k1,2 /var/log/secure | head          # timestamp jumps
ls -la /var/log/                           # mtime anomalies
awk '{print $1,$2,$3}' /var/log/secure | uniq -c | head  # gaps

Signs: mtime changes, sudden time jumps, log volume drops.

10. Time Sync & Log Correlation

10.1 Time Sync

timedatectl
chronyc tracking
chronyc sources -v
timedatectl set-ntp true

10.2 Cross-Host Timelines

journalctl --utc
date +%Z
# Configure timezone in ELK/Loki

10.3 Trace ID Correlation

Nginx $request_id + journald _BOOT_ID enable cross-host request tracing. In Loki/ELK: {job="nginx"} |= "req_id=abc".

11. Quick-Reference Commands

journalctl -xe
journalctl -b
dmesg
cat /var/log/messages
cat /var/log/secure
cat /var/log/cron
last
lastb
journalctl -u nginx
journalctl -u mysqld
vmstat 1
mpstat 1
iostat -x 1
ss -s
ss -tulnp
netstat -an
tcpdump -i eth0 -nn port 80
lsof
strace -p <pid>
free -h
df -h
df -i
ps -ef
pstree -p
top
htop

12. Retention & Archival Policy

System logs (messages, secure): 90 days

Kernel logs (dmesg): 30 days

App logs (nginx, tomcat): 30-90 days

DB slow query: 30 days

Audit logs: 1 year

Backup logs: 1 year

Compliance logs: 1-7 years

12.2 Archive to Object Storage

find /var/log/ -name "*.gz" -mtime +30 -exec aws s3 cp {} s3://log-archive/$(hostname)/{} \;

12.3 Cleanup Script (cron daily)

find /var/log/ -name "*.gz" -mtime +90 -delete
find /var/log/ -name "*.log.[0-9]" -mtime +30 -delete

13. Log System Observability

13.1 Collector Health

systemctl status filebeat
systemctl status promtail
systemctl status systemd-journald
systemctl status rsyslog

13.2 Key Metrics

node_filesystem_avail_bytes

(log disk space) journal_entries_total (journald writes)

rsyslog_messages_processed_total

filebeat_events_*

13.3 Prometheus Alert Rules

groups:
- name: log
  rules:
  - alert: LogDiskHigh
    expr: (node_filesystem_avail_bytes{mountpoint="/var/log"} / node_filesystem_size_bytes{mountpoint="/var/log"}) < 0.15
    for: 5m
    labels:
      severity: warning
  - alert: JournaldDown
    expr: up{job="journald_exporter"} == 0
    for: 1m
    labels:
      severity: critical
  - alert: LogAgentDown
    expr: up{job="filebeat"} == 0
    for: 1m
    labels:
      severity: warning

14. Common Misconceptions

"Delete logs to save space" — short-sighted; loses forensic evidence.

"More logs = better" — excessive logs hurt performance, hide signals; use levels (DEBUG/INFO/WARN/ERROR).

"Centralized logs always beat local" — local journald has binary index for fast queries; central suits long-term/cross-host.

"logrotate never breaks" — bad postrotate loses logs; wrong size triggers excessive rotation; always test with logrotate -d.

"Time drift doesn't matter" — all analysis, alerting, ELK timelines depend on consistent time; enforce chrony, monitor drift.

15. Cheat Sheet: Symptom → Log Source

Table mapping 20+ symptoms (boot failure, SSH fail/success, sudo, cron, kernel errors, OOM, NIC issues, disk full, service failure, nginx/MySQL/Redis/Tomcat errors, audit, intrusion traces, cron changes, kernel modules, SUID files) to exact log files and commands (e.g., journalctl -xb, /var/log/secure, dmesg | grep oom, ethtool -S eth0, journalctl -u <service>, ausearch, last/lastb/history, find / -perm -u+s).

16. Summary & Troubleshooting Workflow

Logging stack: kernel → journald → rsyslog → text logs → logrotate → remote. Know which layer to inspect. Checklist per layer: kernel ( dmesg, journalctl -k), system services ( journalctl -u, /var/log/messages), auth ( /var/log/secure, lastb), cron ( /var/log/cron), service-native logs, audit ( /var/log/audit/audit.log, ausearch).

Recommended 8-step workflow: (1) monitoring — what metric, when; (2) time — journalctl --since at anomaly start; (3) service — status & logs; (4) resources — CPU/mem/disk/net; (5) processes — anomalous PIDs, start time, cmdline; (6) network — connections, ports, drops; (7) config — recent changes; (8) changes — package updates, config mgmt, deploy records.

17. Advanced journald

17.1 Custom Fields

C:

sd_journal_send("MESSAGE=hello", "PRIORITY=5", "MY_FIELD=custom", NULL);

Python: journal.send("hello", PRIORITY=5, MY_FIELD="custom"). Query: journalctl MY_FIELD=custom.

17.2 Remote Forward

[Journal]
ForwardToSyslog=yes
ForwardToWall=no
# rsyslog side:
module(load="imuxsock" SysSock.ImJournal="/run/systemd/journal/syslog")

17.3 Rate Limiting

RateLimitIntervalSec=30s
RateLimitBurst=1000

Prevents runaway service from filling journald.

17.4 stdout/stderr Capture

journalctl -u myapp -o cat   # pure text, no metadata
# Default: StandardOutput=journal

18. Advanced rsyslog

18.1 Templates

template(name="MyFormat" type="string" string="%timegenerated:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%
")
*.* /var/log/myapp.log;MyFormat

18.2 Filtering

:programname, isequal, "nginx" /var/log/nginx-app.log
& stop
:msg, contains, "DEBUG" stop

18.3 Queues & Buffering

action(type="omfwd" Target="10.0.10.10" Port="514" Protocol="udp" queue.type="LinkedList" queue.size="10000")

Async queues prevent network hiccups from blocking rsyslog.

18.4 RELP (Reliable TCP)

module(load="omrelp")
action(type="omrelp" Target="10.0.10.10" Port="20514")

RELP over TCP guarantees no message loss.

19. journald Index Internals

19.1 Indexed Fields

_PID

, _UID, _GID, _COMM, _EXE, _CMDLINE, _SYSTEMD_UNIT, _BOOT_ID, _MACHINE_ID, _HOSTNAME, _TRANSPORT (journal/syslog/stdout/kernel), MESSAGE, PRIORITY, SYSLOG_IDENTIFIER.

19.2 Index Files

Located at /var/log/journal/<machine-id>/: system.journal (100 MB–4 GB), user-1000.journal.

19.3 Corruption Recovery

journalctl --verify
journalctl --repair   # may lose corrupted entries

19.4 Cleanup

journalctl --vacuum-size=1G
journalctl --vacuum-time=1month

Irreversible; confirm before running.

20. Kernel Log Depth

20.1 Ring Buffer Size

cat /proc/sys/kernel/printk_ratelimit
cat /proc/sys/kernel/printk_devkmsg
dmesg -s 65536          # runtime increase
# Permanent:
echo 'kernel.printk = 4 4 1 7' >> /etc/sysctl.d/99-kernel.conf

Four values: console, default, minimum, boot-time.

20.2 kdump (Crash Dump)

yum install -y kexec-tools
# /etc/kdump.conf
path /var/crash
core_collector makedumpfile -l --message-level 1 -d 31
# /etc/sysconfig/kdump
KDUMP_COMMANDLINE_APPEND="irqpoll maxcpus=1"
KDUMP_BACKUP_ACTION="reboot"
KDUMP_KEEP_OLD_DUMPS=5
# Trigger:
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

Used for kernel panic/hardware error post-mortem.

20.3 Module/Subsystem Logs

dmesg | grep -i 'module\|init\|firmware'
dmesg | grep -i usb
dmesg | grep -i 'sd\|nvme\|sata'

21. Container Logging

21.1 Viewing

docker logs <container>
docker logs --tail 100 -f <container>
kubectl logs <pod> -c <container>
kubectl logs --previous <pod>          # previous instance
kubectl logs -f <pod> --since=1h

21.2 Docker Log Drivers

json-file

(default, local)

journald

syslog

awslogs

/ gcplogs /

azblob

fluentd

none

Configure in /etc/docker/daemon.json:

{