Master Linux System Log Analysis to Quickly Troubleshoot Issues

Introduction

Log files are the first‑hand data for troubleshooting Linux servers. The article targets junior to mid‑level sysadmins and explains the most important log sources, common analysis commands and typical failure scenarios.

1. Linux Log Basics

1.1 Log system architecture

Linux logging consists of the syslog daemon, rsyslog service and systemd‑journal. Older CentOS 6 uses syslog, CentOS 7+ and most modern distributions use rsyslog, while recent releases (CentOS 8, Ubuntu 20.04+, Debian 10+) also provide systemd‑journal.

1.2 Important log files

/var/log/messages : main system log (kernel, applications).

/var/log/syslog : Ubuntu/Debian main log.

/var/log/dmesg : kernel ring buffer, useful for hardware and boot issues.

/var/log/secure : authentication and authorization events.

/var/log/audit/audit.log : SELinux/audit events (requires audit service).

/var/log/yum.log , /var/log/cron , /var/log/maillog , /var/log/httpd/ or /var/log/nginx/ , /var/log/mysql/ or /var/log/mariadb/ , /var/log/boot.log : other common logs.

1.3 Basic journalctl usage

View all logs: journalctl Follow new entries: journalctl -f Time range:

# last 10 minutes
journalctl --since "10 minutes ago"
# specific interval
journalctl --since "2026-05-13 10:00:00" --until "2026-05-13 11:00:00"

Filter by service, kernel, priority, etc.

1.4 Log levels

Standard syslog levels 0‑7 (emerg, alert, crit, error, warning, notice, info, debug). Focus on error, warning, crit, alert, emerg when troubleshooting.

2. Common Log‑Analysis Commands

2.1 grep family

Basic search, OR/AND patterns, inverse match, line numbers, context, count, case‑insensitive, regex examples.

2.2 awk basics

Column extraction, conditional filtering, counting, formatting output.

2.3 sed basics

Search‑and‑replace, delete lines, print specific lines.

2.4 Command combinations

Pipe grep to awk for counting errors per hour, find most active IPs, count login attempts, etc. Includes a full Bash script for SSH login statistics.

3. Typical Failure‑Scenario Walkthroughs

3.1 Scenario 1 – Server cannot be reached via SSH

Step‑by‑step: alternative access (VNC/IPMI), network connectivity tests (ping, nc, telnet), service status (systemctl status sshd), start service, check configuration (sshd -t), verify listening port, firewall rules, connection limits, common problems and their log‑based diagnostics.

3.2 Scenario 2 – Service start failure (Nginx example)

Check service status, manual start to see error output, examine error and system logs, common causes (port conflict, permission, SELinux, missing libraries) and corresponding fixes.

3.3 Scenario 3 – Disk space exhaustion

df -h, find large files (>100 MB), du to locate big directories, focus on /var/log, clean logs, yum cache, old snaps, Docker, configure logrotate.

3.4 Scenario 4 – Memory leak detection

watch free, ps aux sorted by memory, pmap for a process, jmap for Java, valgrind for native binaries, typical leak causes list.

4. Security Log Analysis

4.1 SSH login analysis

grep Accepted/Failed in /var/log/secure, count successes/failures, top offending IPs, brute‑force patterns.

4.2 fail2ban deployment

Installation, basic jail.local configuration for sshd and nginx, common commands to query status, ban/unban IPs.

4.3 sudo usage audit

grep sudo in secure log, view sudoers with visudo, summarize command usage.

4.4 SELinux audit

ausearch -m avc -ts recent

for AVC events, audit2allow conversion, getenforce, setenforce to switch modes.

5. Application Log Analysis

5.1 Nginx

Tail error.log, grep specific error strings, analyse error trends, use awk on access.log to count status codes, top IPs, top URLs, average and slowest response times.

5.2 MySQL

Inspect error.log, enable and analyse slow-query.log with mysqldumpslow, view binary logs with mysqlbinlog.

5.3 Docker

docker logs

, journalctl CONTAINER_NAME=… for containerd, crictl logs, grep errors.

6. Advanced Techniques

6.1 Custom analysis script

Full Bash script ( analyze_system.sh) that gathers system errors, SSH stats, disk usage, memory/CPU, failed services and writes results to /tmp.

6.2 logwatch automation

Install, run manually or output to file, schedule via cron for daily reports.

6.3 Centralised logging

Configure rsyslog client to forward logs ( *.* @@remote-server:514), server side receiver with imtcp and template, brief ELK stack components list (Filebeat, Logstash, Elasticsearch, Kibana).

6.4 Real‑time monitoring and alerting

Inotify‑wait script that watches /var/log/secure for failed password lines and sends email alerts.

7. Common Log Patterns

7.1 OOM Killer detection

Search dmesg or /var/log/messages for “out of memory”, “killed process”, analyse killed PID, check memory usage with free and ps.

7.2 Disk I/O problems

grep “io timeout”, ext4 errors, dmesg for sd* or ata errors.

7.3 Network issues

netstat -s

for retransmits, dmesg for NIC errors, ip -s link for packet loss.

7.4 Service crashes

Search for core dumps, segfault messages, ABRT reports.

8. Real‑World Cases

8.1 Reboot storm caused by OOM

Log investigation reveals frequent OOM Killer entries, MySQL process killed, oversized innodb_buffer_pool_size, solution: reduce buffer, monitor memory, verify with free and ps.

8.2 Intrusion detection via logs

Massive SSH failed attempts, successful login from unknown user, privilege‑escalation checks, file system scans for setuid binaries and hidden files, mitigation: block attacker IP, remove malicious files, reinstall, enforce key‑based SSH, enable fail2ban.

9. Summary

Log analysis is a core sysadmin skill. Key points: know log locations, master grep/awk/sed/journalctl, adopt a systematic workflow (time → keyword → frequency → trend), close the troubleshooting loop, and keep a command cheat‑sheet.

# Quick reference
cat /var/log/messages
tail -f /var/log/messages
journalctl -f
grep "error" /var/log/messages
awk '{print $5}' /var/log/messages
sed -n '100p' /var/log/messages
last reboot
free -h
ss -s