Mastering systemd on Linux: From boot to logs, a hands‑on guide to avoid sleepless nights

Why systemd matters

During a high‑traffic e‑commerce promotion, a core payment service stopped responding even though its process was still running; the root cause was a mis‑configured Restart policy that prevented systemd from restarting the failed service. The author uses this incident to illustrate that systemd is the central engine of Linux system management and that misunderstanding it can hide the true reason a service fails.

What systemd is

Systemd replaced SysVinit in 2010 to address slow sequential boot and tangled dependency scripts. It launches independent units in parallel, reducing boot time by more than half, and abstracts all resources (services, mounts, devices, timers, sockets) as Units managed uniformly.

Unit types and configuration

Key unit types include Service , Target , Timer , Mount/Automount , and Socket . Unit files reside in /etc/systemd/system/ or /usr/lib/systemd/system/. A typical Service unit for Nginx looks like:

[Unit]
Description=Nginx HTTP Server
After=network.target
Requires=network.target

[Service]
Type=forking
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/usr/sbin/nginx -s stop
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

The [Unit] section declares dependencies; After only orders start‑up, while Requires creates a hard dependency. The author recounts a failure caused by using only After=network.target, which let Nginx start before the network was ready.

Essential systemctl commands

Common commands are grouped by scenario:

View system status

# systemctl --version
# systemctl get-default
# systemctl list-units
# systemctl list-units --type=service
# systemctl status nginx.service

Service management

# systemctl start nginx
# systemctl stop nginx
# systemctl restart nginx
# systemctl reload nginx
# systemctl is-enabled nginx
# systemctl enable nginx
# systemctl enable --now nginx

Note: enable only creates a symlink for next boot; it does not start the service immediately.

Unit file handling

# systemctl cat nginx
# systemctl edit nginx
# systemctl daemon-reload

The daemon-reload step is crucial after editing unit files; forgetting it leaves the old configuration in effect.

Service Type options and common pitfalls

The Type field determines how systemd judges service readiness. Five values are described: simple – default; the process started by ExecStart is the main process. forking – for traditional daemons that fork (e.g., Nginx, Apache). oneshot – runs a command once and exits. notify – the service notifies systemd via sd_notify() when ready. dbus – registers a name on D‑Bus.

A case study shows a Java service mistakenly set to simple while it forks; systemd interpreted the parent exit as failure. Changing to forking or using notify resolves the issue.

Logging with journalctl

Systemd’s journald centralizes logs, replacing scattered syslog files. Useful commands include:

# journalctl
# journalctl -f
# journalctl -u nginx.service
# journalctl -n 100
# journalctl --since today
# journalctl --since "2024-01-01 10:00" --until "2024-01-01 12:00"
# journalctl -p err

Logs are kept in memory by default; creating /var/log/journal makes them persistent across reboots.

Timer units versus cron

Timer units replace cron with tighter integration to systemd. Advantages include precise scheduling, automatic journald logging, failure‑triggered actions, and random delays to avoid load spikes. A simple daily cleanup timer:

# cleanup.timer
[Unit]
Description=每天清理临时文件

[Timer]
OnCalendar=daily
Persistent=true
Unit=cleanup.service

[Install]
WantedBy=timers.target

The OnCalendar field accepts flexible expressions (e.g., daily, hourly, Mon *-*-* 06:00:00). Persistent=true ensures missed runs are executed after a reboot.

Real‑world troubleshooting cases

Case 1 – Service start timeout – systemctl start hung in activating. The fix was changing Type=simple to Type=notify or adding TimeoutStartSec=300.

Case 2 – Service restarted outside systemd – A script manually relaunched a process, leaving systemd unaware, so status showed inactive. The solution: let systemd manage the process.

Case 3 – Boot order problem – Service A failed because its dependency B wasn’t ready. Correcting After and Requires declarations resolved the issue.

Best‑practice checklist

Choose the correct Type for each service.

Declare both After and Requires for true dependencies; use network-online.target for guaranteed network availability.

Run systemctl daemon-reload after any unit file change.

Use journalctl -u <service> to view all logs for a service instead of searching /var/log.

Prefer Timer units over cron for scheduled tasks; they log automatically and can recover missed runs.

Set resource limits ( MemoryMax, CPUQuota) for production services to prevent a single service from exhausting system resources.

Understanding systemd’s design—units, declared dependencies, unified logging, and integrated timers—turns a complex system into a predictable, debuggable framework.