From Beginner to Pro: Write a Production‑Ready systemd Service File

Overview

The article aims to teach readers how to write a production‑grade systemd .service file that includes resource limits, security hardening, health checks, and proper logging. It starts from systemd’s architecture and ends with a complete, verified template.

Systemd Architecture and Core Concepts

Systemd is more than an init system; it is the PID 1 process that manages services, logging (journald), timers, devices, and networking. Units are stored in three locations with decreasing priority: /etc/systemd/system/ (admin overrides), /run/systemd/system/ (runtime), and /usr/lib/systemd/system/ (package defaults).

Unit Types

Service : manages daemons (e.g., nginx.service)

Socket : socket activation (e.g., sshd.socket)

Timer : scheduled tasks (e.g., logrotate.timer)

Target : logical grouping, similar to runlevels

Slice : cgroup hierarchy for resource grouping

Most everyday work involves .service, .socket, and .timer units.

Service File Structure

A standard .service file consists of three sections:

[Unit]
# metadata and dependencies

[Service]
# execution parameters

[Install]
# enable/disable behavior

[Unit] Section

Defines description, documentation, and dependencies. Use After= and Wants= to express start order and weak dependencies. Example:

After=network-online.target postgresql.service
Wants=network-online.target postgresql.service

[Service] Section

The core of the unit. Important directives include:

Type : determines how systemd decides the service is ready. Recommended types are notify (program sends sd_notify) or exec. simple is the default but may report success even if the binary path is wrong. forking is for traditional daemons and requires PIDFile=. oneshot is for one‑time tasks.

ExecStart , ExecReload , ExecStop : command lines.

Restart and related options control automatic restarts.

TimeoutStartSec , TimeoutStopSec : prevent hanging starts or stops.

WatchdogSec : requires the program to send periodic heartbeats.

Resource limits : CPUQuota=, MemoryMax=, MemoryHigh=, IOWeight=, LimitNOFILE=, etc.

Security hardening : NoNewPrivileges=yes, ProtectSystem=strict, PrivateTmp=yes, ReadWritePaths=, SystemCallFilter=, and many others.

Logging : StandardOutput=journal, LogRateLimitBurst=, LogLevelMax=info.

[Install] Section

Specifies how the unit is enabled, typically with WantedBy=multi-user.target.

Service Type Decision Tree

Does the program fork and exit the parent?
├─ Yes → Type=forking + PIDFile=
└─ No → Does it support sd_notify?
    ├─ Yes → Type=notify (preferred)
    └─ No → Is it a one‑time task?
        ├─ Yes → Type=oneshot (optionally RemainAfterExit=yes)
        └─ No → Type=exec (recommended) or simple

Restart Policies and Rate Limiting

Common production choice is Restart=on-failure. Combine with exponential back‑off using the newer options available in systemd 256+:

RestartSec=5s
RestartSteps=5
RestartMaxDelaySec=60s
StartLimitIntervalSec=300
StartLimitBurst=5

These settings avoid rapid restart loops while still trying to recover from transient failures.

Resource Limits and Memory Management

Use both hard and soft limits:

# Hard memory cap – OOM kill if exceeded
MemoryMax=2G
# Soft limit – kernel will reclaim before OOM
MemoryHigh=1536M
# Disable swap for predictability
MemorySwapMax=0
# CPU quota (percentage of a single core)
CPUQuota=200%
# CPU weight for fair sharing when contended
CPUWeight=50

Typical high‑concurrency services also need larger file descriptor limits:

LimitNOFILE=65536

Security Hardening

Zero‑cost protections:

NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectKernelLogs=yes
ProtectClock=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
ReadWritePaths=/var/lib/myapp /var/log/myapp
ReadOnlyPaths=/etc/myapp

These settings dramatically reduce the impact of a compromised process.

Logging with journald

Standard output and error are captured automatically. Configure rate limiting to avoid log storms:

LogRateLimitIntervalSec=30s
LogRateLimitBurst=10000

Global journald rotation is controlled via /etc/systemd/journald.conf (e.g., SystemMaxUse=2G, Compress=yes).

Timers – Replacing Cron

Systemd timers integrate with journald, support dependency management, and can recover missed runs with Persistent=yes. Example of a daily backup timer:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Database Backup Timer

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=yes
RandomizedDelaySec=15min
AccuracySec=1s

[Install]
WantedBy=timers.target

The corresponding service defines the actual backup command and can reuse the same resource‑limit and security settings as any other service.

Socket Activation

Systemd can listen on a socket and start the service on first connection, providing faster boot and zero‑downtime restarts. Minimal socket unit example:

# /etc/systemd/system/myapp.socket
[Unit]
Description=My Application Socket

[Socket]
ListenStream=0.0.0.0:8080
Backlog=4096

[Install]
WantedBy=sockets.target

The associated .service does not need special code; the program simply reads the socket from file descriptor 3 (standard for socket‑activated services).

Production‑Ready Service Template (Go Example)

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application API Server
Documentation=https://docs.example.com/myapp
After=network-online.target postgresql.service redis.service
Wants=network-online.target postgresql.service redis.service
ConditionPathExists=/etc/myapp/config.yaml

[Service]
Type=notify
NotifyAccess=main
ExecStartPre=/usr/local/bin/myapp validate --config /etc/myapp/config.yaml
ExecStart=/usr/local/bin/myapp serve --config /etc/myapp/config.yaml
ExecReload=/bin/kill -s HUP $MAINPID
User=myapp
Group=myapp
WorkingDirectory=/var/lib/myapp
EnvironmentFile=-/etc/myapp/env
Environment=GOMAXPROCS=4 GIN_MODE=release

# Restart & timeout
Restart=on-failure
RestartSec=5s
RestartSteps=5
RestartMaxDelaySec=60s
StartLimitIntervalSec=300
StartLimitBurst=5
TimeoutStartSec=30s
TimeoutStopSec=60s
WatchdogSec=30s

# Resource limits
CPUQuota=200%
CPUWeight=100
MemoryMax=2G
MemoryHigh=1536M
MemorySwapMax=0
TasksMax=4096
LimitNOFILE=65536
LimitNPROC=4096

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectKernelLogs=yes
ProtectClock=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
ReadWritePaths=/var/lib/myapp /var/log/myapp
ReadOnlyPaths=/etc/myapp

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp
LogRateLimitIntervalSec=30s
LogRateLimitBurst=10000

[Install]
WantedBy=multi-user.target

The template can be adapted for Java, Python, or any other language by adjusting Type, JVM memory flags, and timeout values.

Deployment Steps

# 1. Create a system user
sudo useradd --system --no-create-home --shell /usr/sbin/nologin myapp

# 2. Create required directories and set ownership
sudo mkdir -p /var/lib/myapp /var/log/myapp /etc/myapp
sudo chown myapp:myapp /var/lib/myapp /var/log/myapp

# 3. Deploy binary and config
sudo cp myapp /usr/local/bin/
sudo cp config.yaml /etc/myapp/

# 4. Install the service file
sudo cp myapp.service /etc/systemd/system/

# 5. Reload systemd and enable the service
sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service

# 6. Verify
systemctl status myapp.service
journalctl -u myapp.service -n 50 --no-pager

Checklist for Service Files

Correct Type matching program behavior?

Dependencies expressed with After and Wants?

Runs as non‑root ( User / Group)?

Appropriate Restart and StartLimit settings?

Reasonable TimeoutStartSec / TimeoutStopSec?

Hard ( MemoryMax) and soft ( MemoryHigh) memory limits?

CPU quota to prevent monopolizing cores? LimitNOFILE high enough for high‑concurrency services? NoNewPrivileges=yes enabled? ProtectSystem=strict with explicit ReadWritePaths?

Log rate limiting configured?

Run systemd-analyze security and keep score ≤ 3?

Advanced Topics

Beyond basic services, systemd offers:

systemd‑nspawn : lightweight containers that share the host kernel but have isolated filesystems and cgroups.

Portable Services : bundle an application and its dependencies into an OS image that can be attached with portablectl, providing a middle ground between plain units and full containers.

systemd‑sysext and Composefs : overlay immutable root filesystems with additional extensions, useful for immutable OS deployments.

References

systemd official man pages (systemd.service(5), systemd.exec(5), systemd.resource-control(5))

Arch Wiki – systemd

Lennart Poettering’s “systemd for Administrators” series

systemd‑nspawn(1) documentation

Portable Services design documents

systemd‑sysext(8) manual