Master Systemd: Build Production‑Ready Service Files from Scratch

Overview

Systemd has evolved from a simple init system into a full‑stack manager that handles services, logging, timers, device management and network configuration. Modern distributions (Ubuntu 24.04, RHEL 9, Debian 12) ship Systemd 256+ with cgroup v2 as the default. This article aims to teach you how to write a production‑grade .service file that includes resource limits, security hardening, health checks and log management.

Systemd Architecture and Core Concepts

Systemd is more than an init system

The PID 1 process /usr/lib/systemd/systemd parses Unit files, resolves dependencies, starts/stops/monitors services and manages cgroup resources. Supporting components such as journald, logind, udevd, networkd and resolved run alongside it.

Units, Targets, Slices

Units are the basic objects managed by Systemd; the file suffix determines the type (e.g., .service, .socket, .timer, .mount, .target, .slice, .path, .device). Targets are logical groups similar to SysV runlevels. Slices are cgroup abstractions for resource grouping.

Service   .service   Manage daemons               nginx.service
Socket    .socket    Socket activation            sshd.socket
Timer     .timer     Scheduled jobs                logrotate.timer
Mount     .mount     Mount points                 home.mount
Target    .target    Logical grouping             multi-user.target
Slice     .slice     Resource grouping             user.slice
Path      .path      File monitoring              cups.path
Device    .device    Device management (udev)     auto‑generated

Unit File Locations

Three directories are searched in order of precedence: /etc/systemd/system/ – highest, for administrator overrides. /run/systemd/system/ – runtime generated units. /usr/lib/systemd/system/ – package defaults (never edit directly).

Use systemctl edit to create drop‑in files under /etc/systemd/system/ instead of editing the original file.

Service File Structure

A standard .service file consists of three sections:

[Unit]
# metadata and dependencies

[Service]
# execution parameters

[Install]
# enable/disable information

[Unit] Section

Defines description, documentation, ordering and dependency directives such as After=, Wants=, Requires=. Example:

Description=My Application Service
Documentation=https://docs.example.com
After=network-online.target postgresql.service
Wants=network-online.target

[Service] Section

Key directives:

Type= – determines how Systemd decides the service is started. Options include simple, exec, forking, oneshot, notify. notify is recommended for services that can call sd_notify().

ExecStart= , ExecStartPre= , ExecReload= – commands to launch, pre‑run checks and reload.

User= / Group= – run as a non‑root user.

Restart= – restart policy (e.g., on-failure).

RestartSec= , RestartSteps= , RestartMaxDelaySec= – control back‑off timing (available in Systemd 256+).

TimeoutStartSec= , TimeoutStopSec= – start/stop timeouts.

WatchdogSec= – health‑check heartbeat interval (requires the program to send sd_notify()).

CPUQuota= , CPUWeight= , MemoryMax= , MemoryHigh= , MemorySwapMax= – cgroup v2 resource limits.

LimitNOFILE= , LimitNPROC= , TasksMax= – RLIMITs.

NoNewPrivileges= , ProtectSystem= , ProtectHome= , PrivateTmp= , ReadWritePaths= , ReadOnlyPaths= – security sandboxing.

StandardOutput= , StandardError= , SyslogIdentifier= , LogRateLimitIntervalSec= , LogRateLimitBurst= – journald logging configuration.

Service Type Details

Five main types:

Type      Startup determination                     Typical use
simple    ExecStart process considered ready immediately   Go binaries, Node.js
exec      ExecStart must return successfully before ready   Same as simple but stricter
forking   Parent forks then exits; child becomes main       Traditional daemons (nginx, mysql)
oneshot   ExecStart finishes before service is active      One‑off tasks (init scripts)
notify    Service calls sd_notify(READY=1) when fully initialized   Programs that support sd_notify

Choosing the wrong type can cause systemctl start to report success while the process never started, or cause restart logic to fail.

Dependency Management

Two dimensions:

Order – After=/Before= only affect start order.

Strength – Wants= (weak), Requires= (strong), BindsTo= (bind lifecycle), Requisite= (assert).

Production recommendation: use Wants= + After= for most dependencies; reserve Requires= for truly mandatory services.

Restart Policies and Rate Limiting

Common values for Restart=: no – never restart (default). on-success – restart only on exit code 0. on-failure – restart on non‑zero exit, signal kill, timeout, watchdog. always – restart regardless of reason.

Typical production setting: Restart=on-failure with RestartSec=5s. Add exponential back‑off using RestartSteps and RestartMaxDelaySec. Guard against endless loops with StartLimitIntervalSec and StartLimitBurst.

Watchdog (Process Health Check)

Configure WatchdogSec=30s (or any interval). The service must periodically call sd_notify(0, "WATCHDOG=1"). Example in Go:

import "github.com/coreos/go-systemd/v22/daemon"

func watchdogLoop() {
    interval, _ := daemon.SdWatchdogEnabled(false)
    if interval == 0 { return }
    ticker := time.NewTicker(interval/2)
    for range ticker.C {
        daemon.SdNotify(false, daemon.SdNotifyWatchdog)
    }
}

Resource Limits (cgroup v2)

CPU:

[Service]
CPUQuota=200%      # up to 2 cores (hard limit)
CPUWeight=50       # soft weight for contention
AllowedCPUs=0-3   # optional pinning

Memory:

[Service]
MemoryMax=2G      # hard OOM kill limit
MemoryHigh=1536M  # soft limit, kernel will reclaim
MemoryMin=256M    # guaranteed minimum
MemorySwapMax=0   # disable swap

IO, file descriptor, process count limits are similarly expressed with IOWeight=, IOReadBandwidthMax=, LimitNOFILE=, LimitNPROC=, etc.

Security Hardening

Zero‑cost hardening that should be enabled for every service:

[Service]
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectKernelLogs=yes
ProtectClock=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
ReadWritePaths=/var/lib/myapp /var/log/myapp
ReadOnlyPaths=/etc/myapp

Run systemd-analyze security myapp.service to obtain a numeric exposure score; aim for a score of 3 or lower.

Logging with journald

Standard output and error are captured automatically. Service‑level logging options:

[Service]
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp
LogLevelMax=info
LogRateLimitIntervalSec=30s
LogRateLimitBurst=10000

Global journald configuration ( /etc/systemd/journald.conf) controls persistence, disk usage, rotation and compression. Example:

[Journal]
Storage=persistent
SystemMaxUse=2G
SystemMaxFileSize=128M
SystemKeepFree=4G
Compress=yes

Timer Units (Replacing Cron)

Timers provide integrated logging, dependency handling, resource limits and missed‑run recovery ( Persistent=yes). Example timer for a daily backup:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Database Backup Timer

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=yes
RandomizedDelaySec=15min
AccuracySec=1s

[Install]
WantedBy=timers.target

# /etc/systemd/system/db-backup.service
[Unit]
Description=Database Backup Job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup-db.sh
User=backup
Group=backup
MemoryMax=512M
CPUQuota=50%
ProtectSystem=strict
PrivateTmp=yes
NoNewPrivileges=yes
ReadWritePaths=/var/backups

Manage timers with systemctl enable --now db-backup.timer, list with systemctl list-timers, and trigger manually with systemctl start db-backup.service.

Socket Activation

Systemd can listen on a socket and start the service on the first connection, providing start‑up acceleration, on‑demand activation and zero‑downtime restarts.

# /etc/systemd/system/myapp.socket
[Unit]
Description=My Application Socket

[Socket]
ListenStream=0.0.0.0:8080
Backlog=4096

[Install]
WantedBy=sockets.target

The matching myapp.service can read the socket from file descriptor 3. Go’s net package and Systemd’s sd_listen_fds() both support this pattern.

Complete Production‑Grade Service Example (Go Web Service)

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application API Server
Documentation=https://docs.example.com/myapp
After=network-online.target postgresql.service redis.service
Wants=network-online.target postgresql.service redis.service
ConditionPathExists=/etc/myapp/config.yaml

[Service]
Type=notify
NotifyAccess=main
ExecStartPre=/usr/local/bin/myapp validate --config /etc/myapp/config.yaml
ExecStart=/usr/local/bin/myapp serve --config /etc/myapp/config.yaml
ExecReload=/bin/kill -s HUP $MAINPID
User=myapp
Group=myapp
WorkingDirectory=/var/lib/myapp
EnvironmentFile=-/etc/myapp/env
Environment=GOMAXPROCS=4
Environment=GIN_MODE=release
Restart=on-failure
RestartSec=5s
RestartSteps=5
RestartMaxDelaySec=60s
StartLimitIntervalSec=300
StartLimitBurst=5
TimeoutStartSec=30s
TimeoutStopSec=60s
WatchdogSec=30s
CPUQuota=200%
CPUWeight=100
MemoryMax=2G
MemoryHigh=1536M
MemorySwapMax=0
TasksMax=4096
LimitNOFILE=65536
LimitNPROC=4096
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectKernelLogs=yes
ProtectClock=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
ReadWritePaths=/var/lib/myapp /var/log/myapp
ReadOnlyPaths=/etc/myapp
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp
LogRateLimitIntervalSec=30s
LogRateLimitBurst=10000

[Install]
WantedBy=multi-user.target

Deployment Steps

# 1. Create a system user
sudo useradd --system --no-create-home --shell /usr/sbin/nologin myapp

# 2. Create required directories
sudo mkdir -p /var/lib/myapp /var/log/myapp /etc/myapp
sudo chown myapp:myapp /var/lib/myapp /var/log/myapp

# 3. Deploy binary and config
sudo cp myapp /usr/local/bin/
sudo chmod 755 /usr/local/bin/myapp
sudo cp config.yaml /etc/myapp/config.yaml

# 4. Install the service file
sudo cp myapp.service /etc/systemd/system/

# 5. Reload systemd
sudo systemctl daemon-reload

# 6. Enable and start
sudo systemctl enable --now myapp.service

# 7. Verify
systemctl status myapp.service
journalctl -u myapp.service -n 50 --no-pager

Java Service Example

# /etc/systemd/system/myapp-java.service
[Unit]
Description=My Java Application
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
ExecStart=/usr/bin/java \
    -Xms512m -Xmx1536m \
    -XX:+UseZGC \
    -jar /opt/myapp/myapp.jar \
    --spring.config.location=/etc/myapp/
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
Restart=on-failure
RestartSec=10s
StartLimitIntervalSec=300
StartLimitBurst=3
TimeoutStartSec=120s
TimeoutStopSec=60s
MemoryMax=3G
MemoryHigh=2560M
MemorySwapMax=0
CPUQuota=400%
LimitNOFILE=65536
TasksMax=4096
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ReadWritePaths=/var/lib/myapp /var/log/myapp /tmp
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp-java

[Install]
WantedBy=multi-user.target

Checklist for Service Files

[ ] Correct Type selected for the program?
[ ] Dependencies declared with After= and Wants= (or Requires= if truly mandatory)?
[ ] Running as non‑root (User/Group set)?
[ ] Restart policy defined (Restart=, RestartSec=, StartLimitBurst=)?
[ ] Timeouts reasonable (TimeoutStartSec/TimeoutStopSec)?
[ ] Memory limits (MemoryMax, MemoryHigh) appropriate?
[ ] CPU limits (CPUQuota) set?
[ ] LimitNOFILE high enough (≥65536 for high‑concurrency services)?
[ ] NoNewPrivileges=yes added?
[ ] ProtectSystem=strict and ReadWritePaths listed?
[ ] LogRateLimitBurst configured?
[ ] systemd-analyze security score ≤3?

Advanced Topics

systemd‑nspawn – lightweight containers that use the same cgroup, journald and network stack as the host. Managed via machinectl and the [email protected] template. Ideal for build‑environment isolation or wrapping legacy applications without Docker.

Portable Services – package an application and its dependencies into an OS image (raw or squashfs). Use portablectl attach to mount the image and generate a Service unit automatically. Provides dependency isolation without a full container runtime, useful for edge devices.

systemd‑sysext and ComposeFS – overlay read‑only extensions on immutable root filesystems (e.g., Fedora CoreOS, Flatcar). Enables modular OS updates and per‑node extensions while keeping the base OS unchanged.

Reference Material

systemd official documentation (man pages).

systemd.service(5) – complete list of Service directives.

systemd.exec(5) – execution environment and security options.

systemd.resource-control(5) – cgroup resource limits.

Arch Wiki – systemd practical guide.

systemd‑nspawn(1) – container runtime documentation.

Portable Services documentation.

systemd‑sysext(8) – system extensions management.