Why Is Your Linux Server Booting So Slowly? Practical Boot Chain Optimization

Background and Phenomenon

Slow server boot is a real problem: in a physical data center it delays hand‑over of machines, in cloud environments it postpones service recovery after scaling or failover, and in Kubernetes it hurts pod scheduling efficiency.

Typical symptoms include 3‑5 minutes before the login prompt, systemd-analyze time showing >2 minutes total, or fast hardware (SSD, multi‑core CPU) still booting slower than older machines.

Phase 1: Measurement – Locate the Bottleneck

1.1 Quantify each stage

固件（BIOS/UEFI）
→ bootloader（GRUB）
→ 内核引导（kernel）
→ initramfs
→ systemd（基础服务和用户服务）
→ 业务服务就绪

Run systemd-analyze time to get a breakdown, e.g.:

Startup finished in 1min 30.284s (firmware) + 5.123s (bootloader) + 2.234s (kernel) + 15.678s (initramfs) + 1min 12.456s (userspace) = 3min 6.775s

The output shows which phase consumes the most time.

1.2 Identify slow services

systemd-analyze blame | head -30

Typical output:

1min 2.345s NetworkManager-wait-online.service
    45.123s mysqld.service
    32.456s docker.service
    28.901s redis.service
    15.678s postfix.service

Focus on the top entries.

1.3 Trace the critical chain

systemd-analyze critical-chain

Shows the longest dependency path, e.g. mysqld.service blocked by NetworkManager-wait-online.service for over a minute.

1.4 Inspect kernel timestamps

dmesg -T | head -100

Large gaps between timestamps (e.g., 65 seconds) indicate long waits in firmware, RAID initialization, or network storage mounting.

Phase 2: Step‑by‑Step Optimizations

2.1 Firmware (BIOS/UEFI) Optimizations

Disable unused peripherals (floppy, serial, parallel).

Turn off PXE boot if not needed.

Disable VT‑d/AMD‑V when hardware does not support it.

Check RAID controller health and BBU status; a failing BBU disables write cache and slows I/O.

Consider disabling Secure Boot if signature verification adds noticeable delay.

Typical actions:

# Dell: sudo omreport chassis alerts
# HPE: sudo hpacucli ctrl all show status

2.2 GRUB Optimizations

Reduce the menu timeout (default 5 s) to 1 s or 0 s for headless servers.

Remove unused kernel entries.

If GRUB password protection is unnecessary, disable it.

Example:

# Edit /etc/default/grub
GRUB_TIMEOUT=1
sudo update-grub   # Debian/Ubuntu
sudo grub2-mkconfig -o /boot/grub2/grub.cfg   # RHEL/CentOS

2.3 Kernel Boot Optimizations

Trim initramfs size by omitting unnecessary modules (extra RAID, network, filesystem drivers).

Remove unneeded kernel parameters such as quiet and splash for servers.

If the root filesystem is encrypted, consider using a key file for automatic unlock (with physical security controls).

Example to rebuild a lean initramfs:

# /etc/dracut.conf.d/custom.conf
omit_dracutmodules+="network nfs"
sudo dracut -f --omit "network nfs" /boot/initramfs-$(uname -r).img $(uname -r)

2.4 initramfs Stage Optimizations

Disable LVM if not required.

Remove unnecessary RAID or network drivers.

For Btrfs roots, avoid automatic balance during boot.

Detect and edit the initramfs modules, then regenerate.

2.5 systemd Service Optimizations

NetworkManager-wait-online.service : either fix the underlying network delay or shorten its timeout via an override file ( TimeoutStartSec=10).

Replace network-online.target with network.target when full network readiness is not required.

Parallelize services by removing unnecessary After= dependencies.

Delay non‑critical services (logging, monitoring, backup) using OnBootSec=2min in a timer unit.

Example override for a service:

# /etc/systemd/system/redis.service.d/override.conf
[Unit]
After=network.target local-fs.target
Wants=local-fs.target

2.6 Database Service Optimizations

Store data directories on local disks instead of NFS.

Disable InnoDB buffer‑pool pre‑load for very large pools ( innodb_buffer_pool_load_at_startup=OFF).

Reduce PostgreSQL checkpoint frequency if appropriate.

2.7 Docker Daemon Optimizations

Use overlay2 storage driver.

Prune unused images regularly.

Configure log driver limits.

Example daemon.json:

{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {"max-size": "100m", "max-file": "3"},
  "live-restore": true
}

2.8 Remote Filesystem (NFS) Optimizations

Add _netdev and async options in /etc/fstab to delay mount until the network is up.

Set reasonable timeouts ( timeo=30,retrans=3,soft).

# /etc/fstab
10.0.0.100:/data /data nfs4 defaults,_netdev,async,timeo=30,retrans=3,soft 0 0

2.9 Disable Unnecessary System Services

Postfix, tuned, abrtd, cups, etc., can be disabled if not used.

sudo systemctl disable --now postfix
sudo systemctl mask cups

Phase 3: Visualize with bootchart

Install systemd-bootchart, enable it, reboot, and inspect the generated PNG to see CPU and I/O usage per process.

sudo yum install systemd-bootchart -y   # RHEL/CentOS
sudo apt install systemd-bootchart -y # Debian/Ubuntu
sudo systemctl enable bootchart

Phase 4: Verify Optimizations

Record baseline and post‑optimization times:

# Before
systemd-analyze time > /tmp/boot_before.txt
systemd-analyze blame | head -30 > /tmp/blame_before.txt
# After changes and reboot
systemd-analyze time > /tmp/boot_after.txt
systemd-analyze blame | head -30 > /tmp/blame_after.txt
# Compare
diff /tmp/boot_before.txt /tmp/boot_after.txt
diff /tmp/blame_before.txt /tmp/blame_after.txt

Also capture kernel timestamps with dmesg -T to verify reductions in each phase.

Phase 5: Continuous Monitoring

Create a one‑shot systemd unit that writes boot‑time metrics to a Prometheus textfile collector, enabling alerts when user‑space boot exceeds a threshold.

# /etc/systemd/system/boot-metric.service
[Unit]
Description=Export boot time metric to Prometheus
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/bash -c '\
  FIRM=$(systemd-analyze time | grep firmware | awk "{print $2}" | sed "s/s//"); \
  KERN=$(systemd-analyze time | grep kernel | awk "{print $2}" | sed "s/s//"); \
  USER=$(systemd-analyze time | grep userspace | awk "{print $2}" | sed "s/s//"); \
  echo "linux_boot_firmware_seconds ${FIRM:-0}" > /run/boot-metric.prom; \
  echo "linux_boot_kernel_seconds ${KERN:-0}" >> /run/boot-metric.prom; \
  echo "linux_boot_userspace_seconds ${USER:-0}" >> /run/boot-metric.prom'

[Install]
WantedBy=multi-user.target

Enable and start the unit, then let Prometheus scrape the file.

Conclusion

The core workflow is: systemd-analyze time → systemd-analyze blame → journalctl -u <service> → dmesg -T → targeted optimizations → re‑measure → monitor. In most cases a single slow service dominates the delay; fixing it yields the biggest gain. Always validate changes with before/after data to ensure the optimization is effective.