Linux Server Time Sync Best Practices with NTP and chrony

Problem Background

Incorrect system time on Linux servers leads to log timestamp mismatches, certificate validation failures, MySQL replication lag, missed cron jobs, and compliance warnings. The article starts by listing real‑world incidents that illustrate the impact of time drift.

Core Time Concepts

System Time (Software Clock) : Maintained by the kernel, exposed via date, gettimeofday, clock_gettime(CLOCK_REALTIME).

Hardware Clock (RTC) : Battery‑backed chip accessed via /dev/rtc0 and the hwclock command. It can run in UTC mode (recommended) or local mode.

Monotonic Clock : Never jumps; used for timeouts and rate‑limiting. NTP never adjusts this clock.

Timezone : Stored in /etc/localtime (symlink to /usr/share/zoneinfo/…) and configured with timedatectl.

NTP Protocol Basics

NTP (RFC 5905) runs over UDP 123. Key terms include stratum, offset, delay, jitter, reachability, poll interval, and leap‑second status. The article shows the four‑step packet exchange and how chrony computes offset and delay.

chrony vs ntpd

chrony advantages : Converges in seconds, handles intermittent networks, better clock‑skew estimation, default slewing (no step jumps), shorter config, built‑in NTS support.

ntpd advantages : Long‑standing, default on many legacy distributions, strict RFC compliance, extensive third‑party audit material.

Choosing an Implementation

For new clusters, use chrony. Legacy environments may keep ntpd if migration is not feasible, but test before switching.

Installation

sudo apt update
sudo apt install -y chrony
sudo systemctl enable --now chronyd
chronyc tracking

Similar commands are provided for RHEL/CentOS ( dnf install -y chrony) and openEuler.

Typical Configuration (chrony.conf)

# /etc/chrony.conf
pool pool.ntp.org iburst maxsources 4
server ntp.aliyun.com iburst
server time1.cloud.tencent.com iburst
makestep 1.0 3          # step if offset > 1 s on first 3 updates
rtcsync                 # keep hardware clock in sync
allow 192.168.0.0/16
allow 10.0.0.0/8
deny 0.0.0.0/0
log tracking measurements statistics
logdir /var/log/chrony

Several ready‑to‑use templates are described, e.g., internal‑only NTP, K8s node, edge device, and NTS‑encrypted setups.

Verification and Monitoring

Check service status: systemctl status chronyd Inspect synchronization: chronyc tracking and chronyc sources -v Force convergence: chronyc waitsync 60 0.001 Export metrics with chrony_exporter (offset_seconds, frequency_ppm, reach, leap_status, etc.) and create Prometheus alerts for large offset or loss of reachability.

Kubernetes Nodes

K8s components (kube‑apiserver, metrics‑server, etcd) are highly sensitive to clock drift. The article recommends a node‑specific config that only pulls time from internal NTP servers and disables NTP serving on the node.

Troubleshooting Common Issues

All sources show ^? or ^x: check firewall UDP 123, DNS resolution, or restrict directives.

Offset jumps repeatedly: verify source stability, CPU load, hypervisor time‑keeping, or reduce poll frequency. chronyc reports zero offset but date is far off: look for seccomp blocks on settimeofday or external time‑changing processes.

Reach stays 0: confirm correct server IP, that the upstream listens on UDP 123, and that bidirectional firewall rules are open.

Compliance (e.g., China GB/T 22239‑2)

Time sources must be trustworthy for audit logging, intrusion detection, and data integrity. The guide outlines a checklist: at least two independent NTP sources, internal stratum‑1 servers, periodic chronyc tracking screenshots, and automated Ansible or Bash scripts for compliance verification.

Rollback Strategies

# Backup current config
sudo cp /etc/chrony.conf /etc/chrony.conf.bak.$(date +%Y%m%d)
# Restore previous version if needed
sudo cp /etc/chrony.conf.bak.$(date +%Y%m%d) /etc/chrony.conf
sudo systemctl restart chronyd
chronyc tracking

Procedures are also provided for restoring system time from the hardware clock and for handling K8s node rollbacks.

Key Takeaways

Unify all servers to UTC; keep hardware clock in UTC mode.

Use a layered source hierarchy: internal NTP → cloud provider NTP → public pool.

Prefer slewing; allow stepping only for large initial offsets.

Enable NTS for cryptographic protection of NTP traffic.

Monitor offset, reach, and stratum with Prometheus; alert on >100 ms offset or loss of reachability.

Apply the provided configuration templates to physical servers, VMs, containers, edge devices, and K8s nodes.

Automate compliance checks with Ansible or Bash scripts.

Use monotonic clocks in applications to avoid side effects of system‑time jumps.