Mastering Chrony: Fast, Precise Time Sync for Distributed Systems

Overview

Accurate time synchronization is essential for distributed systems. Chrony is the modern replacement for ntpd and is the default NTP client on most Linux distributions, offering faster startup, higher precision and better tolerance to network jitter.

Key Features

Fast sync : initial synchronization completes in minutes instead of tens of minutes.

High precision : typically achieves microsecond‑level accuracy.

Strong adaptability : handles intermittent connectivity and large clock jumps.

Low resource usage : daemon consumes only a few megabytes of RAM.

Prerequisites

Operating system: CentOS 7+ or Ubuntu 18.04+ (Chrony is pre‑installed on recent releases).

Chrony version: 3.5+ (4.x recommended for NTS support).

Network: ability to reach internal or public NTP servers (UDP 123).

Hardware clock (RTC) must be functional; virtual machines rely on the host clock.

Installation

# CentOS/RHEL
sudo yum install -y chrony

# Ubuntu/Debian
sudo apt update
sudo apt install -y chrony

# Disable other NTP services
sudo systemctl stop ntpd
sudo systemctl disable ntpd
sudo systemctl mask systemd-timesyncd

Configuration

Basic /etc/chrony.conf

# /etc/chrony.conf (CentOS) or /etc/chrony/chrony.conf (Ubuntu)

# Public NTP servers (replace with nearest servers)
server ntp1.example.com iburst
server ntp2.example.com iburst
pool pool.ntp.org iburst maxsources 4

# Internal NTP server (recommended for enterprises)
server 192.168.1.1 iburst prefer

# Record clock drift
driftfile /var/lib/chrony/drift

# Allow large jumps on first sync (threshold 1 s, up to 3 attempts)
makestep 1.0 3

# Keep system clock in sync with hardware RTC
rtcsync

# Logging
logdir /var/log/chrony
log measurements statistics tracking

# Access control (allow trusted subnets only)
allow 10.0.0.0/8
allow 192.168.0.0/16

# Bind management commands to localhost for security
bindcmdaddress 127.0.0.1
bindcmdaddress ::1

Parameter notes : iburst – send multiple requests at startup to accelerate initial sync. prefer – give this server higher priority. makestep 1.0 3 – allow a step adjustment if offset exceeds 1 s during the first three measurements. rtcsync – periodically write the system time to the hardware RTC.

Enterprise NTP hierarchy

# Level‑1 NTP server (directly sync to public/GPS)
server ntp1.internal iburst
server ntp2.internal iburst
local stratum 3 orphan
allow 10.0.0.0/8
keyfile /etc/chrony.keys

# Level‑2 client servers
server ntp-master-1.internal iburst prefer
server ntp-master-2.internal iburst
local stratum 10
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync

Special environments

Virtual machines (large jumps possible): makestep 1.0 -1 Containers (need SYS_TIME capability): run with docker run --cap-add SYS_TIME … High‑precision (finance/telecom) – combine with PTP: refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0 Offline environments – use local hardware clock:

refclock RTC

Service Management

# Start and enable Chrony
sudo systemctl start chronyd
sudo systemctl enable chronyd

# Verify service status
systemctl status chronyd

# Check synchronization status
chronyc tracking
chronyc sources -v

Monitoring Script Example

#!/bin/bash
# Thresholds (seconds)
WARN_THRESHOLD=0.1   # 100 ms
CRIT_THRESHOLD=1.0   # 1 s

OFFSET=$(chronyc tracking | awk '/System time/ {print $4}')
OFFSET_ABS=${OFFSET#-}
LEAP_STATUS=$(chronyc tracking | awk '/Leap status/ {print $4}')

if (( $(echo "$OFFSET_ABS > $CRIT_THRESHOLD" | bc -l) )); then
  echo "CRITICAL: Time offset exceeds $CRIT_THRESHOLD s"
  exit 2
elif (( $(echo "$OFFSET_ABS > $WARN_THRESHOLD" | bc -l) )); then
  echo "WARNING: Time offset exceeds $WARN_THRESHOLD s"
  exit 1
fi

if [ "$LEAP_STATUS" != "Normal" ]; then
  echo "WARNING: Leap status is $LEAP_STATUS"
  exit 1
fi

echo "OK: Time offset $OFFSET s, Leap status $LEAP_STATUS"
exit 0

Real‑World Cases

TiDB cluster clock drift

Problem : a TiDB node was ~5 s ahead, causing transaction failures ("timestamp is ahead of PD").

Root cause : mis‑configured NTP server address (DNS resolution failure).

Resolution :

# Update /etc/chrony.conf with reachable IPs
cat > /etc/chrony.conf <<'EOF'
server 10.0.1.100 iburst prefer
server 10.0.1.101 iburst
makestep 1.0 -1
rtcsync
driftfile /var/lib/chrony/drift
EOF

systemctl restart chronyd
chronyc makestep
chronyc tracking
chronyc sources

Virtual‑machine clock drift

Problem : minutes‑level drift on KVM VMs despite Chrony reporting sync.

Investigation steps :

Check VM clock source:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

Verify host clock accuracy: date and chronyc tracking on the host.

Inspect VM XML clock configuration (e.g., <clock offset='utc'>…</clock>).

Fixes :

# Option 1: Switch VM clocksource to tsc
echo "tsc" > /sys/devices/system/clocksource/clocksource0/current_clocksource

# Option 2: Adjust Chrony for VM environments
cat >> /etc/chrony.conf <<'EOF'
makestep 1.0 -1   # allow step adjustments at any time
minpoll 4
maxpoll 6
EOF

systemctl restart chronyd

Best Practices

Configure at least three NTP sources for redundancy; Chrony automatically selects the best.

Prefer servers with the lowest network latency. Example latency test:

for server in ntp1 ntp2 ntp3; do
  echo -n "$server: "
  chronyc sourcestats | grep $server | awk '{print $6}'
 done

Deploy a layered NTP hierarchy (public → tier‑1 → tier‑2 → all servers) in large environments.

Enable NTP authentication:

# Generate a key
chronyc keygen 1 SHA256 >> /etc/chrony.keys
# Enable in config
keyfile /etc/chrony.keys
server ntp1.internal iburst key 1

Restrict client access with allow / deny and bind management commands to localhost.

Use Network Time Security (NTS) when supported (Chrony 4.x+):

server time.cloudflare.com iburst nts

Monitor key metrics (offset, frequency, root delay, stratum, source count) and set alert thresholds, e.g., offset > 100 ms warning, > 1 s critical.

Diagnostic Checklist

Check system time: date, timedatectl.

Verify Chrony service: systemctl is-active chronyd.

Inspect sources: chronyc sources or chronyc sources -v.

Review offset and leap status: chronyc tracking.

Test network connectivity to each NTP server (ping or nc -uz UDP 123).

Ensure firewall allows outbound UDP 123.

Recovery Procedure

Identify the scope of the issue (single node or batch).

Temporarily set the time manually (use with caution) or force a step with chronyc makestep.

Fix the root cause: correct /etc/chrony.conf, resolve network/DNS problems, or replace faulty hardware.

Validate recovery: run chronyc tracking until offset is within the acceptable range.

Update monitoring and alert rules to prevent recurrence.

Reference Parameters

server

: specify an NTP server; add iburst for fast startup. pool: use an NTP pool for automatic multi‑server selection. makestep: define when large jumps are allowed (threshold seconds, count). rtcsync: keep hardware RTC synchronized with the system clock. driftfile: store clock drift rate for faster future sync. local: configure the host as an NTP server for other clients. allow/deny: access control for client queries. minpoll/maxpoll: set polling interval range (2^n seconds).