Fix Server Time Drift with Chrony: A Step‑by‑Step Linux Synchronization Guide

1. Define the problem: time zone, system clock and sync status are not the same

Linux may display a time zone such as Asia/Shanghai while the kernel clock is still synchronized to UTC. Changing the time zone does not fix clock drift; you must verify the displayed time, the kernel clock, the RTC hardware clock, and whether Chrony has selected a trusted source.

1.1 Check current time, time zone and NTP sync flag

timedatectl status

Look for Local time , Universal time , RTC time , Time zone and System clock synchronized . The flag only indicates that the system believes it is synchronized; you still need Chrony’s tracking output to see the actual offset and reference source.

1.2 Inspect Chrony’s estimated offset, hierarchy and last update

chronyc tracking

Ensure Reference ID is not 00000000. A lower Stratum is better, but not always the best. Pay attention to Last offset, RMS offset and Leap status. If Last offset keeps growing, investigate the source quality or network rather than masking the issue with a one‑off makestep.

1.3 List candidate sources and the currently selected one

chronyc sources -v

The leading ^ marks network sources, * marks the currently selected source, + marks usable sources, - marks excluded sources, and ? indicates unreachable or insufficient samples. At least one * should be present, and you should configure multiple sources across different failure domains.

1.4 Verify service name, status and start‑up failures

systemctl status chronyd --no-pager -l

RHEL, Rocky, AlmaLinux and CentOS Stream use chronyd.service; Debian and Ubuntu use chrony.service. Fix any configuration‑file path, permission or port‑conflict errors before proceeding.

1.5 Confirm installed package and service unit per distribution

if command -v rpm >/dev/null 2>&1; then
  rpm -q chrony
  systemctl list-unit-files | grep -E '^chronyd\.service'
elif command -v dpkg-query >/dev/null 2>&1; then
  dpkg-query -W -f='${binary:Package} ${Version}
' chrony
  systemctl list-unit-files | grep -E '^chrony\.service'
else
  echo "Unrecognized package manager; verify manually"
fi

This step confirms the package is installed and avoids using the wrong service name on Debian‑based systems.

2. Plan the time‑source hierarchy and deployment scope

A robust production topology uses two or more internal time servers that obtain UTC from an approved upstream or GNSS/PTP source; business nodes only query the internal servers. This limits external exposure, simplifies auditing, and keeps the cluster stable if the external network fails.

Before rollout, record the existing configuration, current offset and the time‑sensitivity of critical services (databases, message queues, authentication systems). Perform a gray‑scale deployment on a small node set during low‑traffic windows. If the initial sync shows a large offset, notify application owners and consider pausing jobs that depend on absolute time.

2.1 Install Chrony and enable the service on RHEL‑based systems

sudo dnf install -y chrony
sudo systemctl enable --now chronyd
sudo systemctl is-enabled chronyd
sudo systemctl is-active chronyd

Ensure no other NTP client (ntpd, openntpd, systemd‑timesyncd) is running; only one time‑sync client should be active.

2.2 Install Chrony and enable the service on Debian/Ubuntu

sudo apt-get update
sudo apt-get install -y chrony
sudo systemctl enable --now chrony
sudo systemctl is-enabled chrony
sudo systemctl is-active chrony

Check whether systemd-timesyncd is enabled by default and disable it if Chrony is to be used.

2.3 Verify there are no competing time‑sync services

systemctl --type=service --all | grep -E 'chronyd|chrony|ntpd|openntpd|systemd-timesyncd'

Identify any existing time‑sync services before switching; back up configurations and plan a rollback before disabling the old service.

3. Configure the client: backup, replace and verify

Use the baseline configuration file /etc/chrony.conf for RHEL‑based systems or /etc/chrony/chrony.conf for Debian/Ubuntu. Always back up the current file with a timestamp before editing.

3.1 Backup configuration with a timestamped copy

#!/usr/bin/env bash
set -euo pipefail
CONF_FILE="/etc/chrony.conf"
BACKUP_DIR="/var/backups/chrony"
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
sudo install -d -m 0750 "$BACKUP_DIR"
sudo cp -a "$CONF_FILE" "$BACKUP_DIR/chrony.conf.$STAMP"
printf 'backup=%s
' "$BACKUP_DIR/chrony.conf.$STAMP"

The script creates a traceable backup; the backup directory should be part of the host‑level backup policy.

3.2 Example baseline configuration

# /etc/chrony.conf (RHEL example)
pool <NTP_POOL> iburst maxsources 4
server <NTP_SERVER> iburst prefer

driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
minsources 2

log tracking measurements statistics
logdir /var/log/chrony

pool

is suitable for DNS names that resolve to multiple upstreams; server is for a fixed source. iburst speeds up initial sampling, makestep 1.0 3 allows a one‑second step correction only in the first three updates, and minsources 2 requires at least two sources before updating.

3.3 Atomically replace the configuration while preserving permissions

#!/usr/bin/env bash
set -euo pipefail
CONF_FILE="/etc/chrony.conf"
TMP_FILE="$(mktemp)"
trap 'rm -f "${TMP_FILE}"' EXIT
sudo tee "${TMP_FILE}" >/dev/null <<'EOF'
pool <NTP_POOL> iburst maxsources 4
server <NTP_SERVER> iburst prefer
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
minsources 2
log tracking measurements statistics
logdir /var/log/chrony
EOF
sudo install -o root -g root -m 0644 "${TMP_FILE}" "${CONF_FILE}"

The script writes the new file to a temporary location, then installs it with the correct ownership and mode.

3.4 Validate configuration syntax before reload

sudo chronyd -p -f /etc/chrony.conf

This parses the file and prints the effective configuration without starting the daemon. Adjust for version‑specific options if needed.

3.5 Restart or reload Chrony and verify status

sudo systemctl restart chronyd
sudo systemctl is-active --quiet chronyd
chronyc tracking

After a restart, ensure the service is active and that tracking shows Leap status: Normal and a reasonable Last offset.

4. Network, access control and server‑side scenarios

Clients only need outbound UDP 123 access. Servers that provide NTP must open inbound UDP 123 and configure allow rules for trusted client CIDRs.

4.1 Open NTP for a controlled client CIDR on firewalld

sudo firewall-cmd --permanent \
  --add-rich-rule='rule family="ipv4" source address="<CLIENT_CIDR>" service name="ntp" accept'
sudo firewall-cmd --reload
sudo firewall-cmd --list-rich-rules

4.2 Open NTP for a controlled client CIDR on UFW (Ubuntu/Debian)

sudo ufw allow from <CLIENT_CIDR> to any port 123 proto udp
sudo ufw status numbered

4.3 Allow internal clients to query the server

# Add only when the host should serve NTP
allow <CLIENT_CIDR>
# Do not enable "allow all" for troubleshooting; default is to reject external queries.

4.4 Verify UDP 123 listener

sudo ss -u -l -n -p | grep -E ':123\b'

When the host acts as a server, you should see a socket owned by chronyd. If another process occupies the port, stop that service first.

4.5 Capture a short NTP packet trace

sudo timeout 30 tcpdump -ni any 'udp port 123'

Use the capture to confirm that requests are sent and responses received; it does not prove source quality.

5. Diagnose out‑of‑sync conditions

Start from the local state and work outward: service status, configuration loading, source reachability, source selection, and clock jumps.

5.1 View per‑source sample count, offset and jitter

chronyc sourcestats -v

High standard deviation or low sample count may indicate network jitter, VM pause, or a bad upstream.

5.2 Show online/offline source statistics

chronyc activity

If all sources are offline or unresolved, check DNS, routing, and firewall rules before adding more public sources.

5.3 Inspect NTP negotiation details for a specific source

chronyc ntpdata <NTP_SERVER>

Replace <NTP_SERVER> with an address that appears in chronyc sources -v. This shows stratum, root delay and dispersion.

5.4 Retrieve Chrony logs since boot

sudo journalctl -u chronyd -b --no-pager -o short-iso

Look for parsing failures, unreachable sources, permission errors, or other anomalies.

5.5 Compare the parsed configuration with the package‑provided defaults

sudo chronyd -p -f /etc/chrony.conf | sed -n '1,160p'

This shows the effective configuration after includes and version‑specific processing.

5.6 Bulk compare UTC across multiple nodes

#!/usr/bin/env bash
set -euo pipefail
HOST_FILE="<HOST_LIST_FILE>"
SSH_USER="<SSH_USER>"
while IFS= read -r host; do
  [[ -z "$host" || "$host" == \\#* ]] && continue
  ssh -o BatchMode=yes -o ConnectTimeout=5 "$SSH_USER@$host" \
    'printf "%s " "$(hostname -f)"; date -u +%FT%T.%3NZ; chronyc tracking | sed -n "s/^Last offset[[:space:]]*:[[:space:]]*//p"'
 done < "$HOST_FILE"

Use this read‑only script to spot nodes with large offsets; combine with RTT measurements for a reliable assessment.

5.7 Detect virtualization‑related clock pauses

sudo journalctl -b --no-pager | grep -Ei 'clock|time.*jump|suspend|resume|vmware|hyper-v|kvm'

Correlate any pause or resume events with changes in chronyc tracking and business alerts.

6. Correction, verification and continuous inspection

Large offsets should be corrected cautiously. Chrony’s gradual correction is safe for running services; makestep performs an immediate jump and may disrupt authentication, caches, or scheduled jobs.

6.1 Perform a manual step correction during a change window

chronyc makestep
chronyc tracking
chronyc sources -v

Record the pre‑change time and business state, then verify that authentication, task scheduling and log ordering remain correct.

6.2 Create a systemd timer for periodic health checks

# /etc/systemd/system/chrony-health.service
[Unit]
Description=Check Chrony synchronization health

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/chrony-health.sh

# /etc/systemd/system/chrony-health.timer
[Unit]
Description=Run Chrony health check every five minutes

[Timer]
OnBootSec=3min
OnUnitActiveSec=5min
Persistent=true

[Install]
WantedBy=timers.target

6.3 Health‑check script used by the timer

#!/usr/bin/env bash
set -euo pipefail
TRACKING="$(chronyc tracking)"
SOURCES="$(chronyc sources -v)"
grep -q '^Leap status[[:space:]]*:[[:space:]]*Normal$' <<<"${TRACKING}"
grep -q '^\*' <<<"${SOURCES}"
printf '%s
' "${TRACKING}"
printf '%s
' "${SOURCES}"

The script exits with a non‑zero status if the leap status is not Normal or no source is marked with *, allowing systemd or external monitoring to raise an alarm.

6.4 Enable and verify the timer

sudo chmod 0755 /usr/local/sbin/chrony-health.sh
sudo systemctl daemon-reload
sudo systemctl enable --now chrony-health.timer
systemctl list-timers chrony-health.timer --all
sudo journalctl -u chrony-health.service -n 50 --no-pager

Confirm the timer is registered and that recent executions succeeded.

6.5 Verify automatic recovery after a node reboot

systemctl is-enabled chronyd
systemctl is-active chronyd
chronyc waitsync 30 0.100 0.050 2
timedatectl status

After a reboot, ensure the service is enabled, active, and that waitsync reports synchronization within the acceptable offset and jitter thresholds.

7. Rollback strategy: restore the original configuration instead of stacking changes

If a new time source is unavailable, minsources causes prolonged desynchronization, the service fails to start, or business experiences time jumps, stop further changes and restore the previously verified backup.

7.1 Restore a timestamped backup and restart the service

#!/usr/bin/env bash
set -euo pipefail
CONF_FILE="/etc/chrony.conf"
BACKUP_FILE="/var/backups/chrony/chrony.conf.<UTC_TIMESTAMP>"
sudo test -r "${BACKUP_FILE}"
sudo cp -a "${BACKUP_FILE}" "${CONF_FILE}"
sudo chronyd -p -f "${CONF_FILE}"
sudo systemctl restart chronyd
chronyc tracking

Replace <UTC_TIMESTAMP> with the actual timestamp generated in step 9. This restores the configuration without touching firewall rules or package versions.

7.2 Roll back a previously added firewalld rich rule

sudo firewall-cmd --permanent \
  --remove-rich-rule='rule family="ipv4" source address="<CLIENT_CIDR>" service name="ntp" accept'
sudo firewall-cmd --reload
sudo firewall-cmd --list-rich-rules

Execute only if the exact rule was added earlier; otherwise client machines will lose NTP access.

8. Acceptance checklist

A successful Chrony deployment must satisfy: the service unit matches the distribution, the configuration parses without errors, at least one controlled source is reachable and selected, Leap status is Normal, critical nodes’ offsets are within business‑defined tolerances, and both configuration and firewall changes have documented rollback procedures. Continuous inspection should generate alerts when the selected source disappears or synchronization degrades, ensuring the time‑service chain remains verifiable and maintainable.