Deploy a Three‑Tier Chrony Time Sync Architecture with µs‑Level Monitoring

Applicable Scenarios & Prerequisites

Applicable Business : Distributed systems, financial trading systems, log aggregation and analysis, database clusters, K8s container platforms.

Prerequisites :

Linux kernel ≥ 3.10 (≥ 4.0 recommended for hardware timestamp support)

Chrony ≥ 3.5 (4.x recommended, NTS encryption support)

Network latency to public NTP servers < 100 ms (Stratum 1)

Hardware: PTP/PPS time source (optional, financial scenario)

Root or sudo privileges to modify system time

Environment & Version Matrix

Component

Version Requirement

OS Support

Time Accuracy

Chrony

3.5+ (recommended 4.3+)

RHEL 7/8/9, Ubuntu 18.04/20.04/22.04

±50 µs (LAN)

NTPsec

1.2+

Same as above

±1 ms (WAN)

PTP (linuxptp)

2.0+

Kernel 4.0+

±1 µs (hardware support)

GPS Receiver

-

Serial/USB

±1 µs (hardware source)

Quick Checklist

Uninstall existing NTP services (ntpd / systemd‑timesyncd)

Deploy three‑tier time server architecture (Stratum 1/2/3)

Configure Chrony server as LAN NTP source

Configure Chrony client to sync with internal servers

Configure hardware clock sync (RTC)

Verify synchronization status (offset/jitter/latency)

Configure clock‑step protection (makestep threshold)

Monitor clock offset and alert (Prometheus + Node Exporter)

Test clock‑step scenarios (manual time adjustments)

Configure NTS encrypted sync (prevent time‑pollution attacks)

Implementation Steps

Step 1: Remove Conflicting NTP Services

Check existing services :

# Check ntpd
systemctl status ntpd 2>/dev/null || echo "ntpd not found"

# Check systemd‑timesyncd
systemctl status systemd-timesyncd 2>/dev/null

# Check Chrony
systemctl status chronyd 2>/dev/null

Uninstall ntpd (if installed):

# RHEL/CentOS
sudo systemctl stop ntpd
sudo systemctl disable ntpd
sudo yum remove -y ntp

# Ubuntu/Debian
sudo systemctl stop ntp
sudo systemctl disable ntp
sudo apt remove -y ntp

Disable systemd‑timesyncd :

sudo systemctl stop systemd-timesyncd
sudo systemctl disable systemd-timesyncd

Verify no conflicting services :

ps aux | grep -E "ntpd|timesyncd" | grep -v grep
# Expected: no output

Step 2: Install Chrony

RHEL/CentOS : sudo yum install -y chrony Ubuntu/Debian :

sudo apt update
sudo apt install -y chrony

Verify installation :

chronyc -v

Step 3: Deploy Three‑Tier Time Server Architecture

Architecture diagram:

┌─────────────────────────────────────────────────┐
│ Stratum 0: GPS/Atomic Clock (hardware source)   │
└───────────────────┬─────────────────────────────┘
                    │
┌───────────────────▼─────────────────────────────┐
│ Stratum 1: Public NTP servers (e.g., time.google.com) │
└───────────────────┬─────────────────────────────┘
                    │
┌───────────────────▼─────────────────────────────┐
│ Stratum 2: Internal primary NTP servers (2‑3 redundant) │
│ ntp-01.internal (10.0.0.11)                     │
│ ntp-02.internal (10.0.0.12)                     │
└───────────────────┬─────────────────────────────┘
                    │
┌───────────────────▼─────────────────────────────┐
│ Stratum 3: Business servers (hundreds to thousands) │
│ web-01, db-01, k8s-node-*, …                     │
└─────────────────────────────────────────────────┘

Key principles :

Stratum 2 (internal primary) – at least 2 redundant servers, each syncing to 3+ Stratum 1 sources.

Stratum 3 (business servers) – sync to 2+ Stratum 2 sources.

Avoid closed loops: servers must not sync to each other mutually.

Step 4: Configure Chrony Server (Stratum 2)

Edit /etc/chrony/chrony.conf (RHEL 8 / Ubuntu 20.04):

# ========== Upstream time sources (Stratum 1) ==========
# Google Public NTP
server time1.google.com iburst minpoll 4 maxpoll 6 prefer
server time2.google.com iburst minpoll 4 maxpoll 6
server time3.google.com iburst minpoll 4 maxpoll 6
server time4.google.com iburst minpoll 4 maxpoll 6

# Cloudflare NTP
server time.cloudflare.com iburst minpoll 4 maxpoll 6

# China NTP pool
pool cn.pool.ntp.org iburst maxsources 4 minpoll 4 maxpoll 6

# ========== Local fallback (low accuracy) ==========
local stratum 10

# ========== Client access control ==========
allow 10.0.0.0/8
allow 172.16.0.0/12
allow 192.168.0.0/16
deny all

# ========== Hardware clock sync ==========
rtcsync

# ========== Clock step protection ==========
# On first start, if offset > 1 s, adjust up to 3 times
makestep 1.0 3

# ========== Drift and log files ==========
driftfile /var/lib/chrony/drift
logdir /var/log/chrony
log measurements statistics tracking

# ========== Performance options ==========
bindcmdaddress 0.0.0.0
cmdallow 10.0.0.0/8

Key parameter explanations : iburst: rapid initial sync (sends 8 packets immediately). minpoll 4 maxpoll 6: poll interval 16‑64 s (default 64‑1024 s is too long). prefer: prioritize this source. allow: permit internal clients. makestep 1.0 3: if offset > 1 s, step the clock up to 3 times.

Start the Chrony service :

sudo systemctl enable chronyd
sudo systemctl start chronyd
systemctl status chronyd

Verify upstream sync :

chronyc sources -v

Step 5: Configure Chrony Client (Stratum 3)

Edit /etc/chrony/chrony.conf:

# ========== Upstream time source (internal Stratum 2) ==========
server ntp-01.internal iburst minpoll 4 maxpoll 6 prefer
server ntp-02.internal iburst minpoll 4 maxpoll 6

# ========== Hardware clock sync ==========
rtcsync

# ========== Clock step protection ==========
makestep 1.0 3

# ========== Drift and log files ==========
driftfile /var/lib/chrony/drift
logdir /var/log/chrony
log measurements statistics tracking

Restart the service: sudo systemctl restart chronyd Validate synchronization :

chronyc sources -v

Step 6: Configure Hardware Clock Sync (RTC)

# Show hardware clock
sudo hwclock --show

# Show system time
date

# Compare difference
sudo hwclock --compare

# Sync system time to hardware clock
sudo hwclock --systohc

# Verify rtcsync is enabled
grep rtcsync /etc/chrony/chrony.conf

Step 7: Verify Time Sync Status

chronyc sources
chronyc tracking
chronyc clients

Step 8: Configure Clock‑Step Protection

Typical scenario: VM migration, suspend/resume, manual time adjustments.

# Allow step on first 10 syncs if offset > 1 s
makestep 1.0 10

# Financial scenario – prohibit any step (only slew)
# makestep 0.001 -1

Step 9: Monitor Offset & Alert with Prometheus

Create a collector script /usr/local/bin/chrony_exporter.sh that runs chronyc tracking, parses fields and writes Prometheus metrics to /var/lib/node_exporter/textfile_collector/chrony.prom. Make it executable and add a cron job to run every minute.

Prometheus alert rules (example):

groups:
- name: chrony-alerts
  rules:
  - alert: ChronySyncLost
    expr: chrony_stratum > 10 or absent(chrony_stratum)
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Chrony lost synchronization (Stratum > 10)"
  - alert: ChronyHighOffset
    expr: abs(chrony_system_time_seconds) > 0.01
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Time offset exceeds 10 ms"
  - alert: ChronyHighRMSOffset
    expr: chrony_rms_offset_seconds > 0.001
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "RMS offset exceeds 1 ms"
  - alert: ChronyHighRootDelay
    expr: chrony_root_delay_seconds > 0.1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Root delay exceeds 100 ms"

Performance & Capacity

Time Accuracy Benchmarks

Scenario

Stratum

Network

Expected Accuracy

Public NTP sync

2

Latency to Stratum 1 < 50 ms

±1 ms

LAN sync

3

Latency to Stratum 2 < 5 ms

±100 µs

Data‑center (same rack)

3

Latency < 1 ms

±50 µs

PTP hardware timestamp

1

Dedicated network

±1 µs

Chrony Server Capacity

Supported clients: 10 000+ on a 4 CPU / 8 GB server.

CPU overhead per client: < 0.01 %.

Network bandwidth per client: < 1 Kbps (64 s poll interval).

Security & Compliance

Access Control

# Allow only specific subnets
allow 10.0.0.0/8
deny all

# Rate limiting (prevent DDoS)
ratelimit interval 3 burst 8

# Restrict command port access
bindcmdaddress 127.0.0.1
cmdallow 127.0.0.1
cmdallow 10.0.0.0/8

Prevent Time‑Pollution Attacks

# Enable NTS encryption
server time.cloudflare.com iburst nts
# Require at least two sources and reject those > 1 s away
minsources 2
maxdistance 1.0

Audit Logging

log tracking measurements statistics
logdir /var/log/chrony

Integrate with syslog for centralized logging.

Common Issues & Troubleshooting

Symptom

Diagnostic Command

Possible Root Cause

Quick Fix

Permanent Fix

Stratum = 16 (unsynced) chronyc sources All time sources unreachable

Check network / firewall

Add more time sources

Offset > 1 ms chronyc tracking High network latency or poor source quality

Switch to lower‑latency source

Deploy internal time servers

Client cannot sync sudo tcpdump -i any port 123 Server not allowing client (missing allow)

Add appropriate allow rule

Review firewall rules

Clock jump observed journalctl -u chronyd | grep makestep Severe system clock drift

Adjust makestep parameters

Check hardware clock / TSC stability

RTC not syncing

hwclock --compare

rtcsync

not enabled

Add rtcsync to config

—

NTS connection failure chronyc authdata NTS server unreachable / firewall block

Test TCP 4460 connectivity

Open firewall or use alternate source

Change & Rollback Playbook

Maintenance Window

Backup /etc/chrony/chrony.conf Record baseline offset with chronyc tracking Prepare rollback configuration

Notify time‑sensitive services (e.g., trading systems)

Canary Strategy

Stage 1: Validate on a single server.

sudo cp /etc/chrony/chrony.conf.new /etc/chrony/chrony.conf
sudo systemctl restart chronyd
watch -n 10 'chronyc tracking | grep "System time"'

Stage 2: Bulk rollout with Ansible.

ansible all -m copy -a "src=/etc/chrony/chrony.conf dest=/etc/chrony/"
ansible all -m systemd -a "name=chronyd state=restarted"

Health Checks

# Verify sync source
chronyc sources | grep '^\^*'
# Verify offset < 1 ms
chronyc tracking | grep "System time" | awk '{if ($4 > 0.001) exit 1}'
# Verify Stratum <= 5
chronyc tracking | grep "Stratum" | awk '{if ($3 > 5) exit 1}'

Rollback Conditions & Commands

Offset > 10 ms for 5 min

Stratum > 5

Business reports timestamp errors

# Restore previous config
sudo cp /etc/chrony/chrony.conf.backup /etc/chrony/chrony.conf
sudo systemctl restart chronyd
# Force immediate step
sudo chronyc -a makestep
chronyc tracking

Best Practices

Always use a three‑tier architecture – business servers never sync directly to public NTP.

Configure at least three upstream sources for majority voting.

Prefer low‑latency LAN sources over WAN.

Enable iburst for fast startup.

Always enable rtcsync to keep the hardware clock in sync.

Continuously monitor System time and RMS offset in production.

In financial or logging environments, disable clock jumps (use makestep 0 -1).

Regularly verify synchronization status (weekly chronyc sources).

Open firewall ports UDP 123 (NTP) and TCP 4460 (NTS).

Document topology, source selection rationale, and configuration changes.

Appendix

Full Production Configuration Templates

Stratum 2 Server – /etc/chrony/chrony.conf:

# Upstream time sources
server time1.google.com iburst minpoll 4 maxpoll 6 prefer
server time2.google.com iburst minpoll 4 maxpoll 6
server time3.google.com iburst minpoll 4 maxpoll 6
pool cn.pool.ntp.org iburst maxsources 4 minpoll 4 maxpoll 6

# Local fallback
local stratum 10

# Client access control
allow 10.0.0.0/8
allow 172.16.0.0/12
allow 192.168.0.0/16
deny all
ratelimit interval 3 burst 8

# Hardware clock sync
rtcsync

# Clock step protection
makestep 1.0 3

# Logging & monitoring
driftfile /var/lib/chrony/drift
logdir /var/log/chrony
log measurements statistics tracking

# Command port
bindcmdaddress 0.0.0.0
cmdallow 10.0.0.0/8

Stratum 3 Client – /etc/chrony/chrony.conf:

# Internal time servers
server ntp-01.internal iburst minpoll 4 maxpoll 6 prefer
server ntp-02.internal iburst minpoll 4 maxpoll 6

# Hardware clock sync
rtcsync

# Clock step protection
makestep 1.0 3

# Logging
driftfile /var/lib/chrony/drift
logdir /var/log/chrony
log tracking

Ansible Playbook for Automated Deployment

---
- name: Deploy Chrony Time Synchronization
  hosts: all
  become: yes
  vars:
    chrony_servers:
      - ntp-01.internal
      - ntp-02.internal
  tasks:
    - name: Install Chrony
      package:
        name: chrony
        state: present

    - name: Stop conflicting NTP services
      systemd:
        name: "{{ item }}"
        state: stopped
        enabled: no
      loop:
        - ntpd
        - systemd-timesyncd
      ignore_errors: yes

    - name: Deploy configuration file
      template:
        src: templates/chrony.conf.j2
        dest: /etc/chrony/chrony.conf
        owner: root
        group: root
        mode: '0644'
      notify: restart chronyd

    - name: Ensure Chrony is running
      systemd:
        name: chronyd
        state: started
        enabled: yes

    - name: Verify sync status
      command: chronyc tracking
      register: tracking
      failed_when: "'Stratum' not in tracking.stdout"

    - name: Show sync status
      debug:
        msg: "{{ tracking.stdout_lines }}"

  handlers:
    - name: restart chronyd
      systemd:
        name: chronyd
        state: restarted

Chrony Health‑Check Script (/usr/local/bin/check_chrony.sh)

#!/bin/bash

# Verify Chrony service is active
if ! systemctl is-active --quiet chronyd; then
  echo "CRITICAL: chronyd not running"
  exit 2
fi

# Get tracking data
TRACKING=$(chronyc tracking 2>/dev/null) || { echo "CRITICAL: chronyc tracking failed"; exit 2; }

# Check Stratum (should be <=5 and not 16)
STRATUM=$(echo "$TRACKING" | grep "Stratum" | awk '{print $3}')
if [[ $STRATUM -gt 5 ]] || [[ $STRATUM -eq 16 ]]; then
  echo "CRITICAL: Stratum $STRATUM (unsynced)"
  exit 2
fi

# Check system time offset (absolute value < 10 ms)
OFFSET=$(echo "$TRACKING" | grep "System time" | awk '{print $4}')
OFFSET_ABS=$(echo "$OFFSET" | tr -d '-')
if (( $(echo "$OFFSET_ABS > 0.01" | bc -l) )); then
  echo "WARNING: Offset $OFFSET exceeds 10 ms"
  exit 1
fi

echo "OK: Stratum $STRATUM, Offset $OFFSET"
exit 0

Integration with Nagios/Icinga

# /etc/nagios/nrpe.d/chrony.cfg
command[check_chrony]=/usr/local/bin/check_chrony.sh

Test environment: RHEL 8.8 / Ubuntu 22.04 LTS, Chrony 4.3. Tested on 2025‑10‑31. Maintenance cycle: quarterly configuration review, monthly source reachability checks.