HTTPS Certificate Auto-Renewal: Deployment, Expiry Monitoring & Failure Alerting

Problem Background

Certificate expiration is a predictable yet often overlooked failure that simultaneously disrupts all HTTPS endpoints — browsers block access, API handshakes fail, mobile apps error, and payment callbacks are lost. Unlike typical outages, server metrics (CPU, memory, process, port) appear normal; the only evidence lies in the certificate's notAfter field. Without monitoring this field, teams only discover issues via user complaints.

Automation itself is simple ( certbot one-liner), but three challenges remain: (1) renewal must complete automatically and fail loudly, not silently; (2) renewed certificates must actually be loaded by services (Nginx reload often missed); (3) expiry monitoring must be independent of the renewal pipeline — if the renewal script dies, monitoring must still alert.

Applicable Scenarios

Single-node Nginx/Apache managing multi-domain certificates

Let's Encrypt free certificates for public services

Commercial CA certificates (Aliyun, Tencent Cloud, DigiCert) needing auto-deployment and monitoring

Kubernetes Ingress with cert-manager

Internal services using self-signed or enterprise CA certificates

Not covered: EV/OV certificates requiring manual approval, wildcard domains without DNS API access, and client certificate (mTLS) management.

Core Technical Concepts

ACME Protocol & Validation Methods

Let's Encrypt uses ACME: client generates key pair and CSR, requests challenge, receives token, completes validation (file placement or DNS record), server verifies externally, then issues certificate. Three validation types:

HTTP-01 : Place file at http://domain/.well-known/acme-challenge/<token>. Requires port 80 publicly reachable, correct DNS. Use case: single domain, non-wildcard.

DNS-01 : Add TXT record _acme-challenge.domain. Requires DNS API credentials. Use case: wildcard, no port 80.

TLS-ALPN-01 : Special TLS handshake on port 443. Requires port 443 available. Use case: cannot stop 443.

Critical: HTTP-01 uses port 80, unrelated to HTTPS. If port 80 is firewalled or redirected to HTTPS, validation fails — a common cause of sudden renewal failures.

Let's Encrypt Renewal Window

Certificates valid 90 days. Official recommendation: start renewal when <30 days remain. certbot renew checks each certificate daily; those with >30 days skip (output: Certificate not yet due for renewal), only those ≤30 days attempt renewal, re-running validation. Running daily is safe and avoids rate limits. Rate limits exist (e.g., certificates per registered domain per week) — check current docs; cannot rely on frequent retries.

Certificate File Structure

/etc/letsencrypt/live/example.com/├── cert.pem      -> ../../archive/example.com/cert1.pem├── chain.pem     -> ../../archive/example.com/chain1.pem├── fullchain.pem -> ../../archive/example.com/fullchain1.pem├── privkey.pem   -> ../../archive/example.com/privkey1.pem└── README

cert.pem

— Leaf certificate, rarely used alone chain.pem — Intermediate certificates, rarely used alone fullchain.pem — Leaf + intermediates, Nginx ssl_certificate uses this privkey.pem — Private key, used for

ssl_certificate_key

live/

contains symlinks to versioned files in archive/. Renewal adds new version (e.g., cert2.pem) and updates symlinks. Always point Nginx to live/ paths so reload picks up new certs. /etc/letsencrypt/ is 0700 (root only); Nginx master runs as root (workers as nobody), so reading works.

Post-Renewal Service Reload (Critical)

certbot renew

only updates disk files; Nginx must reload. Three methods:

Deploy hook (recommended): Script in /etc/letsencrypt/renewal-hooks/deploy/ runs only on successful renewal. Example: systemctl reload nginx after nginx -t. --deploy-hook parameter:

certbot renew --deploy-hook "systemctl reload nginx"

--post-hook

(runs every time, even if no renewal) — not recommended.

Verify hook: ls -la /etc/letsencrypt/renewal-hooks/deploy/ — ensure script exists and executable.

Permissions & systemd Timer

certbot

needs root: /etc/letsencrypt/ is 0700, privkey.pem is 0600, webroot write access. Use systemd timer (modern distros):

# /etc/systemd/system/certbot-renew.timer[Unit]Description=Run certbot renew twice dailyDocumentation=https://eff-certbot.readthedocs.io/[Timer]OnCalendar=*-*-* 00,12:17:00RandomizedDelaySec=3600Persistent=true[Install]WantedBy=timers.target# /etc/systemd/system/certbot-renew.service[Unit]Description=Certbot renewalDocumentation=https://eff-certbot.readthedocs.io/[Service]Type=oneshotExecStart=/usr/bin/certbot renew --quiet

Enable:

systemctl daemon-reload && systemctl enable --now certbot-renew.timer

. Design notes: 00,12:17:00 avoids hourly peaks; RandomizedDelaySec=3600 spreads load; Persistent=true catches up after downtime; --quiet reduces noise but errors still logged.

Independent Expiry Monitoring

Must decouple from renewal pipeline. Principles: independent collection (read notAfter directly), independent alerting (works even if renewal script dead), multi-threshold (30/14/7/3 days). Three implementations:

Blackbox probe : Monitor connects to 443, reads cert. Pros: checks actual served cert. Cons: needs external probe points.

File check : Read local cert files. Pros: simple. Cons: misses "file updated but not loaded".

Dual probe : Both. Pros: detects discrepancy. Cons: more complex.

Recommend dual probe: blackbox catches "live cert expiring", file check catches "local updated but service not reloaded".

Key Commands

# Local cert detailsopenssl x509 -in /etc/letsencrypt/live/example.com/fullchain.pem -noout -dates -subject -issuer# Remote cert (SNI required!)echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -dates -subject# Days remaining calculationEXPIRY=$(openssl x509 -in /etc/letsencrypt/live/example.com/fullchain.pem -noout -enddate | cut -d= -f2)EXPIRY_EPOCH=$(date -d "${EXPIRY}" +%s)NOW_EPOCH=$(date +%s)echo $(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))

Nginx Certificate Config Essentials

server {    listen 443 ssl;    http2 on;    server_name example.com;    ssl_certificate      /etc/letsencrypt/live/example.com/fullchain.pem;    ssl_certificate_key  /etc/letsencrypt/live/example.com/privkey.pem;    ssl_protocols TLSv1.2 TLSv1.3;    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:...;    ssl_prefer_server_ciphers off;    ssl_session_cache shared:SSL:10m;    ssl_session_timeout 1d;    ssl_session_tickets off;    ssl_stapling on;    ssl_stapling_verify on;    ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;}

Notes: http2 on; is separate directive in Nginx ≥1.25.1 (old: listen 443 ssl http2;). ssl_prefer_server_ciphers off lets client choose. OCSP stapling ( ssl_stapling on) requires server reach CA OCSP endpoint — verify first or disable (modern browsers handle OCSP leniently).

Certificate Chain Integrity

Incomplete chain causes "some clients work, others error" (old Android, Java, CLI tools). Verify:

echo | openssl s_client -connect example.com:443 -servername example.com -showcerts 2>/dev/null | grep -E "^ *[0-9]+ s:|^ *[0-9]+ i:"

Expect at least two levels (leaf + intermediate). If only level 0, intermediate missing — usually because ssl_certificate uses cert.pem instead of fullchain.pem.

Implementation Roadmap

Phased Rollout

Phase 1: Monitoring only, no auto-renewal. Risk: None.

Phase 2: Manual issuance, validate flow. Risk: Low.

Phase 3: Auto-renewal + Nginx reload. Risk: Medium.

Phase 4: Full automation + complete alerting. Risk: Medium.

Phase 1 is valuable: inventory all certs (expiry, count, location) before automating. Many teams skip this and miss certificates.

Pre-Implementation Information Gathering

# 1. All cert files on systemsudo find /etc /usr/local -name "*.pem" -o -name "*.crt" 2>/dev/null | head -50# 2. Nginx referenced certssudo nginx -T 2>/dev/null | grep -E "ssl_certificate|server_name" | sort -u# 3. Apache certssudo apachectl -S 2>/dev/null; sudo grep -rE "SSLCertificateFile|SSLCertificateKeyFile" /etc/httpd/ /etc/apache2/ 2>/dev/null# 4. Existing certbot certssudo certbot certificates# 5. Other ACME clientswhich acme.sh certbot 2>/dev/null; ls -la /root/.acme.sh/ 2>/dev/null

All read-only; safe to run. nginx -T prints full config — may contain secrets, don't share output.

Step-by-Step Implementation

5.0 Pre-Flight Checks

DNS resolution: Domain must resolve to server public IP for HTTP-01. If CDN/LB, use DNS-01 or allow validation path at CDN.

Port 80 reachability: ss -lntp | grep -E ':80|:443', check firewall/cloud security groups. External test:

curl -s -o /dev/null -w '%{http_code}
' --max-time 5 "http://domain/.well-known/acme-challenge/test"

— 404 means port 80 works. Validation path must be excluded from HTTPS redirect.

Existing cert status: sudo certbot certificates or inspect files with openssl x509 -noout -subject -enddate.

Service supports reload: systemctl show nginx -p CanReload — if no, must restart (causes interruption).

Time sync: timedatectl status — System clock synchronized: yes, NTP service: active. ACME is time-sensitive; clock skew breaks requests and expiry calculations.

5.1 Install certbot

# RHEL/CentOS/openEuler/AlmaLinux/Rockysudo dnf install -y certbot python3-certbot-nginx# Debian/Ubuntusudo apt update && sudo apt install -y certbot python3-certbot-nginx# Snap (official universal)sudo snap install --classic certbot; sudo ln -s /snap/bin/certbot /usr/bin/certbot

Verify: certbot --version. Avoid pip install certbot (PEP 668 externally-managed-environment errors); don't use --break-system-packages.

5.2 Dry-Run First Issuance

DOMAIN=example.com; [email protected]; WEBROOT=/var/www/htmlsudo mkdir -p "${WEBROOT}/.well-known/acme-challenge"sudo chown -R www-data:www-data "${WEBROOT}" 2>/dev/null || sudo chown -R nginx:nginx "${WEBROOT}"sudo certbot certonly --webroot --webroot-path "${WEBROOT}" -d "${DOMAIN}" -d "www.${DOMAIN}" --email "${EMAIL}" --agree-tos --no-eff-email --dry-run

Success output ends with The dry run was successful. Validates DNS, port 80, webroot path, ACME comms. Cannot test production rate limits or final trust chain.

5.3 Production Issuance

sudo certbot certonly --webroot --webroot-path /var/www/html -d example.com -d www.example.com --email [email protected] --agree-tos --no-eff-email --keep-until-expiring

--keep-until-expiring

skips if cert exists and not due. Verify: sudo ls -la /etc/letsencrypt/live/example.com/ and

openssl x509 -in .../fullchain.pem -noout -dates -subject -issuer

. On failure, check /var/log/letsencrypt/letsencrypt.log — don't retry blindly (consumes quota).

5.4 Configure Nginx

Create /etc/nginx/conf.d/example.com.conf with HTTP→HTTPS redirect (preserving /.well-known/acme-challenge/), HTTPS server using fullchain.pem and privkey.pem, modern TLS config, HSTS (start with max-age=300), and proxy settings if needed. Apply:

sudo nginx -t && sudo systemctl reload nginx && systemctl is-active nginx

.

5.5 Deploy Hook for Auto-Reload

sudo mkdir -p /etc/letsencrypt/renewal-hooks/deploycat > /etc/letsencrypt/renewal-hooks/deploy/10-reload-nginx.sh <<'EOF'#!/bin/bashset -euo pipefailLOG_FILE=/var/log/letsencrypt/deploy-hook.logTIMESTAMP="$(date '+%Y-%m-%d %H:%M:%S')"RENEWED_DOMAINS="${RENEWED_DOMAINS:-unknown}"RENEWED_LINEAGE="${RENEWED_LINEAGE:-unknown}"echo "[${TIMESTAMP}] Renewal success domain=${RENEWED_DOMAINS} lineage=${RENEWED_LINEAGE}" >> "${LOG_FILE}"if ! /usr/sbin/nginx -t >> "${LOG_FILE}" 2>&1; then    echo "[${TIMESTAMP}] nginx config test failed, skipping reload" >> "${LOG_FILE}"    exit 1fiif systemctl reload nginx; then    echo "[${TIMESTAMP}] nginx reload successful" >> "${LOG_FILE}"else    echo "[${TIMESTAMP}] nginx reload failed" >> "${LOG_FILE}"    exit 1fiNEW_EXPIRY="$(openssl x509 -in "${RENEWED_LINEAGE}/fullchain.pem" -noout -enddate 2>/dev/null | cut -d= -f2)"echo "[${TIMESTAMP}] New cert expiry: ${NEW_EXPIRY}" >> "${LOG_FILE}"exit 0EOFsudo chmod 0755 /etc/letsencrypt/renewal-hooks/deploy/10-reload-nginx.shsudo chown root:root /etc/letsencrypt/renewal-hooks/deploy/10-reload-nginx.shsudo touch /var/log/letsencrypt/deploy-hook.log; sudo chmod 0600 /var/log/letsencrypt/deploy-hook.log

Key design: set -euo pipefail, quoted variables, nginx -t before reload, reload not restart, dedicated log, non-zero exit on failure so certbot renew reflects hook status.

5.6 Scheduled Renewal (systemd timer preferred)

See timer/service units above. Cron alternative requires explicit PATH, absolute /usr/bin/certbot, and output redirection to log. systemd timer wins on random delay, unified logging ( journalctl), missed-job catch-up ( Persistent=true), and dependency management.

5.7 Expiry Monitoring Setup

Option 1: Script + systemd timer + alert webhook — /usr/local/bin/check-cert-expiry.sh scans /etc/letsencrypt/live, computes days left, tracks cert serial numbers to detect actual rotation, logs structured output ( domain=... days_left=...), exits 0/1/2/3 for OK/WARNING/CRITICAL/ERROR. Integrate alert via env-sourced webhook/token. Timer runs daily at 08:41 with SuccessExitStatus=0 1 2 so only true errors (3) mark service failed.

Option 2: Prometheus + blackbox_exporter — module https_cert with insecure_skip_verify: true (to get probe_ssl_earliest_cert_expiry even if chain broken). Alert rules: warning <21 days (1h for), critical <7 days (5m for), expired <0 (1m for), probe failure up==0 (5m for). Thresholds must match renewal window and team response time.

Option 3: Local vs Remote comparison — compare notAfter from local file and remote openssl s_client. Mismatch reveals "renewed but not reloaded" (local newer), "wrong cert loaded" (remote newer), or "renewal broken" (both old). Add to daily patrol.

5.8 Renewal Drills

certbot renew --dry-run

— validates renewal logic, does not run deploy hooks . certbot renew --force-renewal — forces renewal (consumes quota, risky), only in test env with quota headroom. systemctl start certbot-renew.service — triggers timer service; log should show Certificate not yet due for renewal.

Manual hook test: export RENEWED_DOMAINS / RENEWED_LINEAGE and run hook script; verify deploy-hook.log and Nginx worker restart via ps -eo pid,lstart,cmd | grep 'nginx: worker'.

5.9 Daily Patrol & Reporting

Script generates CSV report: domain, type, subject, issuer, not_before, not_after, days_left. Also lists all server_name from Nginx to find uncertified domains. Sanitize commas in subject ( s/,/;/g).

Troubleshooting Guide

9.1 Cert Expired — Emergency

Confirm:

openssl s_client -connect example.com:443 -servername example.com | openssl x509 -noout -dates

Manual issue:

sudo certbot certonly --webroot --webroot-path /var/www/html -d example.com -d www.example.com --force-renewal

Reload: sudo nginx -t && sudo systemctl reload nginx Verify:

curl -vI https://example.com 2>&1 | grep -E "expire date|SSL certificate"

Post-mortem: why auto-renewal failed.

9.2 Renewal Failure — Log Analysis

Check /var/log/letsencrypt/letsencrypt.log. Common errors: Invalid response ... 404 → Webroot path mismatch → Align --webroot-path with Nginx

root

Invalid response ... 301

→ Port 80 redirects to HTTPS → Exempt validation path from redirect Timeout during connect → Port 80 blocked → Check firewall, security group, cloud SG DNS problem: NXDOMAIN → DNS missing → Fix DNS records Too many certificates already issued → Rate limit hit → Stop retry, use --dry-run, wait for reset

Validate webroot end-to-end: create test file, curl http://domain/.well-known/acme-challenge/testfile from external host — must return 200 with content, not 301.

9.3 Renewed But Service Uses Old Cert

Compare local vs remote notAfter. If local newer: check deploy hook exists, executable, logs, manual run, nginx -t && systemctl reload nginx. If still old: Nginx config points wrong path, multiple server blocks, CDN/LB terminates TLS (check dig and response headers for Server: cloudflare, X-Cache).

9.4 Incomplete Chain

Run openssl s_client -showcerts — if only one cert, fix ssl_certificate to use fullchain.pem, reload, verify Verify return code: 0 (ok).

9.5 Partial Client Failures

Check SAN coverage ( openssl x509 -in cert.pem -noout -ext subjectAltName), chain completeness, cipher suites, SNI support.

Risk Mitigation

Expiry: Monitor all domains, dual alert channels, defined owner/SLA, quarterly drill.

Reload failure: Pre-check disk space, port conflicts, permissions, worker limits; post-reload verify systemctl is-active and listening ports.

Config change: Backup ( cp -a /etc/nginx /etc/nginx.bak-$(date +%Y%m%d%H%M%S)), nginx -t, change window, rollback via rsync -av --delete /backup/ /etc/nginx/.

HSTS misconfig: Ramp max-age (300 → 86400 → 31536000), add includeSubDomains only after all subdomains HTTPS-ready, avoid preload unless certain — removal takes months.

DNS API credentials: File 0600 root:root, use least-privilege sub-account per zone, audit logs, rotate, never commit to Git. Scan repo history:

git log --all --full-history -- '*credentials*' '*dns*.ini' '*vault*'

.

certbot auto-modifies config: --nginx / --apache plugins rewrite config; conflicts with config management (Ansible). Prefer certonly and manage config manually.

Secrets handling: Private keys never leave server ( 0600 root), encrypt backups (

tar -czf - ... | openssl enc -aes-256-cbc -pbkdf2 -salt -out backup.enc

), strong password, store password separately.

Verification Checklist

Cert files exist & readable

Private key permissions 600

Cert not expired (>21 days)

Chain complete (≥2 certs in fullchain.pem)

Private key matches cert (modulus or pubkey diff)

Nginx config references correct

fullchain.pem

nginx -t

passes

Service active

External validation (from outside cluster): curl HTTPS (expect ssl_verify_result=0), openssl s_client for chain/verify, test HTTP→HTTPS redirect, verify challenge path returns 404 (not 301).

Rollback Procedures

Restore old cert: In /etc/letsencrypt/live/example.com/, relink fullchain.pem → ../../archive/example.com/fullchain1.pem (lower number = older), same for privkey.pem, cert.pem, chain.pem. Verify target exists first. Then nginx -t && systemctl reload nginx.

Restore Nginx config: Backup current, rsync -avn --delete /backup/ /etc/nginx/ (dry-run), then execute. Confirm direction: source=backup, target=live.

Emergency issuance: Self-signed 7-day cert (browser/API warnings), borrow from staging, commercial CA quick-issue, or temporary HTTP downgrade (last resort, plaintext exposure).

Production Best Practices

Maintain certificate asset inventory (domain, purpose, issuer, expiry, renewal method, owner, config location, dependencies, alert links). Auto-generate via script.

Alert tiers: 30d info (daily report), 21d warning (chat), 14d critical (chat+call, 2h response), 7d emergency (call+SMS, immediate), expired (all channels). Thresholds based on renewal window + response time ×2.

Alert payload must include: domain, expiry timestamp, days left, issuer, server/config location, last renewal attempt + error, runbook link.

Multi-node: per-node independent renewal (no single point, but certs differ) OR central issuance + Ansible distribution (consistent, but distribution is single point) OR shared storage (single cert, but storage SPOF and key transit risk). Ansible example included with cert/key sync, match verification, expiry check, handler reload.

Wildcard vs multi-SAN: few fixed domains → single cert with SANs; many subdomains (>20) → wildcard (requires DNS-01, credential risk); cross-level subdomains ( a.b.example.com) need extra cert (wildcard is single-level). SAN limit ~100, practical limit ~10.

Time sync mandatory ( chronyd / ntpd), timezone irrelevant for certs (UTC) but matters for logs. Keep ca-certificates updated (requires service reload after update).

Anti-Patterns to Avoid

Monitoring only the issuance server, not all serving nodes.

Discarding certbot renew output ( >/dev/null 2>&1) — keep logs, check exit codes.

Monitoring only local files, not live served certs.

Using staging CA for production (check server in renewal/*.conf).

Assuming renewal success = service updated; always verify local vs remote notAfter.

All domains on one cert with single validation method — one domain's validation failure blocks all.

Manual changes undocumented — record time, reason, result.

Summary

Three pillars: (1) Renewal truly automated — timer enabled, webroot path correct, validation path not redirected, rate limits respected, not yet due for renewal log count proves pipeline alive. (2) Cert actually loaded — deploy hook runs nginx -t && systemctl reload nginx, verify by comparing local and remote notAfter. (3) Expiry monitoring independent — separate collection path, multi-threshold alerts, dual probe (blackbox + file).

Key lessons: fullchain.pem not cert.pem; HSTS ramp slowly; protect validation path from redirects; --dry-run skips hooks — test separately; private key 0600; cover all nodes; wildcard = DNS credentials risk; "nothing happening" daily is the success signal. Quarterly end-to-end drill catches config drift. Master these, and certificate expiry drops off your incident list.