Ansible for Batch Server Inspection, Config Management & Zero-Downtime Deployments

Introduction: Why Batch Operations Need Engineering Constraints

The author describes taking over a 60-node cluster spanning three environments (dev/staging/prod) with mixed OSes (CentOS 7.9, Rocky Linux 8/9, Ubuntu 22.04). Daily operations included weekly inspections via a serial shell script, configuration management scattered across documents, and manual application deployments taking 25 minutes for 40 nodes with high risk of human error.

The initial shell script exposed classic problems: serial execution (time grows linearly), no idempotency (re-runs cannot verify convergence), silent failures (errors swallowed by || echo), no change records, no dry-run capability, quoting/escaping bugs, and uncontrolled concurrency overwhelming the bastion host's MaxStartups.

Ansible's Fit and Limits

Ansible addresses these by declaring target state, using modules for idempotency, providing --check --diff for dry-runs, serial for batch control, and callbacks/logs for audit. However, it is not suited for high-frequency second-level operations, complex long-running orchestrations (use Argo/Tekton), thousands of nodes needing sub-second sync (consider SaltStack/Puppet agent or immutable images), or deep Windows management.

Environment and Version Constraints

Control node: Rocky Linux 9.4, Python 3.11.7, ansible-core 2.20.x (community package Ansible 13). Managed nodes: CentOS 7.9 (Python 3.6.8), Rocky 8.10/9.4, Ubuntu 22.04. Connection via deploy user with SSH keys and passwordless sudo limited to required commands.

Key version pitfalls: ansible-core vs ansible (community bundle); install only ansible-core and lock collections via requirements.yml. Python version requirements rise with ansible-core (2.17+ needs Python 3.10+). CentOS 7's Python 3.6 may require pinning ansible-core to 2.16/2.17 or installing newer Python and setting ansible_python_interpreter. Since Ansible 2.19, prefer ansible_facts['key'] over ansible_key for fact access.

Core Concepts

Inventory: Grouping and Variable Inheritance

Use a two-dimensional grouping by environment and role. Example INI inventory shows [web], [api], [db], [centos7] for legacy Python interpreter override, and [prod:children] combining them. Parent group variables apply to children via "child → parent" lookup (nearest wins). Range syntax [01:08] requires zero-padded hostnames. Use ansible_host when inventory name differs from connection address.

Variable Precedence (Eight Practical Layers)

From lowest to highest: role defaults/main.yml → group_vars/all → group_vars/<group> → group_vars/<parent> → host_vars/<host> → play vars → set_fact / register → -e/--extra-vars. Rules: keep role defaults in defaults/main.yml (not vars/main.yml which is too high), and reserve -e for one-off overrides only.

Facts: Turning Machine State into Data

gather_facts

runs setup module by default. Use gather_subset to limit collection (e.g., hardware,network). Enable fact caching ( fact_caching=jsonfile, gathering=smart) to speed repeated plays. Refresh facts explicitly after changes (hostname, disks) with a targeted setup task.

Idempotency: The Core Promise

Idempotent means re-running yields changed=0. Modules like copy, template, sysctl are naturally idempotent; command / shell always return changed=true. To make them idempotent: use creates (skip if file exists), stat + when, changed_when:false for read-only commands, or changed_when based on output (e.g., 'pending migrations' in stdout). Avoid shell redirections/pipes in shell tasks; use lineinfile or sysctl module instead.

Handlers: When to Restart vs Reload

Handlers run once at play end per unique name. Order not guaranteed; use listen with multiple handlers or meta: flush_handlers for strict sequencing. The validate parameter (e.g., validate: '/usr/sbin/sshd -t -f %s') checks syntax before overwriting. Prefer reload over restart for SSH (keeps existing sessions); test new sessions in a separate window.

Execution Model: forks, serial, throttle

forks

(default 5) controls global concurrent connections. serial splits hosts into batches, each completing the entire play before next batch (core of rolling updates). throttle limits concurrency for a single task (e.g., API calls). Example: serial: "25%" with max_fail_percentage: 0 stops the rollout if any host in a batch fails.

Check Mode and Diff

--check --diff

shows planned changes without executing. Limitations: command / shell skipped by default; tasks depending on prior results see predicted values; no rollback capability. Use for preview, then small-batch real runs.

Three-Layer Architecture and Three Pipelines

Structure: inventory/ (who), roles/ (what), playbooks/ (when). Roles: base (baseline), harden (security), inspect (read-only checks), app_deploy (release), app_rollback (rollback). Playbooks: site.yml (full convergence), inspect.yml, deploy.yml, rollback.yml.

Three independent pipelines: Inspection (read-only, high concurrency), Config Convergence (idempotent, --check then serial, drift rate metric), Deployment (artifact verify → version dir → symlink switch → restart → health check → auto-rollback; metrics: batch failure rate, rollback time). Failures isolate the host ( block/rescue or any_errors_fatal) so rest continue.

Directory Layout and ansible.cfg

Standard layout under /opt/ansible with inventory/prod, roles, playbooks, collections. Key ansible.cfg settings: host_key_checking=True (pre-populate known_hosts via ssh-keyscan), pipelining=True (requires requiretty off), ControlPersist=300s for SSH multiplexing, forks=20 (tune via time ansible all -m ping), interpreter_python=auto_silent, JSON fact caching, stdout_callback=yaml, callbacks_enabled=profile_tasks,timer, logging to /var/log/ansible/ansible.log.

Connectivity and Pre-flight Checks

Create deploy user with restricted sudo via Cmnd_Alias (read vs write commands). Validate with visudo -cf. Distribute keys via ssh-copy-id, authorized_key module (with --ask-pass), or pre-seed in images. Three-layer check: 1) ansible all -m ping (SSH+Python), 2) ansible all -m command -a 'id -u' --become (sudo), 3) ansible all -m setup -a 'filter=ansible_distribution*' (facts). Wrap in a preflight.yml playbook using failed_when:false + assert to collect all issues before halting. Time skew >5s vs control node is critical for certs, logs, JWT.

Inspection Pipeline: Structured Data Output

Inspection role collects: OS/kernel, disk usage (per mount, excluding virtual FS), memory, load (1/5/15 min normalized per vCPU), zombie processes, time offset (using lookup('pipe','date +%s') on controller, not ansible_date_time), certificate expiry (remote openssl + date math), key ports ( ss -tuln), kernel params ( sysctl -n), reboot required ( needs-restarting -r on RHEL, /var/run/reboot-required on Debian). Thresholds defined in defaults/main.yml.

Evaluation tasks produce inspect_failed list with level (ok/warn/crit). Report generation uses run_once: true + delegate_to: localhost with a Jinja2 CSV template iterating ansible_play_hosts_all and hostvars. Also emit Prometheus textfile metrics for node_exporter ( --collector.textfile.directory). Insight: inspection's end state is metrics, not just reports.

Configuration Management: Base and Harden Roles

Base Role: One-Time Convergence, Long-Term Compliance

Manages timezone, chrony (internal NTP), kernel params via single template /etc/sysctl.d/99-tuning.conf + sysctl --system (avoids conflict with sysctl module), PAM limits ( /etc/security/limits.d/90-ansible.conf) AND systemd global limits ( /etc/systemd/system.conf.d/90-limits.conf with DefaultLimitNOFILE + daemon-reexec), journald size/retention. Handlers: reload sysctl, restart chronyd, daemon-reload systemd, restart journald.

Critical nuance: /etc/security/limits.conf only affects PAM login sessions, not systemd services. Verify with systemctl show -p LimitNOFILE app-api.service or /proc/<pid>/limits.

Harden Role: Safety-First SSH Changes

Backup original sshd_config with timestamp, write new config to temp file, validate with sshd -t -f, atomically replace, meta: flush_handlers to reload immediately, then verify connectivity from controller with a fresh SSH connection. On failure, rollback to latest backup and reload. Apply in three batches (low/medium/high risk) with 24h observation; before batch 2/3, verify all hosts accept key-only auth via

ansible all -m command -a 'true' --ssh-extra-args '-o PreferredAuthentications=publickey -o PasswordAuthentication=no'

.

Drift Detection

Run ansible-playbook site.yml --check --diff periodically (cron weekly). Count changed tasks as drift. Log output for audit.

Application Deployment: Versioned Directories + Symlink + Health Checks + Auto-Rollback

Model: /opt/app/releases/<timestamp>-<version>/, current -> releases/..., shared/{logs,data}. Benefits: instant rollback via ln -sfn, failed deploy doesn't corrupt running version, version traceability via VERSION file (git commit, build time, user, artifact URL, SHA256). app_deploy role: validate required app_version / app_artifact_url, create app user, directories, download artifact ( get_url with checksum), verify gzip -t, unarchive to new release dir ( creates for idempotency), write VERSION, symlink shared dirs, deploy systemd unit (template with LimitNOFILE, hardening flags), record previous current target, atomically switch symlink ( force:true), flush_handlers to restart, wait for health endpoint ( uri with retries/delay, ignore_errors:true), on failure rollback symlink + restart + re-verify, then fail to stop batch ( max_fail_percentage:0). First deploy handled separately (no previous version).

Systemd unit includes TimeoutStopSec=30, KillSignal=SIGTERM, Restart=on-failure, StartLimitBurst=3 to prevent restart storms.

Rolling Deployment: Serial + Drain + Observe

deploy.yml

uses serial: "25%", max_fail_percentage:0, any_errors_fatal:true. pre_tasks: disable in LB ( throttle:1), wait for established connections to drain ( ss -Htn state established '( sport = :8080 )' <5). post_tasks: re-enable in LB, pause: 30s for error-rate observation. Verify via service status, version file, health endpoint, and Prometheus error-rate query.

Cleanup Old Releases (High Risk)

List candidates (keep app_release_keep=5, skip current), display, then delete only if app_cleanup_confirm=true with path sanitization ( rm -rf --).

Common Commands Cheatsheet

Connectivity: ansible all -m ping, ansible-inventory --graph, ansible-doc. Ad-hoc:

ansible web -m command -a 'systemctl status nginx' --limit web-prod-01

, async -B 60 -P 5. Playbook: --syntax-check, --list-tasks, --check --diff, --start-at-task, --step, -e 'app_version=1.8.3' --limit 'web:&prod' (pattern: : union, & intersection, ! exclude). Debug: -vvv, ANSIBLE_DEBUG=1. Vault: ansible-vault create/edit/encrypt_string, --ask-vault-pass.

Logging and Observability

Built-in callbacks: profile_tasks (per-task timing), timer (total). JSON callback ( ANSIBLE_STDOUT_CALLBACK=ansible.builtin.json) for Loki/ELK ingestion. ARA (Ansible Run Analysis) stores full run data in DB with UI; must use no_log:true on sensitive tasks. Emit Prometheus metrics ( ansible_last_run_timestamp, ansible_hosts_total) with alert on staleness (>25h).

Troubleshooting: Six High-Frequency Issues

UNREACHABLE : check network ( nc -zv host 22), key auth ( ssh -v), firewall, fail2ban. Timeout during banner exchange indicates high load or blocking /etc/motd.

Missing sudo password : become_ask_pass=True without -K or sudoers requires password. Fix sudoers or use --ask-become-pass (not for cron).

Module not found : collection missing or wrong collections_path (relative to CWD). Use absolute path or cd /opt/ansible.

Python interpreter : CentOS 7 lacks python3 symlink; set ansible_python_interpreter=/usr/bin/python3.6 in inventory or rely on auto_silent.

Slow execution : enable pipelining / ControlPersist, fact caching, increase forks (20-50), raise timeout / retries for flaky hosts.

Broken idempotency : command / shell without changed_when:false / creates; templates with volatile vars (e.g., ansible_date_time); lineinfile regex mismatch; permission drift.

Risk Management and Production Gates

Six high-risk operations with pre-checks and rollback: sysctl (gray-scale 1 node), sshd (new window test, backup, out-of-band), app restart (LB drain), version cleanup (disabled by default, whitelist), firewalld (allow before deny), user/perm changes (check service user, backup /etc/passwd).

Four gates: 1) Permission control (playbook dirs 750/700, vault key 600 separate), 2) Execution window (block deploys 09:00-22:00 unless FORCE_DEPLOY=true), 3) Canary (1 node → 10% → full), 4) Verification (service, version, health, business error rate from Prometheus).

Secrets Management

Never plaintext in playbooks/inventory/group_vars; never in -e (shell history, ps); never vault password file beside encrypted files. Use ansible-vault on group_vars/prod/vault.yml, reference in all.yml, add no_log:true on tasks using secrets. For higher security, fetch from external Vault/KMS via lookup('community.hashi_vault.vault_kv2_get', ...).

Encoding and CI/CD Boundaries

Set stdout_encoding=utf-8 in ansible.cfg. Ensure templates are UTF-8 no-BOM, LF only (check with grep -r $'\xEF\xBB\xBF' and grep -r $'\r'). Ansible handles machine-level convergence; CI/CD (Jenkins/GitLab) builds artifacts and triggers deploy.yml with version/URL. Baseline convergence runs on independent cron, not tied to deploy. Emergency rollback via manual rollback.yml.

Conclusion

Inspection = machine state → comparable data (CSV/metrics). Config management = continuous proof of desired state ( --check drift rate). Deployment = failure containment (version dirs + symlink, serial + drain, health check + auto-rollback). The real challenge is discipline: resist shell, -e, full-batch runs, ad-hoc tweaks. Ansible is permissive; production is not.