Skip to content
SaltStack Automation from Zero to One: Practical Batch Management Implementation
Operations 21 min read

SaltStack Automation from Zero to One: Practical Batch Management Implementation

This article provides a step-by-step guide to implementing SaltStack for operations automation, covering master/minion deployment, key-based authentication, state tree and pillar configuration, grain-based targeting, canary deployments with dry-run verification, batched rollouts, health checks, troubleshooting techniques, and Git-based rollback procedures on Ubuntu 22.04.

Raymond Ops
Raymond Ops
Raymond Ops

When the number of hosts is small, manual SSH changes appear fast enough; as the fleet grows, configuration drift, missed nodes, and untracked ad-hoc fixes quickly amplify. SaltStack’s value is not merely parallel command execution but codifying target hosts, desired state, authorization boundaries, change batches, and execution results into repeatable processes.

Define the Control Boundary

The Master stores state files (SLS), Pillar data, and trusted Minion keys; Minions initiate connections to the Master, and after authentication they receive commands or state applications. State enforcement is declarative (e.g., nginx installed, file correct, service running); remote execution is imperative (e.g., read logs, check ports). The former serves long-term configuration, the latter forensics and controlled ad-hoc operations.

Do not start by targeting all hosts with a wildcard, and never enable auto_accept. Accepting a key is an authorization act in the management plane; mistakenly accepting an unknown host brings it into the command distribution scope. Minion IDs must be stable and readable (e.g., prod-web-01), not dependent on DHCP addresses.

# Verify versions and services on Master and Minion
salt --version
salt-master --version
salt-minion --version
systemctl is-enabled salt-master 2>/dev/null || true
systemctl is-enabled salt-minion 2>/dev/null || true

Example output (format only, not a substitute for on-site evidence): salt 3007.1 (Chlorine) If the organization maintains an internal software repository, prefer the approved source and pin validated versions. The following Ubuntu package paths assume the mirror administrator has confirmed package availability.

# Install Master on dedicated management node
sudo apt-get update
sudo apt-get install -y salt-master
sudo systemctl enable --now salt-master
sudo systemctl status salt-master --no-pager
# Install Minion on each managed host
sudo apt-get update
sudo apt-get install -y salt-minion
sudo systemctl enable salt-minion

Minions must reach the Master on TCP/4505 and TCP/4506. Firewalls should allow only the managed CIDR blocks; do not open ports to the internet for troubleshooting. A failed connection test only proves TCP unreachability; routing, DNS, addressing, and listener status must also be checked.

# On Minion, test connectivity to Master ports
MASTER_HOST="<SALT_MASTER_IP>"
for port in 4505 4506; do
  timeout 3 bash -c "</dev/tcp/${MASTER_HOST}/${port}" \
    && echo "${MASTER_HOST}:${port} reachable" \
    || echo "${MASTER_HOST}:${port} unreachable"
done
# On Master, verify listeners and logs
sudo ss -lntp '( sport = :4505 or sport = :4506 )'
sudo systemctl status salt-master --no-pager
sudo journalctl -u salt-master -n 100 --no-pager

Complete Initial Onboarding with Key Audit

First onboarding requires configuring the Master address and a stable ID on the Minion. Modifying service configuration has impact: back up first, write minimal configuration, restart the Minion, and immediately inspect logs. Restarting the Minion briefly loses remote management capability but does not directly restart business services; still schedule within a controlled window.

#!/usr/bin/env bash
set -euo pipefail

MASTER_IP="<SALT_MASTER_IP>"
MINION_ID="<MINION_ID>"
CONF_DIR="/etc/salt/minion.d"
CONF_FILE="${CONF_DIR}/10-master.conf"

sudo install -d -m 0755 "${CONF_DIR}"
sudo cp -a "${CONF_FILE}" "${CONF_FILE}.bak.$(date +%F-%H%M%S)" 2>/dev/null || true
sudo tee "${CONF_FILE}" >/dev/null <<EOF
master: ${MASTER_IP}
id: ${MINION_ID}
EOF
sudo systemctl restart salt-minion
sudo systemctl status salt-minion --no-pager

First onboarding cannot rely on Salt’s own distribution because the Minion is not yet managed. Before accepting, verify that the ID matches the asset inventory, instance records, and source IP; if inconsistent, delete the pending key and investigate — do not accept first and handle later.

# On Master, review pending keys
sudo salt-key -L
sudo salt-key -F
sudo salt-key -f '<MINION_ID>'
# Accept only a single verified Minion
sudo salt-key -a '<MINION_ID>' -y
salt '<MINION_ID>' test.version
salt '<MINION_ID>' test.ping
test.ping

only proves the Salt channel and Minion process work; it does not prove disk, business ports, or state tree correctness. When Minion did not return appears, sequentially check Minion logs, Master key, port connectivity, and ID conflicts; do not repeatedly delete and accept keys to mask the problem.

# Collect authentication and service evidence from both ends
# Master side
sudo salt-key -F
sudo journalctl -u salt-master --since '30 minutes ago' --no-pager | tail -n 200

# Minion side
sudo salt-call --local test.version
sudo journalctl -u salt-minion --since '30 minutes ago' --no-pager | tail -n 200

State Tree, Pillar, and Target Selection

/srv/salt

as the SLS root and /srv/pillar as the environment parameter root are a maintainable starting point. Pillar is not a password vault: it is rendered and sent to matching Minions. Sensitive data must be scoped to the minimum match set, Master file permissions tightened, and high-sensitivity credentials stored in an approved secrets system.

# /etc/salt/master.d/10-roots.conf
file_roots:
  base:
    - /srv/salt/base
  prod:
    - /srv/salt/prod

pillar_roots:
  base:
    - /srv/pillar/base
  prod:
    - /srv/pillar/prod

Snapshot the Master configuration before changes; after loading, verify environment visibility with the fileserver runner. This is not a full YAML lint, so lint before commit and rehearse on a test Master.

#!/usr/bin/env bash
set -euo pipefail

SOURCE_FILE="/etc/salt/master.d/10-roots.conf"
BACKUP_DIR="/var/backups/salt-master"

sudo install -d -m 0700 "${BACKUP_DIR}"
sudo cp -a "${SOURCE_FILE}" "${BACKUP_DIR}/10-roots.conf.$(date +%F-%H%M%S)" 2>/dev/null || true
sudo systemctl restart salt-master
sudo systemctl is-active --quiet salt-master
salt-run fileserver.envs

The Top file determines entry points and matching. This article uses grain roles to select web nodes; grains suit scheduling classification but are not a security boundary because Minions can self-report grains.

# /srv/salt/prod/top.sls
prod:
  'roles:web':
    match: grain
    - common
    - nginx
# Set role on canary node and verify targeting
salt '<MINION_ID>' grains.get roles
salt '<MINION_ID>' grains.setval roles '["web"]'
salt -G 'roles:web' test.ping
salt -G 'roles:web' grains.item id osrelease roles

Common states hold only cross-business consistency requirements. Package names below assume Ubuntu 22.04; RHEL-family distributions must use corresponding package names and be tested first.

# /srv/salt/prod/common/init.sls
common-packages:
  pkg.installed:
    - pkgs:
      - curl
      - ca-certificates
      - chrony

common-timezone:
  timezone.system:
    - name: Asia/Shanghai

ops-audit-directory:
  file.directory:
    - name: /var/log/ops-audit
    - user: root
    - group: adm
    - mode: '0750'
    - makedirs: True

Business states use require and watch to explicitly constrain package, file, and service ordering. watch triggers a reload only when configuration changes, reducing disturbance; this assumes the service unit supports reload.

# /srv/salt/prod/nginx/init.sls
nginx-package:
  pkg.installed:
    - name: nginx

/etc/nginx/conf.d/app.conf:
  file.managed:
    - source: salt://nginx/files/app.conf.jinja
    - template: jinja
    - user: root
    - group: root
    - mode: '0644'
    - require:
      - pkg: nginx-package

nginx-service:
  service.running:
    - name: nginx
    - enable: True
    - reload: True
    - watch:
      - file: /etc/nginx/conf.d/app.conf
# /srv/salt/prod/nginx/files/app.conf.jinja
upstream app_backend {
  server {{ pillar['app']['upstream_host'] }}:{{ pillar['app']['upstream_port'] }};
}
server {
  listen 80;
  server_name _;
  location / {
    proxy_pass http://app_backend;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

The template reads only the Pillar keys required by this state. Missing values should error out, not generate an empty upstream.

# /srv/pillar/prod/top.sls and web.sls
prod:
  'roles:web':
    match: grain
    - web

# /srv/pillar/prod/web.sls
app:
  upstream_host: <upstream address>
  upstream_port: <upstream port>

Rehearsal, Canary, Batching, and Verification

Production state changes never start with a full fleet. First confirm targets, then run a dry-run ( test=True) on canary nodes. A rehearsal shows planned changes but cannot guarantee that package repositories will be reachable in the real window or that a reload won’t be blocked by other configuration errors.

# Confirm targets, refresh Pillar, canary dry-run
salt -G 'roles:web' test.ping
salt '<MINION_ID>' saltutil.refresh_pillar
salt '<MINION_ID>' state.apply nginx saltenv=prod test=True

Example dry-run output:

----------
    ID: nginx-service
  Function: service.running
    Result: None
   Comment: Service nginx is set to be started
   Changes: {}

If the rehearsal shows unexpected file modifications, stop immediately and cross-check the Top file, Pillar, template rendering, and existing node configuration. state.show_sls displays the compiled low-level state, useful for verifying includes and requisites.

# Diagnose rendering, compilation, and data differences
salt '<MINION_ID>' pillar.items
salt '<MINION_ID>' state.show_sls nginx saltenv=prod
salt '<MINION_ID>' cp.get_file_str salt://nginx/files/app.conf.jinja saltenv=prod

Only after confirming the canary’s real execution should batch be used. batch limits concurrent executions but does not equal business protection; batch size must factor in replica count, drain capability, reload duration, and failure domains.

# Execute in 10% batches and verify Nginx configuration
salt -G 'roles:web' --batch='10%' state.apply nginx saltenv=prod
salt -G 'roles:web' --batch='10%' cmd.run 'nginx -t' --timeout=60

A state Result: True does not guarantee user-facing traffic, upstream dependencies, or monitoring probes are healthy. Validation must include Salt return data, local checks, and business monitoring. Long-running tasks should record the JID; do not rely solely on terminal output.

# Submit job asynchronously and query job status
salt -G 'roles:web' --async state.apply nginx saltenv=prod
salt-run jobs.active
salt-run jobs.list_jobs
salt-run jobs.lookup_jid '<JID>'
# On canary host, verify syntax, service, port, and local health check
sudo nginx -t
systemctl is-active --quiet nginx
ss -lntp '( sport = :80 )'
curl --fail --silent --show-error \
  --max-time 5 \
  "http://127.0.0.1/<health-check-path>" >/dev/null

The <health-check-path> must be confirmed by the application owner and have no side effects. Inspections should control timeouts and output volume to avoid flooding the Master with logs.

#!/usr/bin/env bash
set -euo pipefail

TARGET="G@roles:web"
OUTPUT_FILE="/var/log/ops-audit/salt-web-readiness-$(date +%F-%H%M%S).log"

salt -C "${TARGET}" cmd.run \
  'systemctl is-active nginx && nginx -t' \
  --timeout=30 | tee "${OUTPUT_FILE}"
echo "Please cross-reference monitoring with failures in ${OUTPUT_FILE}."

Failure, Risk, and Rollback

When no nodes match, check the targeting expression and grains; when a Minion does not return, check the service, key, network, and clock; when a state fails, read the failed item’s Comment, journald, package manager logs, and target file diffs. Root-cause conclusions must be evidence-based, not guesswork.

# Collect minimal evidence set for a state failure
salt '<MINION_ID>' state.apply nginx saltenv=prod -l info
salt '<MINION_ID>' cmd.run 'journalctl -u nginx -n 80 --no-pager'
salt '<MINION_ID>' cmd.run 'journalctl -u salt-minion -n 120 --no-pager'
salt '<MINION_ID>' cmd.run 'sudo nginx -t'

When refreshing cached data, target only affected nodes; a fleet-wide cache clear expands the blast radius and may not resolve file_roots, permission, or Pillar matching issues.

# Targeted refresh of affected node data
salt '<MINION_ID>' saltutil.sync_all
salt '<MINION_ID>' saltutil.refresh_pillar
salt '<MINION_ID>' saltutil.refresh_modules
salt '<MINION_ID>' state.show_top saltenv=prod

A reliable rollback path: version the state tree with Git tags; record version and target before execution; on canary failure, halt subsequent batches; revert to the last approved version; apply the old state on already-executed nodes; repeat the same health checks. rsync --delete removes files in the destination that are absent from the source — a high-risk operation. Confirm the directory first and validate on a test Master; production should prefer versioned directories with atomic symlinks.

#!/usr/bin/env bash
set -euo pipefail

STATE_REPO="/srv/salt-repo"
ROLLBACK_REF="<previous approved Git tag or commit>"
CANARY_ID="<MINION_ID>"

cd "${STATE_REPO}"
git status --short
git fetch --tags
git checkout --detach "${ROLLBACK_REF}"
sudo rsync -a --delete "${STATE_REPO}/prod/" /srv/salt/prod/
salt "${CANARY_ID}" state.apply nginx saltenv=prod

SaltStack adoption does not end with software installation. Asset naming solves target identification, key audit solves onboarding boundaries, SLS and Pillar solve configuration sources, canary and batching control change radius, job records and business verification close the evidence loop. Start by running through a few non-critical nodes, then gradually expand to roles, building a maintainable automation capability.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

batch processingConfiguration ManagementInfrastructure as CodeOperations Automationcanary deploymentSaltStackRollback StrategiesUbuntu 22.04
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.