Enterprise Server Hardening: Accounts, Ports & Access Control Deep Dive

Why Hardening Fails as a One-Time Effort

The author observes that hardening checklists are often executed once then drift: SSH ports reverted to 22 for automation convenience, ops accounts retain initial passwords, sudo NOPASSWD:ALL left after troubleshooting, firewalls opened with --add-port=0-65535/tcp. Hardening is a continuous capability, not a one-off config. This article covers three pillars with executable commands, verifiable checks, risk notes, and rollback paths.

Absolute prerequisite: an out-of-band channel (cloud VNC, IPMI, serial) independent of the current SSH session. Do not start SSH changes without it.

Scope & Environment Assumptions

Applicable Scenarios

On-prem/private cloud Linux : Fixed IP planning, bastion host or planned

Cloud IaaS VMs : Security groups + host firewall dual layer

Hybrid with containers : Host hardening + container runtime hardening

Compliance (MLPS, ISO 27001) : Need baseline config + check evidence

Team 3-50, no dedicated security : Lightweight maintainable solution

Not Covered / Need Extra

Mature zero-trust + SSO/PAM teams (use existing identity, take only port/permission parts)

Windows servers (different commands)

Kubernetes nodes (larger attack surface in kubelet, runtime, API server)

MLPS Level 3 centralized control (no bastion/audit deployment)

Version Baselines

OS baselines (two main lines, command differences noted per section):
  RHEL 9.4/9.6, Rocky Linux 9.4/9.6   (dnf, firewalld, SELinux enforcing)
  Ubuntu 22.04 LTS, 24.04 LTS           (apt, ufw optional, AppArmor)
Reference RHEL 10 / Rocky 10 (2025 GA):
  - Default sudo-rs replaces traditional sudo (config still /etc/sudoers, most syntax compatible, some advanced features missing, complex sudoers need regression test)
  - Default dnf5
  - sshd defaults to prohibit root password login
  - Default fapolicyd (application allowlist) affects package install/binary execution
Key component versions:
  OpenSSH 8.7p1 (RHEL 9, updated to 9.x), OpenSSH 9.6p1 (Ubuntu 24.04)
  OpenSSH 10.0 (2025-04) defaults to reject ssh-rsa (SHA-1) signatures
  firewalld 1.2/1.3 (RHEL 9)
  sudo 1.9.15 (RHEL 9), 1.9.15p5 (Ubuntu 24.04)
  SELinux policy 39.x (RHEL 9.4+)
  audit 3.1.x

Critical: RHEL 10 vs RHEL 9 differences are not minor; they break scripts (sudo-rs, dnf5, fapolicyd). This article uses RHEL 9 and Ubuntu 24.04 as execution baselines, with RHEL 10 notes where relevant.

Mandatory Pre-Hardening Checks

These must be completed before any changes; skipping raises incident probability.

# 1. Confirm out-of-band access (manual, no command verification)
#    - Cloud: console VNC / serial login works
#    - Physical: IPMI/iDRAC/iLO works
#    - Or: datacenter contact available

# 2. Record pre-hardening state for diff & rollback
BASE="/var/log/hardening/$(date +%F)"
sudo mkdir -p "${BASE}"
sudo sshd -T > "${BASE}/sshd_effective.txt" 2>&1
sudo cp -a /etc/ssh/sshd_config "${BASE}/sshd_config.orig"
sudo cp -a /etc/ssh/sshd_config.d "${BASE}/sshd_config.d.orig" 2>/dev/null || true
sudo cp -a /etc/sudoers "${BASE}/sudoers.orig"
sudo cp -a /etc/pam.d "${BASE}/pam.d.orig"
sudo cp -a /etc/login.defs "${BASE}/login.defs.orig"
sudo firewall-cmd --list-all > "${BASE}/firewalld_before.txt" 2>/dev/null || \
sudo ufw status verbose > "${BASE}/ufw_before.txt" 2>/dev/null || true
getent passwd | sort > "${BASE}/passwd.before"
sudo getent shadow | awk -F: '{print $1":"$2}' > "${BASE}/shadow.before"
sudo sestatus > "${BASE}/selinux_before.txt" 2>/dev/null || true
ls -la "${BASE}"

# 3. Record current logged-in users
who
ss -tnp state established '( sport = :22 )' | head -20

# 4. Confirm current user has sudo and is not sole channel
sudo -l | head -10
id
sudo getent group wheel     # RHEL
sudo getent group sudo      # Debian

sshd -T

outputs the merged effective config — the most valuable evidence for before/after comparison because sshd_config may have Include /etc/ssh/sshd_config.d/*.conf causing actual effective values to differ from file content.

Keep a root session open and untouched:

sudo -i
# Keep this window open until all verification passes. It is the last lifeline if SSH/PAM/firewall changes break access.

Account Management: From Shared Root to Per-Person Accounts

Core Principles (Priority Order)

Disable root remote login — all ops via personal account + sudo so sudo logs answer "who did what when".

One person, one account — prohibit shared accounts ( ops, admin, deploy); shared accounts only for automation with source IP and command restrictions.

Account lifecycle — provision on join, adjust on role change, revoke same day on leave, with auditable revocation actions.

Inventory Before Deletion

Do not delete accounts first. Many accounts have cron jobs, service dependencies, app connections; deletion causes midnight explosions. Inventory first:

# Loggable regular users (uid >= 1000 and not nologin/false)
awk -F: '$3>=1000 && $3<65534 && $7 !~ /(nologin|false)$/ {print $1"\t"$3"\t"$6"\t"$7}' /etc/passwd

# All uid == 0 accounts (must be only root)
awk -F: '$3==0 {print $1}' /etc/passwd

# Empty password accounts
sudo awk -F: '($2==""){print $1}' /etc/shadow

# Recent login times & sources
sudo lastlog | head -30
sudo last -n 30

# Failed login records
sudo lastb | head -20

# Specific account group membership
id alice

Judgment criteria: uid == 0 count > 1 → critical (possible backdoor or legacy root alias)

Empty password accounts → immediate remediation lastlog shows **Never logged in** and not a service account → cleanup candidate

Home directory exists but no login for 3 years → cleanup candidate, but first check for cron/service usage

Check service dependencies (cannot skip):

USERNAME=devold
sudo grep -rn "${USERNAME}" /etc/cron* /var/spool/cron/ 2>/dev/null
sudo crontab -u "${USERNAME}" -l 2>/dev/null
sudo grep -rn "User=${USERNAME}" /etc/systemd/system/ /usr/lib/systemd/system/ 2>/dev/null
sudo ps -u "${USERNAME}" -o pid,cmd --no-headers
sudo find /opt /srv /data /var/www -xdev -user "${USERNAME}" -maxdepth 4 2>/dev/null | head -20

All five checks empty → safe to proceed with cleanup.

Establish Account Baseline

Password Policy

RHEL 9 and Ubuntu 24.04 both use PAM + pwquality with slightly different config entry points.

RHEL 9 / Rocky 9:

# View existing policy
grep -vE '^\s*#|^\s*$' /etc/security/pwquality.conf

# Use drop-in directory to avoid upgrade/package overwrite
sudo mkdir -p /etc/security/pwquality.conf.d
sudo tee /etc/security/pwquality.conf.d/10-enterprise.conf >/dev/null <<'EOF'
minlen = 12
minclass = 3
usercheck = 1
maxrepeat = 3
palindrome = 0
difok = 4
remember = 5
EOF
sudo chmod 644 /etc/security/pwquality.conf.d/10-enterprise.conf

# Password expiry
sudo sed -i 's/^PASS_MAX_DAYS.*/PASS_MAX_DAYS   90/' /etc/login.defs
sudo sed -i 's/^PASS_MIN_DAYS.*/PASS_MIN_DAYS   1/' /etc/login.defs
sudo sed -i 's/^PASS_WARN_AGE.*/PASS_WARN_AGE   14/' /etc/login.defs
grep -E '^PASS_(MAX|MIN|WARN)' /etc/login.defs

Note: /etc/login.defs only affects new accounts ; existing accounts need chage:

# Dry-run first
for u in $(awk -F: '$3>=1000 && $3<65534 {print $1}' /etc/passwd); do
  printf 'would chage -M 90 -m 1 -W 14 %s
' "${u}"
done

# Confirm then execute
for u in $(awk -F: '$3>=1000 && $3<65534 {print $1}' /etc/passwd); do
  sudo chage -M 90 -m 1 -W 14 "${u}"
done

sudo chage -l alice

Ubuntu 24.04:

sudo mkdir -p /etc/security/pwquality.conf.d
sudo tee /etc/security/pwquality.conf.d/10-enterprise.conf >/dev/null <<'EOF'
minlen = 12
minclass = 3
usercheck = 1
maxrepeat = 3
difok = 4
EOF

# Confirm PAM chain references pwquality
grep -n 'pam_pwquality\|pam_unix' /etc/pam.d/common-password

Critical verification: test with a throwaway account

sudo useradd -m pwtest
echo 'pwtest:short' | sudo chpasswd 2>&1
# Expected: failure, "password too short" / "Authentication token manipulation error"

sudo userdel -r pwtest 2>/dev/null

Risk & rollback: Stricter policy affects next password change . Don't raise minlen globally without notice. Recommended: apply policy, observe 2 weeks, then raise threshold. Rollback = delete drop-in: sudo rm /etc/security/pwquality.conf.d/10-enterprise.conf.

Login Failure Lockout

RHEL 9 defaults to pam_faillock in /etc/pam.d/system-auth and password-auth:

grep -n 'pam_faillock' /etc/pam.d/system-auth
# Default lines:
# auth        required      pam_faillock.so preauth silent deny=5 unlock_time=900
# auth        [default=die] pam_faillock.so authfail deny=5 unlock_time=900
# account     required      pam_faillock.so

RHEL 9 recommends authselect over direct PAM edits:

sudo authselect current
sudo authselect enable-feature with-faillock

# Parameters to /etc/security/faillock.conf (RHEL 8+ recommended location)
sudo tee /etc/security/faillock.conf >/dev/null <<'EOF'
deny = 5
unlock_time = 900
fail_interval = 900
even_deny_root = no
EOF

sudo authselect apply-changes
grep -vE '^\s*#|^\s*$' /etc/security/faillock.conf

Why even_deny_root = no : If root gets locked and no other admin account exists, the machine becomes inaccessible. Trade-off: root becomes brute-forceable, but risk is controlled since root remote login is disabled.

Ubuntu 24.04: Default lacks full faillock chain; must manually add with root session active :

sudo cp -a /etc/pam.d/common-auth "/root/common-auth.bak.$(date +%F_%H%M)"

sudo tee /etc/security/faillock.conf >/dev/null <<'EOF'
deny = 5
unlock_time = 900
fail_interval = 900
even_deny_root = no
EOF

sudo vim /etc/pam.d/common-auth
# Resulting structure (order critical):
auth  required          pam_faillock.so preauth
auth  [success=1 default=ignore]  pam_unix.so nullok
auth  [default=die]     pam_faillock.so authfail
auth  sufficient        pam_faillock.so authsucc
auth  requisite         pam_deny.so
auth  required          pam_permit.so

Verify immediately, do not close current session:

sudo -k && sudo -v   # Clear timestamp then re-authenticate; success means PAM chain intact
sudo faillock --user root

If sudo -v fails, restore instantly: sudo cp -a /root/common-auth.bak.* /etc/pam.d/common-auth. That's why the root session must stay open.

View/unlock: sudo faillock --user alice / sudo faillock --user alice --reset.

Risk: unlock_time = 900 lets attackers continuously lock an account (DoS). For on-call accounts, shorten to 300s; or enable faillock only for SSH, not local console.

Sudo Granularity

Principle: default deny, grant on demand, command whitelist enumerable, write operations auditable.

First audit current state — easiest place to find "legacy holes":

sudo grep -rn -E '^[^#].*ALL' /etc/sudoers /etc/sudoers.d/ 2>/dev/null
sudo visudo -c
# Expected: /etc/sudoers: parsed OK

Common dangerous patterns & verdicts: %ops ALL=(ALL) NOPASSWD: ALL → Full privilege escalation = root %ops ALL=(ALL) NOPASSWD: /bin/bash → Spawns shell = root alice ALL=(ALL) /usr/bin/vi /etc/* → vi internal :!sh = root bob ALL=(ALL) /usr/bin/find / -exec * → -exec runs arbitrary commands = root bob ALL=(ALL) /usr/bin/tee * → Can write any file including sudoers = root bob ALL=(ALL) /usr/bin/systemctl restart * → Can restart any service (incl. sshd) = Denial-of-service level, controllable

Verdict standard: If allowed command can spawn shell or write arbitrary files, it equals root. Alternative: write controlled wrapper scripts.

Recommended structure:

/etc/sudoers.d/00-aliases      # Aliases
/etc/sudoers.d/10-ops-read      # Read-only commands
/etc/sudoers.d/20-ops-service   # Service management
/etc/sudoers.d/30-dba           # Database
/etc/sudoers.d/90-emergency     # Emergency (with expiry, post-audit)

/etc/sudoers.d/00-aliases

:

Cmnd_Alias READ_LOG = /usr/bin/journalctl, /usr/bin/tail, /usr/bin/grep, /usr/bin/dmesg
Cmnd_Alias SERVICE_CTL = /usr/bin/systemctl status *, /usr/bin/systemctl restart nginx, \
    /usr/bin/systemctl reload nginx, /usr/bin/systemctl restart php-fpm
Cmnd_Alias NET_DIAG = /usr/sbin/ss, /usr/sbin/ip, /usr/bin/ping, /usr/bin/mtr, \
    /usr/sbin/tcpdump, /usr/bin/dig

User_Alias OPS = alice, bob, carol
User_Alias DBA = dave

/etc/sudoers.d/10-ops-read

:

Defaults:OPS  timestamp_timeout=5
Defaults:OPS  !lecture

OPS  ALL=(root)  NOPASSWD: READ_LOG, NET_DIAG
OPS  ALL=(root)  SERVICE_CTL

/etc/sudoers.d/30-dba

:

DBA  ALL=(root)  /usr/bin/systemctl restart mysqld, \
      /usr/bin/systemctl status mysqld

Key points: timestamp_timeout=5: re-authenticate after 5 minutes, avoids long-lived elevated terminals

Use full paths ( /usr/bin/systemctl) to prevent PATH injection systemctl restart * wildcard can restart any service (including sshd) — causes DoS but no shell; evaluate and accept or narrow

Never use Defaults !authenticate globally

Safe modification:

sudo visudo -f /etc/sudoers.d/10-ops-read
sudo chmod 0440 /etc/sudoers.d/10-ops-read
sudo chown root:root /etc/sudoers.d/10-ops-read
sudo visudo -c
sudo -l -U alice   # Verify from target user perspective

High-frequency trap: Files under /etc/sudoers.d/ must be 0440 owned by root:root, otherwise sudo silently ignores them — no error, you think it's active but it's not.

ls -l /etc/sudoers.d/
# Correct: -r--r----- 1 root root

Temporary elevation (safer than editing sudoers):

sudo tee /etc/sudoers.d/90-temp-alice >/dev/null <<'EOF'
Defaults:alice timestamp_timeout=5
alice ALL=(root) /usr/bin/systemctl restart nginx, /usr/bin/systemctl restart php-fpm
EOF
sudo chmod 0440 /etc/sudoers.d/90-temp-alice
sudo chown root:root /etc/sudoers.d/90-temp-alice

# Auto-delete after 2 hours, avoids "forgot to remove"
sudo systemd-run --on-active=2h --unit=sudo-temp-alice-cleanup \
  /bin/sh -c 'rm -f /etc/sudoers.d/90-temp-alice'
systemctl list-timers --all | grep sudo-temp

This turns "temporary elevation" from "remember to delete" into "auto-delete" — one of the highest ROI improvements in the hardening set.

Account Lifecycle Scripts

Provisioning:

#!/usr/bin/env bash
# /usr/local/sbin/user-onboard.sh
# Usage: sudo user-onboard.sh <username> <ssh_public_key_file>
set -euo pipefail

if [[ $# -ne 2 ]]; then
  echo "usage: $0 <username> <ssh_public_key_file>" >&2
  exit 2
fi

USERNAME="$1"
KEYFILE="$2"
LOG="/var/log/hardening/onboard.log"

mkdir -p "$(dirname "${LOG}")"
log() { printf '%s user=%s %s
' "$(date '+%F %T')" "${USERNAME}" "$*" | tee -a "${LOG}"; }

if [[ ! "${USERNAME}" =~ ^[a-z][a-z0-9_-]{2,31}$ ]]; then
  log "ERROR invalid username"
  exit 3
fi

if id "${USERNAME}" >/dev/null 2>&1; then
  log "ERROR user already exists"
  exit 4
fi

if [[ ! -r "${KEYFILE}" ]]; then
  log "ERROR key file not readable: ${KEYFILE}"
  exit 5
fi

# Validate public key format first
if ! ssh-keygen -l -f "${KEYFILE}" >/dev/null 2>&1; then
  log "ERROR invalid ssh public key"
  exit 6
fi

PRIV_GROUP="wheel"
if ! getent group wheel >/dev/null && getent group sudo >/dev/null; then
  PRIV_GROUP="sudo"
fi

useradd -m -s /bin/bash "${USERNAME}"
usermod -aG "${PRIV_GROUP}" "${USERNAME}"
chage -M 90 -m 1 -W 14 "${USERNAME}"
passwd -l "${USERNAME}"   # Lock password, key-only login

install -d -m 0700 -o "${USERNAME}" -g "${USERNAME}" "/home/${USERNAME}/.ssh"
install -m 0600 -o "${USERNAME}" -g "${USERNAME}" "${KEYFILE}" "/home/${USERNAME}/.ssh/authorized_keys"

log "OK created, groups=$(id -nG "${USERNAME}")"
ls -la "/home/${USERNAME}/.ssh"

sudo chmod 0755 /usr/local/sbin/user-onboard.sh
sudo /usr/local/sbin/user-onboard.sh alice /tmp/alice.pub

passwd -l

makes the account key-only; if legacy jump host needs password login, evaluate separately.

Key generation (employee runs locally, private key never leaves):

ssh-keygen -t ed25519 -a 100 -C "alice@company-2026"
# ed25519 shorter & faster; -a increases passphrase derivation iterations
# Only for legacy compatibility: ssh-keygen -t rsa -b 4096 -o -a 100

Deprovisioning:

#!/usr/bin/env bash
# /usr/local/sbin/user-offboard.sh
# Usage: sudo user-offboard.sh <username>
set -euo pipefail

USERNAME="$1"
LOG="/var/log/hardening/offboard.log"
ARCHIVE="/var/backup/offboard/$(date +%F)"
log() { printf '%s user=%s %s
' "$(date '+%F %T')" "${USERNAME}" "$*" | tee -a "${LOG}"; }

if ! id "${USERNAME}" >/dev/null 2>&1; then
  log "ERROR user not found"
  exit 3
fi

if [[ "${USERNAME}" == "root" ]]; then
  log "ERROR refuse to offboard root"
  exit 7
fi

mkdir -p "${ARCHIVE}"

# 1. Lock account, keep home & UID (for file ownership traceability)
usermod -L "${USERNAME}"
usermod -s /sbin/nologin "${USERNAME}"
chage -E 0 "${USERNAME}"

# 2. Kill active sessions
log "active sessions: $(who -u | awk -v u="${USERNAME}" '$1==u{c++} END{print c+0}')"
pkill -KILL -u "${USERNAME}" || true

# 3. Backup & remove authorized_keys to prevent stale keys on future re-enable
cp -a "/home/${USERNAME}/.ssh" "${ARCHIVE}/${USERNAME}-ssh" 2>/dev/null || true
getent passwd "${USERNAME}" > "${ARCHIVE}/${USERNAME}.passwd.txt"
if [[ -f "/home/${USERNAME}/.ssh/authorized_keys" ]]; then
  mv "/home/${USERNAME}/.ssh/authorized_keys" "${ARCHIVE}/${USERNAME}-authorized_keys"
  log "authorized_keys archived"
fi

# 4. Remove from privileged groups
gpasswd -d "${USERNAME}" wheel 2>/dev/null || true
gpasswd -d "${USERNAME}" sudo 2>/dev/null || true

log "OK disabled (home kept for 90 days per retention policy)"

Why "lock" not "delete": Deleting UID turns leftover files into orphans; a future same-name user would inherit old file permissions — classic incident. Standard practice: lock first, archive after 90 days, then delete.

Audit Evidence

Post-hardening evidence required for MLPS/ISO audits:

# 1. Loggable accounts with last login
sudo awk -F: '$3>=1000 && $3<65534 && $7 !~ /(nologin|false)$/ {print $1}' /etc/passwd \
 | while read -r u; do
   printf '%-16s %s
' "${u}" "$(sudo lastlog -u "${u}" | tail -1)"
 done

# 2. Effective password policy
sudo grep -hvE '^\s*#|^\s*$' /etc/security/pwquality.conf /etc/security/pwquality.conf.d/*.conf 2>/dev/null
sudo grep -E '^PASS_(MAX|MIN|WARN)' /etc/login.defs

# 3. Full sudo rules export
sudo grep -rn -v '^#\|^$' /etc/sudoers /etc/sudoers.d/ 2>/dev/null

# 4. Last 7 days failed login count
sudo journalctl -u sshd --since '7 days ago' | grep -c 'Failed password'

# 5. Empty password check
sudo awk -F: '($2==""){print "EMPTY:", $1}' /etc/shadow

Save as dated files for before/after comparison:

{
  echo "=== $(date '+%F %T') $(hostname) ==="
  awk -F: '$3>=1000 && $3<65534 {print $1, $7}' /etc/passwd
  grep -E '^PASS_' /etc/login.defs
  grep -rhv '^#\|^$' /etc/sudoers.d/ 2>/dev/null
} | sudo tee "/var/log/hardening/evidence-$(date +%F).txt" >/dev/null

Ports & Services: Minimize Exposure Surface

Judgment Standard

First answer: Who needs to access what port on this machine? Ports without answers get closed.

Inventory methods:

# 1. Listening ports & processes
sudo ss -tulnp

# 2. Only externally reachable (exclude 127.0.0.1 and ::1)
sudo ss -tulnp | awk 'NR==1 || ($5 !~ /^127\.0\.0\.1/ && $5 !~ /^\[::1\]/)'

# 3. Stats by protocol & address
sudo ss -tulnp | awk 'NR>1{print $1, $5}' | sort | uniq -c | sort -rn | head -30

On RHEL 9 you'll see 127.0.0.53:53 (systemd-resolved stub) — not exposure. Real exposure is 0.0.0.0: and *: prefixes. ss only shows local listen state, not whether firewall blocks it. Three layers must all be confirmed:

Layer 1: Cloud security group / network ACL (console or cloud API)
Layer 2: Host firewall (firewalld / ufw / nftables)
Layer 3: Service listen address (0.0.0.0 vs 127.0.0.1)

External scan is the only trustworthy evidence — scan from another machine in same subnet:

nmap -Pn -p 1-10000 --open -T4 10.20.30.40

# High-risk ports only, good for bulk patrol
nmap -Pn -p 22,23,25,111,139,445,1433,3306,3389,5432,6379,9200,11211,27017 10.20.30.40

Judgment: ports appearing in nmap output = real exposure. Compare against "should-expose" list; extras get closed.

SSH Hardening: The Critical Remote Entry

SSH is the sole entry for most machines and a hardening hotspot. All changes follow same flow: edit config → sshd -t syntax check → sshd -T view effective values → verify from new terminal → then next change. Do not change 10 items then restart.

Understand Config Precedence

RHEL 9 and Ubuntu 24.04 sshd_config both start with: Include /etc/ssh/sshd_config.d/*.conf OpenSSH applies first-match-wins for most options. If Include is at top, later same-name settings in main sshd_config do not override drop-in values.

Real consequence: you set PermitRootLogin no in main file, but /etc/ssh/sshd_config.d/50-cloud-init.conf has PermitRootLogin yes and Include comes first — your change is ineffective . This is the #1 cause of "thought root disabled, actually not".

sudo grep -n 'Include' /etc/ssh/sshd_config
sudo ls -la /etc/ssh/sshd_config.d/
sudo grep -rn -E 'PermitRootLogin|PasswordAuthentication|Port|PubkeyAuthentication' \
  /etc/ssh/sshd_config /etc/ssh/sshd_config.d/ 2>/dev/null

# Only trustworthy effective values
sudo sshd -T | grep -E '^(port|permitrootlogin|passwordauthentication|pubkeyauthentication|kbdinteractiveauthentication|maxauthtries|x11forwarding|allowtcpforwarding|clientaliveinterval|clientalivecountmax|logingracetime|allowusers|allowgroups) '

Correct handling: Put hardening in a late-sorting drop-in, and neutralize vendor drop-ins first.

# View cloud vendor drop-in (common on cloud VMs)
sudo cat /etc/ssh/sshd_config.d/50-cloud-init.conf 2>/dev/null

# Backup then empty it (better traceability than delete)
sudo cp -a /etc/ssh/sshd_config.d/50-cloud-init.conf \
  "/root/50-cloud-init.conf.bak.$(date +%F)" 2>/dev/null || true
sudo truncate -s 0 /etc/ssh/sshd_config.d/50-cloud-init.conf 2>/dev/null || true

Note: Some cloud vendors rewrite this file on every boot via cloud-init. Fix: disable the module in /etc/cloud/cloud.cfg.d/99-disable-ssh.conf or accept "config management redeploys on boot". Do not use chattr +i hard lock — it breaks image initialization and is hard to debug later.

Hardening Config Itself

sudo install -m 0600 /dev/null /etc/ssh/sshd_config.d/99-hardening.conf
sudo tee /etc/ssh/sshd_config.d/99-hardening.conf >/dev/null <<'EOF'
# Enterprise security baseline, centrally managed, do not edit directly
# Change process: submit change request -> single-node canary -> observe 24h -> fleet rollout

# --- Authentication ---
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitEmptyPasswords no
MaxAuthTries 4
MaxSessions 5

# --- Access restriction (adjust per actual ops group) ---
AllowGroups wheel ops

# --- Session control ---
ClientAliveInterval 300
ClientAliveCountMax 2
LoginGraceTime 30
MaxStartups 10:30:60

# --- Feature reduction ---
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTunnel no
PrintMotd no
Banner /etc/ssh/banner
LogLevel VERBOSE
EOF

cat > /tmp/banner.txt <<'EOF'
**********************************************************************
*  This system is a controlled device, authorized personnel only.     *
*  All operations are fully logged and audited; unauthorized access   *
*  will be pursued with legal liability.                              *
**********************************************************************
EOF
sudo install -m 0644 -o root -g root /tmp/banner.txt /etc/ssh/banner
rm -f /tmp/banner.txt

Per-item rationale with real risks: PermitRootLogin no: Block direct root login. Risk: None (if sudo account exists). Basis: Eliminate highest-value brute target. PasswordAuthentication no: Key-only. Risk: All password-dependent scripts break . Basis: Passwords cannot resist brute force. KbdInteractiveAuthentication no: Disable PAM interactive auth. Risk: Some 2FA schemes rely on it, evaluate. Basis: Equivalent to password login; leaving it = not disabling password. MaxAuthTries 4: Max 4 auth failures per connection. Risk: Too low breaks multi-key users. Basis: Rate-limit brute force. AllowGroups wheel ops: Only these groups can login. Risk: Misconfig locks everyone out . Basis: Whitelist more reliable than blacklist. ClientAliveInterval 300 + CountMax 2: Disconnect after 10min no response. Risk: Long tasks killed (use tmux). Basis: Prevent zombie sessions hogging resources. LoginGraceTime 30: Unauthenticated connection max 30s. Risk: Slow networks may fail. Basis: Prevent half-open connections exhausting MaxStartups. MaxStartups 10:30:60: Unauthenticated connection limits. Risk: None. Basis: Prevent concurrent handshake flood. AllowTcpForwarding no: Disable TCP forwarding. Risk: Breaks jump-host, DB tunnels. Basis: Prevent internal pivot. LogLevel VERBOSE: Log key fingerprints. Risk: Log volume increase. Basis: Audit can trace "which key used".

AllowTcpForwarding no needs caution: Many teams use ssh -L for temporary DB tunnels, internal pivot. If must keep, scope by group:

# Only allow forwarding for dba group (Match block at file end)
Match Group dba
    AllowTcpForwarding yes

Stepwise Tightening Execution Order

Do not apply all at once. Follow order, verify each step before next:

Step 1: LogLevel VERBOSE (observation only, zero risk)
Step 2: MaxAuthTries 4, LoginGraceTime 30, MaxStartups (low risk, may affect slow nets)
Step 3: AllowGroups (confirm target users in group, have backup channel)
Step 4: PasswordAuthentication no (highest risk, must verify key login works)
Step 5: PermitRootLogin no (after confirming regular account sudo works)
Step 6: ClientAliveInterval / session control (affects long tasks, notify ahead)
Step 7: AllowTcpForwarding no (affects tunnel workflows, inventory first)

Each step verification identical:

# 1. Syntax check (stop if fails, do not reload)
sudo sshd -t && echo "config OK"

# 2. View effective values
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|allowgroups|maxauthtries) '

# 3. Graceful reload (keeps existing connections)
sudo systemctl reload sshd

# 4. From new terminal: can login, can sudo
ssh [email protected] 'id; sudo -n true && echo sudo-ok'

# 5. Verify prohibited items actually blocked
ssh [email protected]          # Expect "Permission denied (publickey)"
ssh -o PubkeyAuthentication=no [email protected]  # Expect "Permission denied", no password prompt

# 6. Keep old session 30 min then close

Why reload not restart : reload re-reads config but keeps listening socket and existing connections; even if new config is broken, current sessions survive for fix. restart drops all connections; broken config = total loss of access. reload limitation: Port, ListenAddress, AddressFamily changes require restart. So before changing port, ensure firewall allows new port and test connectivity from another machine.

Port Change Reality Check

Many checklists start with "change SSH port". Recommendation: Treat as optional, not mandatory.

Limited benefit: Only blocks indiscriminate full-internet scans of port 22; targeted attacks scan all ports anyway.

Clear cost: Every automation script, CI, monitoring, bastion config, firewall rule, security group must change; one miss = incident.

Real gain comes from key auth + root disable + source restriction — these three outweigh port change.

If you must change (e.g., reduce log noise), follow this flow:

# 1. Open new port in firewall FIRST (port 22 still open)
sudo firewall-cmd --permanent --add-port=22022/tcp
sudo firewall-cmd --reload
sudo firewall-cmd --list-ports

# 2. Change config
sudo tee /etc/ssh/sshd_config.d/98-port.conf >/dev/null <<'EOF'
Port 22022
EOF
sudo sshd -t && echo OK

# 3. Confirm SELinux allows new port (RHEL mandatory)
sudo semanage port -l | grep ssh_port_t
sudo semanage port -a -t ssh_port_t -p tcp 22022
sudo semanage port -l | grep ssh_port_t

# 4. Restart (port change requires restart)
sudo systemctl restart sshd
sudo ss -tlnp | grep -E '22022|:22 '

# 5. Verify new port from another machine
ssh -p 22022 [email protected] 'echo ok'

# 6. Only after verification, close 22
sudo firewall-cmd --permanent --remove-port=22/tcp
sudo firewall-cmd --reload

Step 3 is most often missed: RHEL SELinux defaults to only allowing sshd on port 22. Without semanage adding policy, sshd fails to start or bind, logging Bind to port 22022 on 0.0.0.0 failed: Permission denied. No need for setenforce 0; adding policy suffices. sudo ausearch -m avc -ts recent | grep sshd Rollback: Revert to 22, systemctl restart sshd, restore firewall. Prerequisite: keep 22 open until new port verified .

Source Restriction

Beyond "who can login", restrict "from where". Three layers:

Cloud security group: Only office egress IPs, VPN CIDRs, bastion IPs.

Host firewall: Rich rules with source CIDR.

sshd_config Match Address : Finer granularity.

sudo firewall-cmd --permanent --remove-service=ssh
sudo firewall-cmd --permanent --add-rich-rule=
'rule family="ipv4" source address="10.20.100.0/24" port port="22" protocol="tcp" accept'
sudo firewall-cmd --permanent --add-rich-rule=
'rule family="ipv4" source address="203.0.113.0/24" port port="22" protocol="tcp" accept'
sudo firewall-cmd --reload
sudo firewall-cmd --list-rich-rules

This step is extremely high risk: If your current IP isn't in allow list, --reload cuts you off immediately. Before executing:

# Confirm your source IP
echo "${SSH_CLIENT}" | awk '{print $1}'

# First trial-run with --timeout, auto-reverts after 2 min, test during window
sudo firewall-cmd --add-rich-rule=
'rule family="ipv4" source address="10.20.100.0/24" port port="22" protocol="tcp" accept' --timeout=120

# Timer expires, rule auto-removed; verify connectivity then write permanent

firewall-cmd --timeout

is the best practice against lockout; almost all firewalld changes can be rehearsed with it first.

Disable Unneeded Services

# 1. All enabled services
systemctl list-unit-files --type=service --state=enabled

# 2. Processes listening on ports
sudo ss -tulnp | awk 'NR>1{print $7}' | grep -o 'users:((".*",' | sed 's/users:(("//; s/",//' | sort -u

# 3. Cross-ref: which enabled services have no listening ports but still run

Typical disable candidates ( evaluate per need, don't copy blindly ): postfix: Local mail delivery. Disable impact: Cron output emails fail (set MAILTO=""); some monitoring/man need SMTP. rpcbind / nfs-server: NFS. Disable impact: NFS mounts fail; check findmnt -t nfs4 first. avahi-daemon: mDNS service discovery. Disable impact: Safe to disable on servers; desktop disable breaks LAN discovery. cups: Print service. Disable impact: No impact. bluetooth: Bluetooth. Disable impact: No impact. firewalld: Firewall. Never disable . sshd: SSH. Never disable .

# Stop first observe, then disable
sudo systemctl stop cups 2>/dev/null
sudo systemctl disable cups 2>/dev/null
systemctl is-enabled cups 2>/dev/null; systemctl is-active cups 2>/dev/null

RHEL note: Direct disable may be pulled back via Wants= from other units. Use mask for hard disable:

sudo systemctl mask cups.socket cups.service 2>/dev/null
systemctl is-enabled cups.socket   # Expect "masked"

Risk: mask may cause package installs to fail if they need to start the service. Rollback: sudo systemctl unmask cups.socket cups.service.

Access Control: Three-Layer Defense

Accounts solve "who", ports solve "from where", this section solves "what can they do after entry". Three layers: sudo (command-level), firewall (network-level), SELinux/AppArmor (syscall-level) .

Network Layer: Default Deny

sudo firewall-cmd --get-default-zone
sudo firewall-cmd --get-active-zones
sudo firewall-cmd --zone=public --list-all

Principle: Define explicit ports + sources, not blanket allow.

# Recommended: explicit port + source
sudo firewall-cmd --permanent --zone=public --add-rich-rule=
'rule family="ipv4" source address="10.20.0.0/16" port port="8080" protocol="tcp" accept'

# Not recommended: no source restriction
sudo firewall-cmd --permanent --add-port=8080/tcp

Creating zones per business group is cleaner:

sudo firewall-cmd --permanent --new-zone=lbonly
sudo firewall-cmd --permanent --zone=lbonly --add-source=10.20.5.0/24
sudo firewall-cmd --permanent --zone=lbonly --add-port=8080/tcp
sudo firewall-cmd --permanent --zone=lbonly --set-target=DROP
sudo firewall-cmd --reload

# Multi-NIC: each interface can bind different zone
sudo firewall-cmd --permanent --zone=lbonly --change-interface=eth1

--set-target=DROP

stricter than default (drops packets without RST, less scanner-friendly); trade-off: clients see timeouts, remember during troubleshooting.

Ubuntu ufw:

sudo ufw status verbose
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow from 10.20.100.0/24 to any port 22 proto tcp
sudo ufw allow 'Nginx Full'
sudo ufw --dry-run enable   # Preview actions
sudo ufw enable
sudo ufw status numbered

ufw enable takes effect immediately. Must allow current source's port 22 before enabling, else instant lockout:

sudo ufw allow from "$(echo "${SSH_CLIENT}" | awk '{print $1}')" to any port 22 proto tcp

nftables direct: RHEL 9 can use nft directly, but do not run alongside firewalld — they overwrite each other.

sudo nft list ruleset
sudo nft list ruleset > "/root/nftables.backup.$(date +%F_%H%M)"  # Must before changes

SELinux: Don't Disable, Tune

RHEL's most often broken layer via "don't know how to tune, so disable". Default targeted policy covers most services; correct approach: on denial, add policy, not setenforce 0.

sudo sestatus
getenforce
# Expect Enforcing

sudo ausearch -m avc -ts recent

sudo dnf install -y setroubleshoot-server
sudo sealert -a /var/log/audit/audit.log | head -40

# Generate candidate policy (review before install, don't -M blindly)
sudo ausearch -m avc -ts today | audit2allow -m my_custom_policy > /tmp/my.te
cat /tmp/my.te

/tmp/my.te

looks like:

module my_custom_policy 1.0;

require {
    type httpd_t;
    type var_log_t;
    class dir { add_name remove_name write };
    class file { create open getattr setattr };
}

#============= httpd_t ==============
allow httpd_t var_log_t:dir { add_name remove_name write };
allow httpd_t var_log_t:file { create open getattr setattr };

Review points: Rules involving shadow_t, passwd_file_t, init_t high-sensitivity types — never install directly ; indicates config error (e.g., web server reading /etc/shadow). Fix path or file label instead.

Non- audit2allow fixes, priority order:

# A. Change file label (most common)
sudo semanage fcontext -a -t httpd_sys_content_t "/data/www(/.*)?"
sudo restorecon -Rv /data/www

# B. Change port label (service port change)
sudo semanage port -a -t http_port_t -p tcp 8081

# C. Toggle booleans (official pre-built switches)
sudo getsebool -a | grep httpd_can_network_connect
sudo setsebool -P httpd_can_network_connect on

# D. Only then custom policy module
sudo checkmodule -M -m -o /tmp/my.mod /tmp/my.te
sudo semodule_package -o /tmp/my.pp -m /tmp/my.mod
sudo semodule -i /tmp/my.pp
sudo semodule -l | grep my_custom

# Rollback
sudo semodule -r my_custom_policy

Ubuntu AppArmor:

sudo aa-status
sudo aa-complain /usr/sbin/named    # Temporary complain for debugging
sudo aa-enforce /usr/sbin/named    # Restore enforce

Profiles in /etc/apparmor.d/; easier to read than SELinux but less expressive. Hardening focus: ensure critical service profiles are in enforce, not writing custom policies.

Audit: Leave Evidence of Actions

systemctl status auditd --no-pager
sudo auditctl -s

sudo tee /etc/audit/rules.d/60-access.rules >/dev/null <<'EOF'
# Privilege escalation
-w /etc/sudoers -p wa -k sudoers_change
-w /etc/sudoers.d/ -p wa -k sudoers_change
-w /etc/passwd -p wa -k account_change
-w /etc/shadow -p wa -k account_change
-w /etc/group -p wa -k account_change

# SSH config
-w /etc/ssh/sshd_config -p wa -k sshd_config
-w /etc/ssh/sshd_config.d/ -p wa -k sshd_config

# Cron jobs
-w /etc/cron.d/ -p wa -k cron_change
-w /etc/crontab -p wa -k cron_change
-w /var/spool/cron/ -p wa -k cron_change

# Privileged commands
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -k root_cmd
EOF

sudo augenrules --load
sudo auditctl -l | wc -l

execve audit performance cost: Logs all root commands; high-frequency container environments generate huge logs. Enable only on host (non-container nodes). Measure log volume first:

sudo ausearch --summary -ts yesterday | head -20
sudo du -sh /var/log/audit/

If >5GB/day, narrow to specific commands:

-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F path=/usr/bin/passwd -k priv_tool

Query:

sudo ausearch -k sudoers_change -i | tail -40
sudo aureport --auth -ts today
sudo aureport --failed --summary -ts today

Sudo + Audit Integration

# /etc/sudoers.d/00-logging
Defaults  log_input, log_output
Defaults  iolog_dir=/var/log/sudo-io/%{user}
Defaults  logfile=/var/log/sudo.log
Defaults  syslog=local2

sudo mkdir -p /var/log/sudo-io
sudo chmod 0700 /var/log/sudo-io
sudo visudo -c
sudo id
sudo grep -c 'COMMAND' /var/log/sudo.log
ls -la /var/log/sudo-io/

log_input

/ log_output enable full session replay — high value for sensitive ops audit, but heavy disk usage and records keystrokes (including mistyped passwords) . More precise: enable only for high-risk commands:

Defaults!/usr/bin/mysqldump  log_output, iolog_dir=/var/log/sudo-io/%{user}

Often overlooked: regular users can see others' running commands. If compliance requires isolation, use hidepid:

# /etc/fstab add mount option for /proc, effective on reboot
# proc  /proc  proc  defaults,hidepid=2,gid=proc  0 0
sudo findmnt -no OPTIONS /proc

hidepid=2

restricts users to seeing only their own processes. Risk: Many monitoring agents and ps -based ops habits break; must test on canary first.

Verification: Hardening Actually Effective?

Checklist verification, not "looks changed". Every item has explicit command + expected output.

#!/usr/bin/env bash
# /usr/local/sbin/hardening-verify.sh — hardening result auto-verify (read-only, exit 0=all pass)
set -uo pipefail

PASS=0
FAIL=0

check() {
  local desc="$1"; shift
  if bash -c "$@" >/dev/null 2>&1; then
    printf '[ PASS ] %s
' "${desc}"
    PASS=$((PASS+1))
  else
    printf '[ FAIL ] %s
' "${desc}"
    FAIL=$((FAIL+1))
  fi
}

check "Only root has uid 0" \
  bash -c '[[ $(awk -F: "\$3==0{print \$1}" /etc/passwd | wc -l) -eq 1 ]]'
check "No empty password accounts" \
  bash -c '! awk -F: "(\$2==\"\")" /etc/shadow | grep -q .'
check "Wheel group members <= 5" \
  bash -c '[[ $(getent group wheel | cut -d: -f4 | tr "," "
" | grep -c .) -le 5 ]]'

check "PermitRootLogin=no" \
  bash -c 'sshd -T 2>/dev/null | grep -qx "permitrootlogin no"'
check "PasswordAuthentication=no" \
  bash -c 'sshd -T 2>/dev/null | grep -qx "passwordauthentication no"'
check "KbdInteractiveAuthentication=no" \
  bash -c 'sshd -T 2>/dev/null | grep -qx "kbdinteractiveauthentication no"'

check "sudoers.d files mode 0440" \
  bash -c '! find /etc/sudoers.d -type f ! -perm 0440 -print | grep -q .'
check "sudoers syntax OK" \
  bash -c 'visudo -c >/dev/null 2>&1'

check "firewalld running" \
  bash -c 'systemctl is-active --quiet firewalld'
check "No full port range open" \
  bash -c '! firewall-cmd --list-ports 2>/dev/null | grep -qE "0-65535|65535"'

check "SELinux Enforcing" \
  bash -c '[[ $(getenforce) == "Enforcing" ]]'

check "auditd running" \
  bash -c 'systemctl is-active --quiet auditd'
check "Audit rules loaded" \
  bash -c '[[ $(auditctl -l 2>/dev/null | grep -c .) -ge 5 ]]'

printf '
==== PASS=%d FAIL=%d ====
' "${PASS}" "${FAIL}"
[[ ${FAIL} -eq 0 ]]

sudo chmod 0755 /usr/local/sbin/hardening-verify.sh
sudo /usr/local/sbin/hardening-verify.sh
echo "exit=$?"

Script notes: "$@" passes command preserving quotes; bash -c escapes $; set -uo pipefail without -e to collect all failures instead of exiting on first.

Schedule daily run with reporting:

sudo tee /etc/cron.d/hardening-verify >/dev/null <<'EOF'
0 7 * * * root /usr/local/sbin/hardening-verify.sh > /var/log/hardening/verify-$(date +\%F).log 2>&1 || logger -t hardening "verification FAILED"
EOF
sudo chmod 644 /etc/cron.d/hardening-verify

%

in cron must be escaped as \%; missing escape = job never runs.

Risk, Rollout & Canary

Phased Rollout (by Impact)

P0 : Inventory, backup, auditd rules. Impact: None (read-only+logs). Window: Any. Rollback: None.

P1 : Password policy, faillock. Impact: Next password change / login failure. Window: Business day. Rollback: Low (delete files).

P2 : Sudo granularity. Impact: Privilege escalation behavior. Window: Business day. Rollback: Low (restore files).

P3 : Disable unused services. Impact: Single host. Window: Off-peak. Rollback: Low.

P4 : Firewall tighten (keep 22). Impact: Single host business ports. Window: Off-peak + app owner present. Rollback: Medium.

P5 : SSH auth (key, disable root). Impact: All logins. Window: Off-peak, canary 1 host first. Rollback: Medium-high.

P6 : SSH port change. Impact: All automation. Window: Maintenance window. Rollback: High.

P7 : Firewall source restriction. Impact: All logins. Window: Maintenance window + out-of-band. Rollback: High.

Canary method: P5-P7 start on 1 non-critical host, observe 24h (cover one deploy + on-call cycle), expand to 10%, then fleet. Record batch ID + host list per wave for instant blast-radius isolation.

Rollback Quick Reference

DATE=2026-11-02
B="/var/log/hardening/${DATE}"

# SSH
sudo cp -a "${B}/sshd_config.d.orig/." /etc/ssh/sshd_config.d/
sudo cp -a "${B}/sshd_config.orig" /etc/ssh/sshd_config
sudo sshd -t && sudo systemctl reload sshd

# sudoers
sudo cp -a "${B}/sudoers.orig" /etc/sudoers
sudo visudo -c

# PAM
sudo cp -a "${B}/pam.d.orig/." /etc/pam.d/
sudo -k && sudo -v

# Firewall: discard runtime changes, revert to permanent config
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

# Disabled services
sudo systemctl unmask cups.socket cups.service
sudo systemctl enable --now cups.socket cups.service

Don't use --runtime-to-permanent for rollback: It pushes current runtime rules into permanent config (used to save temporary changes after verification). Rollback needs --reload.

Top 5 Incident Causes (by frequency)

Switched SSH auth to key-only without verifying keys first. Before cutting password, test a test account full login in new terminal, confirm authorized_keys perms (dir 700, file 600, correct owner) and home dir not group-writable ( StrictModes rejects):

sudo sshd -T | grep strictmodes
ls -ld /home/alice /home/alice/.ssh
ls -l /home/alice/.ssh/authorized_keys
sudo journalctl -u sshd -n 30 --no-pager | grep -i 'authentication refused\|bad ownership\|permission'

AllowGroups group name wrong (e.g., wrote wheel on Ubuntu where group is sudo). Result: everyone locked out, only console rescue. Pre-change verify:

getent group wheel
id alice

Firewall change cuts self off. Always --timeout trial-run first, always keep out-of-band channel.

Sudoers file not 0440 — config silently ignored. Verify with sudo -l -U <user>, not just file content.

SELinux port policy missed, service fails to start, someone runs setenforce 0 to cover up. Correct: semanage port -a add policy, and add ausearch -m avc to patrol checks.

Maintenance: Make Hardening Last

Configuration Management Ownership

Manual hardening drifts on next OS update, cloud image rebuild, or someone's ad-hoc change. All hardening items into Ansible/Salt, auto-restored on rebuild:

# Role snippet (illustrative, not full playbook)
- name: ensure sshd hardening drop-in exists
  ansible.builtin.template:
    src: 99-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/99-hardening.conf
    owner: root
    group: root
    mode: "0600"
    validate: /usr/sbin/sshd -t -f %s
  notify: reload sshd

- name: ensure sudoers drop-ins
  ansible.builtin.copy:
    src: "{{ item }}"
    dest: "/etc/sudoers.d/{{ item | basename }}"
    owner: root
    group: root
    mode: "0440"
    validate: /usr/sbin/visudo -cf %s
  loop:
    - 00-aliases
    - 10-ops-read

validate

is critical: template render error makes Ansible fail without writing, preventing broken sshd_config causing fleet-wide lockout.

Patrol & Alerting

Turn verification script output into monitoring items — more effective than "monthly manual check":

# Output single metric for Zabbix/Telegraf collection
cat > /usr/local/sbin/hardening-metric.sh <<'SCRIPT'
#!/usr/bin/env bash
set -uo pipefail
fail=0
[[ $(awk -F: '$3==0{print $1}' /etc/passwd | wc -l) -eq 1 ]] || fail=$((fail+1))
awk -F: '($2==""){print $1}' /etc/shadow | grep -q . && fail=$((fail+1))
sshd -T 2>/dev/null | grep -qx "permitrootlogin no" || fail=$((fail+1))
sshd -T 2>/dev/null | grep -qx "passwordauthentication no" || fail=$((fail+1))
find /etc/sudoers.d -type f ! -perm 0440 -print 2>/dev/null | grep -q . && fail=$((fail+1))
[[ $(getenforce 2>/dev/null) == "Enforcing" ]] || fail=$((fail+1))
echo "${fail}"
SCRIPT
chmod 0755 /usr/local/sbin/hardening-metric.sh
/usr/local/sbin/hardening-metric.sh   # Should output 0

Collection (Zabbix Agent2 UserParameter):

# /etc/zabbix/zabbix_agent2.d/hardening.conf
UserParameter=hardening.failures,/usr/local/sbin/hardening-metric.sh

sudo systemctl restart zabbix-agent2
zabbix_agent2 -t hardening.failures
# Expect: hardening.failures [s|0]

Threshold is >0 because any failing item means baseline broken. Alert must point to specific item — don't make on-call engineer guess; attach hardening-verify.sh detailed output as alert context.

Change Audit & Exception Management

Exceptions exist: legacy box needs password login, service must run on port 22. Exceptions must be recorded, not silently skipped.

/var/lib/hardening/exceptions.tsv
Fields: hostname  check_id  reason  owner  expire_date  ticket

Example:
db-legacy-01  ssh_password_auth  Legacy ERP client lacks key support  zhangsan  2027-01-31  SEC-2041

Verification script reads this file, downgrades matching items from FAIL to WAIVED, separate count:

#!/usr/bin/env bash
# Check for expired exceptions
EXC="/var/lib/hardening/exceptions.tsv"
if [[ -r "${EXC}" ]]; then
  today=$(date +%F)
  awk -F'\t' -v t="${today}" 'NR>1 && $5 < t {print "EXPIRED:", $0}' "${EXC}"
fi

Expired exceptions must auto-alert — only effective defense against "temporary exception becoming permanent backdoor".

Real Incident Postmortems: Three Hardening-Induced Outages

All from real scenarios, sanitized. Common thread: Change itself correct; failure due to no verification, no canary, no fallback.

Case 1: Disabling Password Auth Broke All Automation

Symptom: PermitRootLogin no + PasswordAuthentication no fleet-wide; 10 min later distributed storage monitoring collection halted, backups failed, patrol scripts mass errors.

Initial guess: Network or auth service anomaly.

Investigation:

# From failing host check SSH logs
sudo journalctl -u sshd -n 50 --no-pager | grep -E 'Failed|refused|Not allowed'
# Key lines:
# Connection closed by authenticating user root 10.20.9.5 port 51234 [preauth]
# User root not allowed because not listed in AllowUsers

Root cause: Backup & patrol scripts used root + password login to each node (passwords stored in plaintext in config). Hardening closed both paths simultaneously.

Fix (lowest risk first):

Create dedicated automation account svc-automation with dedicated SSH key, add to AllowGroups automation.

Change scripts from root@ to svc-automation@, password → key.

Manage keys via Ansible Vault or Vault, no plaintext.

Restore collection & backup, verify one full run.

Verification:

ssh -i /etc/ansible/keys/svc-automation [email protected] 'id'
sudo -l -U svc-automation
ansible all -m ping -u svc-automation --private-key /etc/ansible/keys/svc-automation

Rollback plan: If scripts can't be updated quickly, temporarily re-enable password only on affected hosts (keep PermitRootLogin no), restrict source to automation server IP:

sudo tee /etc/ssh/sshd_config.d/97-emergency.conf >/dev/null <<'EOF'
Match Address 10.20.9.5
    PasswordAuthentication yes
EOF
sudo sshd -t && sudo systemctl reload sshd

Postmortem lesson: Must inventory "who uses password login" before hardening, not assume "everyone uses keys".

# Inventory method: check historical logs for successful password logins
sudo journalctl -u sshd --since '30 days ago' | \
  grep -E 'Accepted password' | awk '{print $NF, $(NF-3)}' | sort | uniq -c | sort -rn | head -20

Output "source IP + user" list = mandatory pre-hardening remediation list.

Case 2: Firewall Rule Killed Load Balancer Health Checks

Symptom: Firewall changed from --add-port=8080/tcp to rich rule (only allow 10.20.0.0/16); external users got 502, but curl 127.0.0.1:8080 on host worked.

Investigation:

sudo firewall-cmd --list-all
# public (active)
# target: default
# sources:
# services: dhcpv6-client ssh
# ports:
# rich rules:
#   rule family="ipv4" source address="10.20.0.0/16" port port="8080" protocol="tcp" accept

Root cause: Load balancer in another CIDR 10.30.0.0/16; its health checks sourced from that CIDR (often SNAT'd). Health checks failed → backend removed → users see 502.

Verification (critical step):

# From load balancer test backend connectivity
curl -sS -m 2 -o /dev/null -w '%{http_code}
' http://10.20.30.40:8080/healthz
# Expect 200, actual timeout → confirms firewall

# On host check if packets arrive (prove blocked on wire vs never arrived)
sudo tcpdump -nn -i eth0 'tcp port 8080' -c 10

If tcpdump sees nothing → upstream security group dropped. If sees SYN but no SYN-ACK → host firewall DROP (with --set-target=DROP or implicit deny).

# Check firewalld deny counters
sudo firewall-cmd --get-log-denied
sudo firewall-cmd --set-log-denied=all   # Temp enable logging (verbose, disable after)
sudo journalctl -k -f | grep -i 'filter_IN'

Fix: Add LB CIDR to allow list, keep original:

sudo firewall-cmd --permanent --zone=public --add-rich-rule=
'rule family="ipv4" source address="10.30.0.0/16" port port="8080" protocol="tcp" accept'
sudo firewall-cmd --reload

Rollback: Delete rich rule, restore --add-port=8080/tcp, re-tighten after CIDR inventory.

Postmortem lesson: Before tightening firewall, must identify all source CIDRs: business callers, LB health checks, monitoring probes, log collectors, backup. Method: "log only, don't block" for a week:

# Use nftables log action for observation (no allow/deny change)
# Or tcpdump sample source stats
sudo timeout 3600 tcpdump -nn -i eth0 'tcp port 8080 and tcp[tcpflags] & tcp-syn != 0' \
  -c 20000 2>/dev/null | awk '{print $3}' | cut -d. -f1-3 | sort | uniq -c | sort -rn | head -20

Run this for a day → real source CIDR distribution, more reliable than asking people.

Case 3: Faillock Locked On-Call Account

Symptom: Midnight alert, on-call engineer mistypes password a few times, account locked for unlock_time=900 (15 min). Had to find someone else.

Root cause: deny=5 too tight for "wrong password + keyboard layout" scenarios; no exception mechanism.

Fix:

Critical accounts (on-call) unlock_time reduced to 300s.

Move to bastion/SSO — remove "remember password" from humans.

Provide controlled self-unlock:

# On-call host: controlled unlock script, requires second person's sudo auth
cat > /usr/local/sbin/fail-unlock.sh <<'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail
if [[ $# -ne 1 ]]; then echo "usage: $0 <username>" >&2; exit 2; fi
u="$1"
if ! id "${u}" >/dev/null 2>&1; then echo "no such user: ${u}" >&2; exit 3; fi
faillock --user "${u}"
faillock --user "${u}" --reset
logger -t fail-unlock "unlocked ${u} by $(id -un)"
faillock --user "${u}"
SCRIPT
chmod 0755 /usr/local/sbin/fail-unlock.sh

Key is the logger line: Every unlock leaves audit trail — later can query "who unlocked whom when".

Rollback: If self-unlock abused, disable script ( chmod 0700) and revert to manual process.

Acceptance Checklist

Hardening complete only when every item confirmed; missing one = not delivered:

[ ] Pre-hardening baseline backed up to /var/log/hardening/<date>/, includes sshd -T, sudoers, pam.d, firewall rules

[ ] Only one uid=0 account (root)

[ ] No empty password accounts

[ ] Password policy effective (pwquality config + weak password rejected in test)

[ ] Faillock effective (tested 5 failures → lock)

[ ] All sudoers.d files mode 0440, visudo -c passes

[ ] Sudo rules: no NOPASSWD:ALL, no shell-spawning commands

[ ] One person one account; shared accounts registered with source & command limits

[ ] SSH: PermitRootLogin no, PasswordAuthentication no, verified with real account login success

[ ] SSH: AllowGroups group name correct, members as expected

[ ] SELinux Enforcing, AVC patrol in place

[ ] Firewall: no full port range open, SSH source restricted

[ ] External nmap matches "should-expose port list"

[ ] auditd running, key file writes & root commands audited

[ ] hardening-verify.sh all PASS

[ ] Verification results fed into monitoring alerts

[ ] exceptions.tsv established, no expired items

[ ] Account provision/deprovision scripts delivered & exercised once

[ ] Config management owns hardening items; rebuild restores automatically

[ ] Rollback steps in runbook, includes out-of-band access method

[ ] Runbook documents "SSH lockout → console login steps" including console URL, credential retrieval (no plaintext in runbook), first actions (systemctl status sshd + journalctl -u sshd -n 100). Without this, previous 19 items useless in real incident.

Summary

Hardening output is capability, not config: Whether it holds in 6 months depends on config management ownership, patrol alerts, exception management.

Accounts are audit foundation: Without one-person-one-account, logs trace to nobody; disable root remote + key auth are highest ROI.

Port reduction needs three layers: Security group, host firewall, service listen address; checking only one guarantees leaks.

sshd -T > sshd_config : Include order decides if your file edits actually apply.

SSH changes require fallback: Out-of-band, root session, reload not restart, pre-change backup — four hard requirements.

firewall-cmd --timeout is cheapest insurance: Any potentially disconnecting rule, trial-run with timeout first.

Don't disable SELinux: semanage port/fcontext + setsebool solve 90% issues; disabling just pushes problems forward.

Permanent exceptions more dangerous than permanent config: Every exception needs owner, expiry, auto-alert.

Hardening-caused incidents mostly not "wrong change" but "no verification" and "no canary".

Final word: The hardest part of security hardening isn't technology — it's advancing without business disruption . Commands are searchable; the discipline of "inventory first, canary second, keep fallback, enable rollback" is what makes hardening actually stick.