How to Respond When Your Server Is Brute‑Force Attacked: Practical SSH Hardening Guide

1. Determine Incident Severity

Record timestamps in UTC before any analysis. Answer four questions:

Is the attack still ongoing? Identify source IPs, usernames, and request frequency.

Has any login succeeded? Verify whether the source is authorized.

After a successful login, are there signs of privilege escalation, persistence, lateral movement, or data access?

Is there an alternative management channel (cloud console, out‑of‑band console, bastion host)?

Classify the incident based on the answers:

Only failed attempts – treat as brute‑force and apply hardening.

Unexpected successful login – treat as a suspected breach.

Unknown keys, scheduled tasks, services, or outbound connections – treat as a confirmed breach and prioritize isolation and rebuilding.

2. Secure the Boundary Before Making Changes

Ensure at least one management channel (cloud console, serial console, or bastion) remains reachable.

Keep the current SSH session open; do not log out.

Open a second terminal to test new configurations before applying them to the original session.

Record existing firewall rules, effective SSH configuration, active sessions, and listening ports.

Create a disk snapshot or a read‑only forensic copy of the VM; do not rely solely on in‑host backups.

Run syntax checks and perform a staged reload (avoid a full restart) before applying changes.

Collect the current state:

date --utc --iso-8601=seconds
timedatectl status
hostnamectl
ss -lntp
ss -tnp state established '( sport = :22 )'
who -a
w
systemctl status sshd --no-pager 2>/dev/null || systemctl status ssh --no-pager

3. Preserve Evidence – Do Not Destroy the Scene

If a suspicious successful login appears, immediately copy cloud‑audit logs, load‑balancer or firewall logs, identity‑provider logs, and a VM snapshot to a trusted external location. Local logs can be tampered with after a root compromise.

Collect read‑only information on the host:

#!/usr/bin/env bash
set -euo pipefail
umask 077
OUT_DIR="EVIDENCE_DIR/ssh-incident-$(date -u +%Y%m%dT%H%M%SZ)"
install -d -m 0700 "$OUT_DIR"
date --utc --iso-8601=seconds > "$OUT_DIR/time.txt"
hostnamectl > "$OUT_DIR/hostnamectl.txt" 2>&1 || true
who -a > "$OUT_DIR/who.txt" 2>&1 || true
w > "$OUT_DIR/w.txt" 2>&1 || true
ss -lntup > "$OUT_DIR/ss-listen.txt" 2>&1 || true
ss -tnp > "$OUT_DIR/ss-connections.txt" 2>&1 || true
ps auxwwf > "$OUT_DIR/ps.txt" 2>&1 || true
systemctl list-units --type=service --state=running --no-pager > "$OUT_DIR/running-services.txt" 2>&1 || true
journalctl -u sshd --since 'START_TIME' --until 'END_TIME' --no-pager > "$OUT_DIR/sshd-journal.txt" 2>&1 || true
find "$OUT_DIR" -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum > "$OUT_DIR/SHA256SUMS"
echo "Evidence directory: $OUT_DIR"

Do not delete or modify logs, rotate files, reinstall software, or reboot the host before the forensic copy is secured.

4. Verify Successful Logins

Search for accepted logins:

journalctl -u sshd --since 'START_TIME' --no-pager | grep -E 'Accepted (password|publickey)'
last -Fai | head -n 100
lastb -Fai | head -n 100

When an Accepted publickey line appears, locate the key fingerprint in the OpenSSH logs and compare it with the asset inventory.

5. Investigate Processes, Accounts, and Persistence

Check running sessions and processes that may indicate malicious activity:

who -a
w
ss -tnp
ps auxwwf
pstree -ap

Inspect executable paths, parent‑child relationships, start times, and hashes to avoid being fooled by renamed processes.

List authorized keys for all users without overwriting existing files:

find /root /home -type f -name authorized_keys -printf '%TY-%Tm-%TdT%TH:%TM:%TS %p
'

Backup, deduplicate, and append new keys after verification. Use ssh-keygen -lf to display fingerprints.

Check for persistence mechanisms such as user‑level crontabs, systemd user units, /etc/rc.local, dynamic linker preload files, and cloud‑init scripts. Verify package integrity:

# Debian/Ubuntu
sudo dpkg -V
# RHEL/CentOS
sudo rpm -Va

6. Temporary Mitigation While the Attack Is Ongoing

Prefer restricting SSH access at the cloud security‑group or perimeter firewall to allow only bastion hosts, VPN, or approved management subnets. This is more effective than blocking individual attacker IPs.

Option A – Cloud Security‑Group Whitelist

List all legitimate source IPs, confirm console access, and keep the current session. Add a temporary rule that permits only the bastion/VPN subnet, verify connectivity from a second terminal, then remove the public‑Internet rule.

Option B – firewalld Rich Rule (RHEL‑based)

sudo firewall-cmd --state
sudo firewall-cmd --get-active-zones
sudo firewall-cmd --list-all
sudo firewall-cmd --list-all --permanent
# Add a temporary rich rule (replace ZONE and MANAGEMENT_SUBNET)
sudo firewall-cmd --zone=ZONE \
  --add-rich-rule='rule family="ipv4" source address="MANAGEMENT_SUBNET" service name="ssh" accept' \
  --timeout=30m
# After verification, make it permanent and reload
sudo firewall-cmd --remove-rich-rule='rule family="ipv4" source address="MANAGEMENT_SUBNET" service name="ssh" accept'
sudo firewall-cmd --reload

Never remove the rule before confirming that a second session can still connect.

7. SSH Configuration Hardening (Backup → Modify → Verify → Reload)

7.1 Locate the Effective Configuration

sudo sshd -t
sudo sshd -T | sort
sudo grep -RniE '^(Include|Port|PermitRootLogin|PasswordAuthentication|KbdInteractiveAuthentication|PubkeyAuthentication|AllowUsers|AllowGroups|Match)' /etc/ssh/sshd_config /etc/ssh/sshd_config.d 2>/dev/null

7.2 Backup

BACKUP_DIR="/root/sshd-backup-$(date -u +%Y%m%dT%H%M%SZ)"
sudo install -d -m 0700 "$BACKUP_DIR"
sudo cp -a /etc/ssh/sshd_config "$BACKUP_DIR/"
if [ -d /etc/ssh/sshd_config.d ]; then sudo cp -a /etc/ssh/sshd_config.d "$BACKUP_DIR/"; fi
sudo sshd -T | sudo tee "$BACKUP_DIR/sshd-T.before" >/dev/null
echo "Backup directory: $BACKUP_DIR"

7.3 Recommended Baseline (place in /etc/ssh/sshd_config.d/90-hardening.conf )

# /etc/ssh/sshd_config.d/90-hardening.conf
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
PermitEmptyPasswords no
MaxAuthTries 3
LoginGraceTime 30
X11Forwarding no
AllowTcpForwarding no

Key directive explanations: PasswordAuthentication no – disables password login; verify key login first. KbdInteractiveAuthentication no – disables keyboard‑interactive authentication. PermitRootLogin no – prevents direct root SSH login; ensure admins can sudo. MaxAuthTries 3 – limits authentication attempts per connection. LoginGraceTime 30 – limits unauthenticated connection time. AllowTcpForwarding no – disables SSH tunneling unless required.

7.4 Syntax Check and Apply

sudo sshd -t
sudo sshd -T | grep -Ei 'passwordauthentication|kbdinteractiveauthentication|pubkeyauthentication|permitrootlogin|maxauthtries|logingracetime|allowtcpforwarding'
sudo systemctl reload sshd 2>/dev/null || sudo systemctl reload ssh

Verify from a second terminal:

ssh -o PreferredAuthentications=publickey -o PasswordAuthentication=no USER@SERVER_IP
hostname -f
id
sudo -n true   # works only if passwordless sudo is configured

7.5 Rollback

If the new configuration breaks access, restore the original files and reload:

sudo cp -a "${BACKUP_DIR}/sshd_config" /etc/ssh/sshd_config
sudo rm -f /etc/ssh/sshd_config.d/90-hardening.conf
sudo sshd -t
sudo systemctl reload sshd 2>/dev/null || sudo systemctl reload ssh

8. Public‑Key Management

Generate modern keys on the client:

ssh-keygen -t ed25519 -a 100 -f ~/.ssh/KEY_NAME -C 'PURPOSE'

Set up the remote directory with correct permissions before copying the public key:

sudo install -d -m 0700 -o USER -g GROUP /home/USER/.ssh
sudo install -m 0600 -o USER -g GROUP PUBLIC_KEY_FILE /home/USER/.ssh/authorized_keys

If authorized_keys already exists, back it up, deduplicate, and append the new key after verification:

sudo cp -a /home/USER/.ssh/authorized_keys /home/USER/.ssh/authorized_keys.bak.$(date +%s)
sudo ssh-keygen -lf /home/USER/.ssh/authorized_keys
# Append the new line after verification

Optional per‑key restrictions (source IP, no‑forwarding, command=) can be added before the key data.

9. Fail2ban – Automatic Rate Limiting

Create a local override instead of editing the distro jail.conf:

# /etc/fail2ban/jail.d/sshd.local
[sshd]
enabled = true
port = ssh
backend = systemd
findtime = 10m
maxretry = 5
bantime = 1h
ignoreip = 127.0.0.1/8 ::1 MANAGEMENT_SUBNET

Enable and test:

sudo fail2ban-client -t
sudo systemctl enable --now fail2ban
sudo fail2ban-client status
sudo fail2ban-client status sshd

Unban a mistakenly blocked IP:

sudo fail2ban-client set sshd unbanip CLIENT_IP

10. Changing the SSH Port

Changing from 22 to a non‑standard port reduces automated scans but does not provide security. If required, follow a staged process:

Configure sshd_config to listen on both old and new ports temporarily.

Open the new port in cloud security groups and host firewalls.

Adjust SELinux or AppArmor policies if needed.

Validate syntax ( sshd -t) and reload.

From a second terminal, connect to the new port and verify full workflow.

Update monitoring, bastion hosts, automation scripts, and client configurations.

After a stable observation window, close the old port.

11. Authentication Rate‑Limiting and MFA

MaxAuthTries

limits attempts per connection; attackers can open many connections. Use MaxStartups (e.g., MaxStartups 10:30:100) to limit concurrent unauthenticated connections.

Implement MFA by combining public‑key authentication with PAM‑based keyboard‑interactive:

UsePAM yes
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication yes
AuthenticationMethods publickey,keyboard-interactive:pam

Verify the PAM stack in /etc/pam.d/sshd to ensure a true second factor (OTP, hardware token, etc.). Test both successful and failed MFA attempts on a non‑privileged account before rolling out to all users.

12. Host Keys and Cryptographic Algorithms

Record host key fingerprints for asset inventory:

sudo find /etc/ssh -maxdepth 1 -type f -name 'ssh_host_*_key.pub' -exec ssh-keygen -lf {} \;

When a client reports "REMOTE HOST IDENTIFICATION HAS CHANGED", verify whether the server was reinstalled, the host key rotated, or the IP was reassigned before updating known_hosts.

Check supported algorithms on the client:

ssh -V
ssh -Q key
ssh -Q cipher
ssh -Q mac

Check the server’s effective algorithms:

sudo sshd -T | grep -Ei 'hostkeyalgorithms|pubkeyacceptedalgorithms|ciphers|macs|kexalgorithms'

Only customize algorithms when compliance or a specific vulnerability requires it; otherwise rely on the distribution’s defaults.

13. Credential Rotation Across the Trust Chain

After a suspected compromise, rotate not only the user password but also all SSH keys, API tokens, cloud instance roles, Kubernetes kubeconfigs, database credentials, and any other secrets the host can reach. Perform rotation from a trusted, uncompromised workstation, not from the suspect host.

Local user and root passwords.

All authorized_keys entries for people and machines.

SSH agent forwarding credentials.

Git deploy keys, CI/CD tokens, artifact‑repo credentials.

Cloud instance roles and access keys.

Database, message‑queue, object‑storage, and third‑party API credentials.

TLS private keys and code‑signing keys.

Lock accounts with usermod --lock only after backing up /etc/passwd, /etc/shadow, and /etc/group. Remember that locking does not stop existing SSH sessions or key‑based logins.

14. Patching and Version Governance

Identify the installed OpenSSH and OpenSSL versions:

# Debian/Ubuntu
dpkg-query -W openssh-server openssl
apt-cache policy openssh-server openssl
# RHEL/CentOS
rpm -q openssh-server openssl
dnf updateinfo list --security

Security fixes are often back‑ported; rely on distro security advisories rather than upstream version numbers. Before upgrading, back up configuration, verify console access, test on a non‑critical host, and ensure the new packages do not require a full daemon restart that could interrupt management.

15. Confirmed Breach – Why Changing Passwords Alone Is Insufficient

If unauthorized access or persistence is detected, assume the entire trust chain is compromised. Preserve evidence, isolate the host, rotate all credentials, and rebuild from a trusted image rather than attempting in‑place cleanup.

16. Monitoring and Alerting

Centralize the following events:

Failed password / Invalid user rates.

Accepted password / Accepted publickey, especially for root or service accounts.

sudo successes and failures, account changes, authorized_keys modifications.

sshd configuration, service status, listening ports, firewall changes.

New systemd units, cron jobs, unexpected outbound connections.

Send logs to an external SIEM, set retention, and create alerts that combine failure rate, target account importance, time‑of‑day, and source‑IP anomalies rather than a static "5 failures" threshold.

17. Post‑Hardening Verification Checklist

sudo sshd -t
sudo sshd -T | grep -Ei 'port|passwordauthentication|kbdinteractiveauthentication|pubkeyauthentication|permitrootlogin|maxauthtries|logingracetime'
sudo ss -lntp | grep sshd
sudo journalctl -u sshd --since '10 minutes ago' --no-pager
ssh -vv -o PreferredAuthentications=publickey USER@SERVER_IP

Confirm that firewall rules, Fail2ban bans, and monitoring dashboards reflect the hardened state.

18. Common Pitfalls

Assuming many failure logs mean a breach – they only indicate attempts.

Assuming no success logs means no compromise – logs may be rotated or tampered.

Believing a non‑standard port provides security – it only reduces noise.

Disabling password authentication without confirming key‑based access – can lock out admins.

Turning off root SSH login without ensuring sudo works for admins.

Relying on Fail2ban instead of network‑level whitelisting.

Conclusion

The core of SSH brute‑force response is to first distinguish between noisy failed attempts and a real intrusion, then back every conclusion with immutable logs, session records, and configuration snapshots. When the host is still clean, enforce upstream source restrictions, mandatory public‑key authentication, minimal privileges, automated bans, and centralized monitoring. If unauthorized access is confirmed, treat it as a security incident: preserve evidence, rotate the entire trust chain, and rebuild from a trusted image. Throughout the process, keep the original session open, verify syntax before reload, and always have a rollback path.