Investigating and Securing a Server After a Suspicious Login

Problem Background

Abnormal server login is a critical security incident. Indicators such as unknown users, unexpected root logins, high CPU usage by unfamiliar processes, and prolonged uptime suggest compromise.

Event Review

Monitoring reported CPU usage >90% at 09:00 on a Monday. Investigation showed: uptime – server up for >200 days. who – unknown user logged in. last – multiple recent root logins not performed by ops. ps aux – high‑CPU process named .cache.

Step 1 – Confirm Compromise

1.1 Check logged‑in users

# who
# Example output:
# root     pts/0    2026-04-29 09:10 (192.168.1.100)
# unknown  pts/1    2026-04-29 08:55 (23.45.67.89)
# If the IP on pts/1 is not a known ops IP, the server is likely compromised.

# Show recent logins
last
# Show only root logins
last root

1.2 Check boot time

# who -b   # system boot time
# uptime   # running time
# A very long uptime without recent patches may indicate missing updates; a recent boot may be an attacker‑initiated reboot to hide traces.

1.3 Identify CPU‑intensive unknown processes

# ps aux --sort=-%cpu | head -20
# Look for process names not belonging to standard services (sshd, systemd, docker, nginx, mysqld, redis-server, java, python, php-fpm, prometheus).
# Names such as .cache, xmrig, kworkerds, kinsing strongly indicate a mining payload.

1.4 Check abnormal network connections

# ss -tunapl               # all sockets
# ss -tunapl | grep ESTABLISHED   # external connections (miners usually connect to pool servers)
# ss -tunapl | grep LISTEN        # listening ports; ports like 3333 or 4444 may be backdoors.

Step 2 – Immediate Damage Control (Stop the Attacker)

Preserve critical data (logs, memory dumps) before taking disruptive actions.

2.1 Disconnect suspicious sessions

# who -a   # list sessions with PID
# Example: disconnect pts/1
sudo pkill -kill -t pts/1
# Disconnect all non‑console sessions
who | grep -v "console" | awk '{print $2}' | grep -v pts/0 | xargs -I {} sudo pkill -kill -t {}

2.2 Disable suspicious accounts

# cat /etc/passwd | grep -v /sbin/nologin | grep -v /bin/false
# Lock a suspicious account (replace <suspicious-username> with the actual name)
sudo passwd -l <suspicious-username>

2.3 Rotate passwords

# Change root password
sudo passwd root
# Change other admin passwords
sudo passwd admin
# Regenerate SSH key pair if key authentication is used
ssh-keygen -t ed25519
# Upload the new public key to authorized_keys via a secure channel.

2.4 Temporarily stop SSH access (if the attacker cannot be located quickly)

# Stop the SSH service (existing sessions remain)
sudo systemctl stop sshd
# Or restrict SSH to a trusted IP (replace <your-office-ip> with the management IP)
# CentOS
sudo firewall-cmd --permanent --add-source=<your-office-ip>/32 --zone=trusted
sudo firewall-cmd --reload
# Ubuntu
sudo ufw allow from <your-office-ip> to any port 22
# If SSH runs on a non‑standard port (replace <ssh-port>)
sudo firewall-cmd --permanent --add-source=<your-office-ip>/32
sudo firewall-cmd --permanent --add-port=<ssh-port>/tcp
sudo firewall-cmd --reload

Step 3 – Trace the Attack Path

3.1 Review SSH brute‑force logs

# CentOS/RHEL: /var/log/secure
# Ubuntu/Debian: /var/log/auth.log
# Failed password attempts (evidence of brute‑force)
sudo tail -500 /var/log/secure | grep -i "failed password"
sudo tail -500 /var/log/auth.log | grep -i "failed password"
# Count failures per IP
sudo tail -2000 /var/log/secure | grep "Failed password" | awk '{print $11}' | sort | uniq -c | sort -rn | head -20
# Look for successful logins after failures
sudo tail -2000 /var/log/secure | grep -E "Accepted|password changed"

3.2 Review successful login records

# Show recent logins (both distros)
sudo last | head -30
# Filter SSH logins
sudo last | grep sshd | head -30
# Show root logins only
sudo last root

3.3 Identify attacker IP addresses

# Extract IPs from accepted logins (replace log path as appropriate)
sudo grep "Accepted password for" /var/log/secure | awk '{print $9" "$11}' | sort | uniq
# For public‑key logins
sudo grep "Accepted publickey" /var/log/secure | awk '{print $9" "$11}' | sort | uniq

3.4 Build an attack timeline

# Show all actions from a specific IP (example 192.168.1.200)
sudo grep "192.168.1.200" /var/log/secure | tail -50
# Check if log files have been truncated (possible clearing)
wc -l /var/log/secure

Step 4 – Locate Backdoors and Malicious Programs

4.1 Find hidden processes via /proc

for pid in $(ls -1 /proc | grep '^[0-9]'); do
  if [ -f /proc/$pid/cmdline ]; then
    cmdline=$(cat /proc/$pid/cmdline 2>/dev/null | tr '\0' ' ')
    if [ -n "$cmdline" ]; then
      echo "PID=$pid CMD=$cmdline"
    fi
  fi
done

4.2 Detect malicious process names

# Common mining names
ps aux | grep -iE "xmrig|miner|kworkerds|kinsing|\.cache|c3pool|cryptonight|hashrate"
# Find processes without an executable path (possible hiding)
ps aux | awk '{print $11}' | grep '^$' | xargs -I {} ps aux | grep -v "PID"
# Find processes with empty name
ps aux | awk '{if($11=="") print}'

4.3 Examine cron jobs for persistence

# Root cron files
sudo cat /var/spool/cron/root
sudo cat /var/spool/cron/crontabs/root
# System‑wide cron directories
ls -la /etc/cron.d/
cat /etc/cron.d/*
cat /etc/crontab
ls -la /etc/cron.*
# Look for suspicious commands (wget, curl)
grep -r "wget\|curl" /etc/cron* 2>/dev/null

4.4 Inspect SSH authorized_keys for backdoors

# Show root authorized_keys
sudo cat /root/.ssh/authorized_keys
# Remove unknown keys
# Also check other users
sudo find /home -name "authorized_keys" -exec ls -la {} \;

4.5 Verify system binaries and libraries

# RPM‑based systems
sudo rpm -Va
# Look for flags S,M,5,D,L,U,G,T indicating modifications.
# Debian/Ubuntu
sudo dpkg -S /bin/bash
sudo apt-get install -y debsums
sudo debsums -a 2>/dev/null | grep -v "OK$" | head -20

4.6 Re‑check network connections

# Established external connections (exclude private ranges)
ss -tunapl | grep ESTABLISHED | grep -v "10\\.|172\\.(1[6-9]|2[0-9]|3[0-1])\\.|192\\.168\\."
# Listening ports
ss -tunapl | grep LISTEN
# Process‑socket mapping
lsof -i -n -P | head -30

Step 5 – Assess Impact Scope

5.1 Review commands executed by the attacker

# Root bash history
sudo cat /root/.bash_history
# Search for typical malicious actions
grep -iE "wget|curl|chmod|mysql|ssh" /root/.bash_history | tail -30

5.2 Examine filesystem changes

# Modified system binaries in the last 7 days
sudo find /bin /sbin /usr/bin /usr/sbin -type f -mtime -7 -ls 2>/dev/null
# Executable files created in /tmp or /var/tmp
sudo find /tmp /var/tmp -type f -executable -mtime -7 -ls 2>/dev/null
# Newly created user accounts (UID/GID >=1000)
sudo awk -F: '{print $1" "$3" "$4}' /etc/passwd | while read user uid gid; do
  if [ $uid -ge 1000 ] && [ $gid -ge 1000 ]; then
    echo "User: $user UID: $uid GID: $gid"
  fi
done

5.3 Verify log integrity

# List log files by size to spot anomalies
ls -lh /var/log/* | sort -k5 -rh | head -20
# Check for empty or unusually small logs
sudo ls -la /var/log/secure
sudo ls -la /var/log/messages
sudo ls -la /var/log/auth.log

Step 6 – Remediation and Hardening

6.1 Kill malicious processes

# Find malicious PIDs
ps aux | grep -iE "xmrig|miner|kworkerds|kinsing|\.cache"
# Force kill
sudo kill -9 <pid>
# Verify if the process respawns
sleep 5
ps aux | grep <process-name>

6.2 Remove malicious cron tasks

# List root crontab
sudo crontab -l -u root
# Edit to delete suspicious lines
sudo vi /var/spool/cron/root
# Remove files in /etc/cron.d that are malicious
sudo rm -f /etc/cron.d/suspicious-file
# Verify remaining cron configuration
sudo cat /etc/crontab
sudo ls -la /etc/cron.d/

6.3 Clean SSH authorized_keys backdoors

# Backup then edit
sudo cp /root/.ssh/authorized_keys /root/.ssh/authorized_keys.bak.$(date +%Y%m%d)
sudo vi /root/.ssh/authorized_keys
# Delete unknown keys
# Check other users' .ssh directories
sudo find /home -name "authorized_keys" -exec ls -la {} \;

6.4 Delete suspicious files

# Search for recent scripts or binaries in temporary directories
sudo find /tmp /var/tmp -type f -mtime -7 \( -name "*.sh" -o -name "*.py" -o -name "*xmrig*" -o -name "*.conf" \) | head -20
# Search for recently modified binaries
sudo find /bin /sbin /usr/bin /usr/sbin -type f -mtime -3 -ls 2>/dev/null
# Backup then delete identified files
sudo cp /tmp/suspicious_file /tmp/suspicious_file.bak.$(date +%Y%m%d)
sudo rm /tmp/suspicious_file

6.5 Restore modified system files

# Reinstall altered RPM packages (example for /bin/ls)
sudo rpm -qf /bin/ls
sudo rpm --force -ivh --nodeps <package-name>
# Ubuntu/Debian – reconfigure damaged packages
sudo dpkg --configure -a

6.6 Reconfigure the firewall

# Reload firewall rules (CentOS)
sudo firewall-cmd --complete-reload
sudo firewall-cmd --list-all
# Allow only trusted IP for SSH (replace <your-office-ip>)
sudo firewall-cmd --permanent --add-source=<your-office-ip>/32
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --reload
# Verify open ports
sudo firewall-cmd --list-ports

Step 7 – Post‑Incident Report

1. Event Overview
- Detection: 2026-04-29 09:00
- Compromise confirmed: 2026-04-29 09:30
- Affected assets: 1 production server

2. Attack Path Analysis
- Entry vector: SSH brute‑force password attack
- Attacker IP: 23.45.67.89 (foreign data center)
- Brute‑force window: 2026-04-28 03:00‑03:15
- First successful root login: 2026-04-28 03:16:23

3. Attacker Actions
1. Added a public key to /root/.ssh/authorized_keys
2. Downloaded and executed mining binary <code>.cache</code>
3. Created a cron job to keep the miner alive
4. Modified sshd_config to allow RootLogin and PubkeyAuthentication

4. Impact Assessment
- Data exfiltration: none detected
- System integrity: some files modified (repaired)
- Business impact: high CPU caused slower responses (recovered)
- Lateral movement: none observed

5. Remediation Steps (checked)
- [x] Disconnected attacker sessions
- [x] Disabled suspicious accounts
- [x] Rotated passwords
- [x] Killed malicious processes
- [x] Removed malicious cron jobs
- [x] Cleaned SSH backdoors
- [x] Deleted suspicious files
- [x] Reinstalled altered system files
- [x] Re‑configured firewall
- [x] Updated fail2ban configuration

6. Root Cause
- Weak SSH password and lack of fail2ban allowed brute‑force success.
- Port 22 exposed to the Internet.

Step 8 – Preventive Measures

8.1 Stop SSH brute‑force

# Disable password authentication, enable public‑key authentication
# Edit /etc/ssh/sshd_config
PasswordAuthentication no
PubkeyAuthentication yes
# Restart SSH service
sudo systemctl restart sshd
# Configure fail2ban (refer to standard fail2ban rules)

8.2 Restrict SSH source IP

# /etc/hosts.allow example
sshd: <your-office-ip>/32 : allow
sshd: ALL : deny
# Or via firewall
sudo firewall-cmd --permanent --add-source=<your-office-ip>/32 --zone=trusted
sudo firewall-cmd --reload

8.3 Enable SSH login alerts

# Append to /root/.bashrc (or appropriate profile)
echo 'echo "Root login from $SSH_CLIENT at $(date)" | mail -s "SSH Root Login Alert" [email protected]' >> /root/.bashrc

8.4 Daily security audit script

#!/bin/bash
ALERT_EMAIL="[email protected]"
LOG="/var/log/security_audit.log"
{
  echo "===== Security Audit $(date) ====="
  # Account check
  current_users=$(cat /etc/passwd | grep -v /sbin/nologin | awk -F: '{print $1}')
  echo "[Account Check] Current users: $current_users"
  # SSH brute‑force attempts
  failed_count=$(grep "Failed password" /var/log/secure 2>/dev/null | wc -l)
  echo "[SSH Brute‑Force] Recent failed attempts: $failed_count"
  # Recent root logins
  recent_root=$(sudo last root | head -5)
  echo "$recent_root"
  # Suspicious processes
  suspicious=$(ps aux | grep -iE "xmrig|miner|kworkerds|kinsing|\.cache" | grep -v grep)
  if [ -n "$suspicious" ]; then
    echo "[Suspicious Processes] $suspicious" | mail -s "Suspicious Process Alert" $ALERT_EMAIL
  fi
  # External network connections
  external_conn=$(ss -tunapl | grep ESTABLISHED | grep -v "10\\.|172\\.|192\\.168\\." | wc -l)
  echo "[External Connections] Count: $external_conn"
} >> $LOG