Build an Impenetrable Linux Server: Step‑by‑Step Security Hardening Guide

Real Incident: A Terrifying Intrusion

Late one night a monitoring alert showed abnormal CPU usage on a web server; investigation revealed a suspicious process and a mining trojan, highlighting the need for strong server protection.

Attack Path Recap

SSH brute‑force attack gained root access

Backdoor installed with persistent connection

Mining program consumed resources

Attempted lateral movement within the internal network

Core Defense Strategies: Multi‑Layer Protection

Layer 1 – SSH Hardening

1. Change the default port

# Edit SSH config file
vim /etc/ssh/sshd_config

# Change port (recommended 10000‑65535)
Port 22022

# Restart SSH service
systemctl restart sshd

2. Disable direct root login

# In sshd_config
PermitRootLogin no

# Create a regular user and add to sudo group
useradd -m -s /bin/bash admin
usermod -aG sudo admin

3. Configure key‑based authentication

# Generate SSH key pair
ssh-keygen -t ed25519 -C "[email protected]"

# Create authorized_keys on the server
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "your_public_key" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

# Disable password authentication
echo "PasswordAuthentication no" >> /etc/ssh/sshd_config
systemctl restart sshd

Layer 2 – Fail2Ban Brute‑Force Protection

Install and configure Fail2Ban

# Ubuntu/Debian
apt update && apt install fail2ban -y

# CentOS/RHEL
yum install epel-release -y && yum install fail2ban -y

Create custom SSH jail

cat > /etc/fail2ban/jail.local <<'EOF'
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 3
ignoreip = 127.0.0.1/8 192.168.0.0/16

[sshd]
enabled = true
port = 22022
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 86400
EOF

systemctl enable fail2ban && systemctl start fail2ban

Check ban status

# View banned IPs
fail2ban-client status sshd

# Manually unban an IP
fail2ban-client set sshd unbanip 192.168.1.100

Layer 3 – Firewall Configuration

UFW simple firewall

# Enable UFW
ufw enable

# Default policies
ufw default deny incoming
ufw default allow outgoing

# Allow SSH on custom port
ufw allow 22022/tcp

# Allow web services
ufw allow 80/tcp
ufw allow 443/tcp

# Show rules
ufw status verbose

Advanced iptables rules

#!/bin/bash
# Flush existing rules
iptables -F
iptables -X
iptables -Z

# Default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# SSH rate‑limit (port 22022)
iptables -A INPUT -p tcp --dport 22022 -m state --state NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22022 -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP
iptables -A INPUT -p tcp --dport 22022 -j ACCEPT

# Web services
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Save rules
iptables-save > /etc/iptables/rules.v4

Layer 4 – Intrusion Detection System

Deploy OSSEC‑HIDS

# Download and install OSSEC
wget https://github.com/ossec/ossec-hids/archive/3.6.0.tar.gz
tar -xzf 3.6.0.tar.gz && cd ossec-hids-3.6.0
./install.sh

# Edit configuration
vim /var/ossec/etc/ossec.conf

Custom monitoring script

#!/bin/bash
LOG_FILE="/var/log/security_check.log"

check_suspicious_processes() {
  echo "[$(date)] Checking suspicious processes..." >> $LOG_FILE
  ps aux --sort=-%cpu | head -n 10 | while read line; do
    cpu=$(echo $line | awk '{print $3}')
    if (( $(echo "$cpu > 80" | bc -l) )); then
      echo "Warning: High CPU process: $line" >> $LOG_FILE
    fi
  done
}

check_network_connections() {
  echo "[$(date)] Checking network connections..." >> $LOG_FILE
  netstat -tlnp | grep -E ':(1234|4444|5555|8080)' && echo "Warning: Suspicious listening port" >> $LOG_FILE
}

check_suspicious_processes
check_network_connections

Layer 5 – File Integrity Monitoring

Use AIDE

# Install AIDE
apt install aide -y   # Debian/Ubuntu
yum install aide -y   # CentOS/RHEL

# Initialize database
aide --init
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# Create check script
cat > /usr/local/bin/aide_check.sh <<'EOF'
#!/bin/bash
aide --check | tee /var/log/aide_check.log
if [ $? -ne 0 ]; then
  echo "File integrity issues detected, see log" 
fi
EOF
chmod +x /usr/local/bin/aide_check.sh

# Add cron job (daily at 02:00)
echo "0 2 * * * /usr/local/bin/aide_check.sh" | crontab -

Advanced Protection Techniques

1. Port Knocking

# Install knockd
apt install knockd -y

# Configure knockd
cat > /etc/knockd.conf <<'EOF'
[options]
  UseSyslog

[openSSH]
  sequence = 7000,8000,9000
  seq_timeout = 5
  command = /sbin/iptables -A INPUT -s %IP% -p tcp --dport 22022 -j ACCEPT
  tcpflags = syn

[closeSSH]
  sequence = 9000,8000,7000
  seq_timeout = 5
  command = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22022 -j ACCEPT
  tcpflags = syn
EOF

# Enable and start service
systemctl enable knockd && systemctl start knockd

2. Honeypot Deployment

# Install Cowrie SSH honeypot
pip3 install cowrie
# Configure Cowrie to listen on port 22 while the real SSH service runs on a non‑standard port

3. Automated Log Analysis

#!/bin/bash
LOGFILE="/var/log/auth.log"
ALERT_EMAIL="[email protected]"

# Detect SSH login failures today
failed_attempts=$(grep "Failed password" $LOGFILE | grep "$(date '+%b %d')" | wc -l)
if [ $failed_attempts -gt 50 ]; then
  echo "Warning: $failed_attempts SSH login failures today" | mail -s "SSH Security Alert - $(hostname)" $ALERT_EMAIL
fi

# Detect new user creation today
new_users=$(grep "new user" /var/log/auth.log | grep "$(date '+%b %d')")
if [ -n "$new_users" ]; then
  echo "Warning: New user accounts detected: $new_users" | mail -s "User Management Alert - $(hostname)" $ALERT_EMAIL
fi

Monitoring and Alerting System

Lightweight Monitoring with Netdata

# Install Netdata
bash <(curl -Ss https://my-netdata.io/kickstart.sh)

# Configure email alerts
vim /etc/netdata/health_alarm_notify.conf
SEND_EMAIL="YES"
DEFAULT_RECIPIENT_EMAIL="[email protected]"

Custom Alert Script (Python)

#!/usr/bin/env python3
import psutil, smtplib, time
from email.mime.text import MIMEText

def check_system_health():
    alerts = []
    cpu = psutil.cpu_percent(interval=1)
    if cpu > 80:
        alerts.append(f"CPU usage high: {cpu}%")
    mem = psutil.virtual_memory()
    if mem.percent > 85:
        alerts.append(f"Memory usage high: {mem.percent}%")
    for part in psutil.disk_partitions():
        usage = psutil.disk_usage(part.mountpoint)
        if usage.percent > 90:
            alerts.append(f"Disk {part.mountpoint} usage high: {usage.percent}%")
    return alerts

def send_alert(alerts):
    if not alerts:
        return
    msg = MIMEText('
'.join(alerts))
    msg['Subject'] = f"Server Health Alert - {time.strftime('%Y-%m-%d %H:%M')}"
    msg['From'] = '[email protected]'
    msg['To'] = '[email protected]'
    print('Sending alert:', '
'.join(alerts))
    # Add actual email sending logic here

if __name__ == "__main__":
    alerts = check_system_health()
    send_alert(alerts)

Emergency Response Plan

Steps After Detecting an Intrusion

Immediate Isolation

# Drop all inbound traffic but keep current SSH session
iptables -A INPUT -j DROP
iptables -I INPUT 1 -s YOUR_IP -j ACCEPT

Preserve Evidence

# Archive critical logs
tar -czf evidence_$(date +%Y%m%d_%H%M).tar.gz \
    /var/log/auth.log \
    /var/log/syslog \
    /var/log/messages

Remove Backdoors

# Review cron jobs
crontab -l
cat /etc/crontab
ls -la /etc/cron.*

# Review enabled services
systemctl list-unit-files --state=enabled
ls -la /etc/init.d/

Conclusion and Recommendations

Implementing the layered measures described above dramatically improves Linux server security. Assess the protection level:

🔴 Basic – Change SSH port and enable key authentication

🟡 Standard – Add Fail2Ban and firewall rules

🟢 Advanced – Deploy IDS and file integrity monitoring

🔵 Expert – Use honeypots and automated response scripts

Best practices:

Regularly update the operating system and packages

Apply the principle of least privilege

Conduct periodic audits and log analysis

Maintain a documented incident response plan

Run security drills regularly