How to Build an Impenetrable Linux Server: A Step‑by‑Step Security Hardening Guide

Introduction

As a seasoned operations engineer, I have witnessed many servers compromised due to weak security configurations. This article shares practical hardening techniques gathered from real incidents to help you build a robust Linux server defense.

Real‑World Attack Case

One night a monitoring alert reported abnormal CPU usage on a web server. Investigation revealed a malicious process mining cryptocurrency, confirming a successful intrusion.

Attack Path Recap

SSH brute‑force attack gained root access.

Backdoor installed and persistent connection established.

Mining program consumed server resources.

Attacker attempted lateral movement within the internal network.

Core Protection Strategy – Multi‑Layer Defense

Layer 1: SSH Hardening

1. Change the default port

# Edit SSH configuration
vim /etc/ssh/sshd_config
# Use a non‑standard port (e.g., 22022)
Port 22022
systemctl restart sshd

2. Disable direct root login

# In sshd_config
PermitRootLogin no
# Create a regular admin user and add to sudo group
useradd -m -s /bin/bash admin
usermod -aG sudo admin

3. Enforce key‑based authentication

# Generate a key pair
ssh-keygen -t ed25519 -C "[email protected]"
# Set up authorized_keys on the server
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "your_public_key" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
# Disable password authentication
PasswordAuthentication no
systemctl restart sshd

Layer 2: Fail2Ban – Brute‑Force Protection

Install and configure Fail2Ban to block repeated failed SSH attempts.

# Ubuntu/Debian
apt update && apt install fail2ban -y
# CentOS/RHEL
yum install epel-release -y && yum install fail2ban -y
# Create local jail configuration
cat > /etc/fail2ban/jail.local <<'EOF'
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 3
ignoreip = 127.0.0.1/8 192.168.0.0/16

[sshd]
enabled = true
port = 22022
filter = sshd
logpath = /var/log/auth.log
bantime = 86400
EOF
systemctl enable fail2ban && systemctl start fail2ban

Check status with fail2ban-client status sshd and unban IPs if needed.

Layer 3: Firewall Configuration

UFW (Uncomplicated Firewall)

# Enable UFW
ufw enable
# Default policies
ufw default deny incoming
ufw default allow outgoing
# Allow SSH on custom port
ufw allow 22022/tcp
# Allow HTTP/HTTPS
ufw allow 80/tcp
ufw allow 443/tcp
ufw status verbose

Advanced iptables rules

#!/bin/bash
# Flush existing rules
iptables -F
iptables -X
iptables -Z
# Default DROP policy
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
# Allow loopback
iptables -A INPUT -i lo -j ACCEPT
# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# SSH rate limiting on custom port
iptables -A INPUT -p tcp --dport 22022 -m state --state NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22022 -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP
iptables -A INPUT -p tcp --dport 22022 -j ACCEPT
# Web services
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
# Save rules
iptables-save > /etc/iptables/rules.v4

Layer 4: Intrusion Detection (OSSEC‑HIDS)

# Download and install OSSEC
wget https://github.com/ossec/ossec-hids/archive/3.6.0.tar.gz
tar -xzf 3.6.0.tar.gz && cd ossec-hids-3.6.0
./install.sh
# Edit configuration
vim /var/ossec/etc/ossec.conf

Layer 5: File Integrity Monitoring (AIDE)

# Install AIDE
apt install aide -y   # Debian/Ubuntu
yum install aide -y   # CentOS/RHEL
# Initialize database
aide --init
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
# Create daily check script
cat > /usr/local/bin/aide_check.sh <<'EOF'
#!/bin/bash
aide --check | tee /var/log/aide_check.log
if [ $? -ne 0 ]; then
  echo "File integrity issues detected, check log for details"
fi
EOF
chmod +x /usr/local/bin/aide_check.sh
# Schedule via cron
echo "0 2 * * * /usr/local/bin/aide_check.sh" | crontab -

Advanced Hardening Techniques

Port Knocking

# Install knockd
apt install knockd -y
# Configure knock sequence
cat > /etc/knockd.conf <<'EOF'
[options]
    UseSyslog

[openSSH]
    sequence = 7000,8000,9000
    seq_timeout = 5
    command = /sbin/iptables -A INPUT -s %IP% -p tcp --dport 22022 -j ACCEPT
    tcpflags = syn

[closeSSH]
    sequence = 9000,8000,7000
    seq_timeout = 5
    command = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22022 -j ACCEPT
    tcpflags = syn
EOF
systemctl enable knockd && systemctl start knockd

Honeypot Deployment (Cowrie)

# Install Cowrie SSH honeypot
pip3 install cowrie
# Configure Cowrie to listen on the standard port (22) while the real SSH runs on a non‑standard port.

Automated Log Analysis

# Simple Bash script to alert on excessive SSH failures
LOGFILE="/var/log/auth.log"
ALERT_EMAIL="[email protected]"
failed=$(grep "Failed password" $LOGFILE | grep "$(date '+%b %d')" | wc -l)
if [ $failed -gt 50 ]; then
  echo "Warning: $failed SSH login failures today" | mail -s "SSH Alert - $(hostname)" $ALERT_EMAIL
fi
# Detect new user creation
new_users=$(grep "new user" /var/log/auth.log | grep "$(date '+%b %d')")
if [ -n "$new_users" ]; then
  echo "New user(s) created: $new_users" | mail -s "User Creation Alert - $(hostname)" $ALERT_EMAIL
fi

Lightweight Monitoring (Netdata)

# Install Netdata
bash <(curl -Ss https://my-netdata.io/kickstart.sh)
# Enable email notifications in /etc/netdata/health_alarm_notify.conf
SEND_EMAIL="YES"
DEFAULT_RECIPIENT_EMAIL="[email protected]"

Custom Alert Scripts (Python)

#!/usr/bin/env python3
import psutil, smtplib
from email.mime.text import MIMEText
import time

def check_system_health():
    alerts = []
    cpu = psutil.cpu_percent(interval=1)
    if cpu > 80:
        alerts.append(f"CPU usage high: {cpu}%")
    mem = psutil.virtual_memory()
    if mem.percent > 85:
        alerts.append(f"Memory usage high: {mem.percent}%")
    for part in psutil.disk_partitions():
        usage = psutil.disk_usage(part.mountpoint)
        if usage.percent > 90:
            alerts.append(f"Disk {part.mountpoint} usage high: {usage.percent}%")
    return alerts

def send_alert(alerts):
    if not alerts:
        return
    msg = MIMEText('
'.join(alerts))
    msg['Subject'] = f"Server Health Alert - {time.strftime('%Y-%m-%d %H:%M')}"
    msg['From'] = '[email protected]'
    msg['To'] = '[email protected]'
    print('Sending alert:', '
'.join(alerts))
    # Add SMTP sending logic here

if __name__ == '__main__':
    alerts = check_system_health()
    send_alert(alerts)

Incident Response Playbook

Steps After Detecting an Intrusion

Immediate Isolation

# Drop all inbound traffic except from your admin IP
iptables -A INPUT -j DROP
iptables -I INPUT 1 -s YOUR_IP -j ACCEPT

Preserve Evidence

# Archive critical logs
tar -czf evidence_$(date +%Y%m%d_%H%M).tar.gz \
    /var/log/auth.log \
    /var/log/syslog \
    /var/log/messages

Remove Backdoors

# Inspect cron jobs and startup services
crontab -l
cat /etc/crontab
ls -la /etc/cron.*
systemctl list-unit-files --state=enabled
ls -la /etc/init.d/

Summary and Recommendations

By applying the layered defenses described above, you can significantly raise the security posture of any Linux server:

🔴 Basic : Change SSH port and enforce key authentication.

🟡 Standard : Add Fail2Ban and strict firewall rules.

🟢 Advanced : Deploy IDS (OSSEC), file integrity monitoring (AIDE), and automated health checks.

🔵 Expert : Implement port knocking, honeypots, and automated response scripts.

Best‑Practice Checklist

Regularly update the operating system and all packages.

Apply the principle of least privilege for users and services.

Perform periodic audits and log analysis.

Maintain a documented incident‑response plan.

Conduct regular security drills and penetration tests.