Building a Complete Linux Enterprise Security System: From IDS to Incident Response

Linux Enterprise Network Security Protection System: From Intrusion Detection to Incident Response

Core Insight Preview: Building a complete Linux security system is not just stacking tools; it requires a closed-loop architecture from design, monitoring, response to continuous improvement. This article shares real-world experience in large enterprises.

Opening: A Real Security Incident Review

At 3 am, alerts flooded the monitoring system. A production web server’s CPU spiked to 95% and network traffic surged. Investigation revealed a DDoS attack combined with SSH brute‑force attempts.

Key takeaway: Single‑point protection is insufficient; enterprises need a comprehensive, multi‑layer security architecture.

Step 1: Build a Layered Defense Architecture

Network Perimeter Layer

# Core firewall rule example
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -m limit --limit 3/min -j ACCEPT
iptables -A INPUT -p tcp --dport 80,443 -j ACCEPT
iptables -A INPUT -j DROP

# Fail2ban key parameters
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
findtime = 600

Practical tip: Many ops engineers focus only on rule syntax and ignore performance. Place frequently matched rules early and use the -m recent module for connection tracking to avoid redundant matches.

Host Hardening Layer

# System hardening check script
#!/bin/bash
echo "=== Linux Security Hardening Check ==="

# Check privileged users
awk -F: '($3 == 0) {print $1}' /etc/passwd

# Check password policy
grep ^PASS /etc/login.defs

# Check SSH configuration security
grep -E "^(PermitRootLogin|PasswordAuthentication|PermitEmptyPasswords)" /etc/ssh/sshd_config

# Check suspicious cron jobs
crontab -l 2>/dev/null | grep -v "^#"

Step 2: Deploy Intrusion Detection System (IDS)

HIDS Deployment: OSSEC Practical Configuration

<!-- ossec.conf core configuration -->
<ossec_config>
  <global>
    <email_notification>yes</email_notification>
    <smtp_server>smtp.company.com</smtp_server>
    <email_from>[email protected]</email_from>
    <email_to>[email protected]</email_to>
  </global>

  <rules>
    <include>rules_config.xml</include>
    <include>pam_rules.xml</include>
    <!-- (additional rule includes omitted for brevity) -->
  </rules>

  <syscheck>
    <frequency>79200</frequency>
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin</directories>
    <directories>/var/www</directories>
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <!-- (other ignore entries omitted) -->
  </syscheck>

  <rootcheck>
    <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>
    <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
    <system_audit>/var/ossec/etc/shared/system_audit_rcl.txt</system_audit>
    <system_audit>/var/ossec/etc/shared/cis_debian_linux_rcl.txt</system_audit>
  </rootcheck>

  <global>
    <white_list>127.0.0.1</white_list>
    <white_list>^localhost.localdomain$</white_list>
    <white_list>10.0.0.0/8</white_list>
  </global>

  <remote>
    <connection>syslog</connection>
    <port>514</port>
    <protocol>udp</protocol>
    <allowed-ips>10.0.0.0/8</allowed-ips>
  </remote>

  <alerts>
    <log_alert_level>1</log_alert_level>
    <email_alert_level>7</email_alert_level>
  </alerts>
</ossec_config>

Network Traffic Analysis: Suricata Configuration

# suricata.yaml key settings
vars:
  address-groups:
    HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
    EXTERNAL_NET: "!$HOME_NET"
    HTTP_SERVERS: "$HOME_NET"
    SMTP_SERVERS: "$HOME_NET"
    SQL_SERVERS: "$HOME_NET"
    DNS_SERVERS: "$HOME_NET"
    TELNET_SERVERS: "$HOME_NET"
    AIM_SERVERS: "$EXTERNAL_NET"

default-rule-path: /etc/suricata/rules
rule-files:
  - suricata.rules
  - /etc/suricata/rules/local.rules

# High‑performance settings
af-packet:
  - interface: eth0
    threads: 4
    cluster-id: 99
    cluster-type: cluster_flow
    defrag: yes

Performance Tuning Highlights

CPU affinity: bind each worker thread to a distinct CPU core.

Memory tuning: increase ring‑buffer size as needed.

Rule optimization: regularly update rule sets and disable unnecessary rules.

Step 3: Build a Security Monitoring Center

ELK Stack Security Log Analysis

{
  "mappings": {
    "properties": {
      "@timestamp": {"type": "date"},
      "host": {"type": "keyword"},
      "source_ip": {"type": "ip"},
      "dest_ip": {"type": "ip"},
      "alert_level": {"type": "integer"},
      "rule_id": {"type": "keyword"},
      "description": {"type": "text"},
      "user": {"type": "keyword"},
      "command": {"type": "text"}
    }
  }
}

Key Security Indicator Monitoring

# Security monitoring script example
#!/bin/bash
# Monitor failed login attempts
failed_logins=$(grep "Failed password" /var/log/auth.log | wc -l)
if [ $failed_logins -gt 100 ]; then
  echo "WARNING: Excessive failed logins: $failed_logins"
fi

# Monitor new user creation
new_users=$(grep "new user" /var/log/auth.log | tail -10)
if [ -n "$new_users" ]; then
  echo "INFO: Detected new users"
  echo "$new_users"
fi

# Monitor privilege escalation
sudo_usage=$(grep "COMMAND" /var/log/auth.log | tail -10)
echo "Recent sudo activity:"
echo "$sudo_usage"

Step 4: Establish an Incident Response Process

Automated Response Script

#!/bin/bash
# Automated incident response script
INCIDENT_TYPE=$1
SOURCE_IP=$2
LOG_FILE="/var/log/security_incident.log"

log_incident() {
  echo "$(date): [$INCIDENT_TYPE] $1" >> $LOG_FILE
}

case $INCIDENT_TYPE in
  "brute_force")
    log_incident "Brute‑force attack detected, source IP: $SOURCE_IP"
    iptables -I INPUT -s $SOURCE_IP -j DROP
    echo "Brute‑force attack alert - IP: $SOURCE_IP" | mail -s "Security Alert" [email protected]
    ;;
  "malware")
    log_incident "Malware activity detected"
    systemctl stop network
    dd if=/dev/mem of=/tmp/memory_dump.img
    ;;
  "data_exfiltration")
    log_incident "Data exfiltration risk detected"
    iptables -P OUTPUT DROP
    netstat -tulnp > /tmp/network_connections.txt
    ;;
esac

Incident Analysis Playbook

Step 1: Rapid Assessment

Identify incident type and scope.

Evaluate business impact.

Decide whether to trigger the response workflow.

Step 2: Evidence Collection

# Evidence collection script
mkdir -p /tmp/incident_$(date +%Y%m%d_%H%M%S)
cd /tmp/incident_$(date +%Y%m%d_%H%M%S)

uname -a > system_info.txt
ps aux > process_list.txt
netstat -tulnp > network_connections.txt
ss -tulnp > socket_stats.txt

cp /var/log/messages .
cp /var/log/secure .
cp /var/log/auth.log .

find /etc -type f -exec md5sum {} \; > etc_md5.txt

Step 3: Threat Elimination

Isolate affected systems.

Remove malicious code.

Patch vulnerabilities.

Step 4: System Recovery

Validate system security.

Restore business services.

Enhance monitoring.

Step 5: Continuous Improvement and Optimization

Security Baseline Check

#!/usr/bin/env python3
import os, subprocess, json

def check_security_baseline():
    results = {}

    # SSH configuration
    ssh_config = {}
    with open('/etc/ssh/sshd_config', 'r') as f:
        for line in f:
            if line.strip() and not line.startswith('#'):
                key, value = line.split(None, 1)
                ssh_config[key] = value.strip()
    results['ssh_root_login'] = ssh_config.get('PermitRootLogin', 'yes') == 'no'
    results['ssh_password_auth'] = ssh_config.get('PasswordAuthentication', 'yes') == 'no'

    # Firewall status
    firewall_status = subprocess.run(['systemctl', 'is-active', 'iptables'], capture_output=True, text=True)
    results['firewall_active'] = firewall_status.stdout.strip() == 'active'

    # Update status
    updates = subprocess.run(['yum', 'check-update'], capture_output=True, text=True)
    results['system_updated'] = updates.returncode == 0

    return results

if __name__ == "__main__":
    baseline = check_security_baseline()
    print(json.dumps(baseline, indent=2))

Threat Intelligence Integration

# Threat intelligence update script
#!/bin/bash
# Update IP blacklist
wget -q https://reputation.alienvault.com/reputation.data -O /tmp/reputation.data
grep "Malicious Host" /tmp/reputation.data | cut -d'#' -f1 > /etc/security/malicious_ips.txt

# Update domain blacklist
curl -s https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts |
  grep "0.0.0.0" | awk '{print $2}' > /etc/security/malicious_domains.txt

# Apply to firewall
while read ip; do
  iptables -I INPUT -s $ip -j DROP
done < /etc/security/malicious_ips.txt

Monitoring Effect Evaluation

Key Security Indicators (KSI)

Detection Metrics

Mean Time to Detect (MTTD) < 15 min

False‑positive rate < 5 %

Detection coverage > 95 %

Response Metrics

Mean Time to Respond (MTTR) < 30 min

Incident handling success > 98 %

Automation ratio > 80 %

Recovery Metrics

Mean Time to Recover < 2 h

Business continuity > 99.9 %

Security Maturity Assessment Model

def calculate_security_maturity():
    weights = {
        'detection': 0.25,
        'prevention': 0.25,
        'response': 0.25,
        'recovery': 0.25
    }
    scores = {
        'detection': assess_detection_capability(),
        'prevention': assess_prevention_capability(),
        'response': assess_response_capability(),
        'recovery': assess_recovery_capability()
    }
    maturity_score = sum(scores[k] * weights[k] for k in weights.keys())
    return maturity_score

Practical Experience Summary

Five Key Success Factors

Automation First : Manual steps are error‑prone and inefficient.

Layered Defense : Single‑point protection inevitably fails.

Continuous Monitoring : Security is a dynamic process, not a static state.

Rapid Response : Time equals money; delays cause loss.

Regular Drills : Theory must be validated by practice.

Common Pitfalls to Avoid

❌ Misconception 1 : Deploying security tools guarantees safety. ✅ Correct : Tools are only means; proper configuration and operation are critical.

❌ Misconception 2 : Over‑reliance on commercial products. ✅ Correct : A hybrid of open‑source and commercial solutions offers flexibility.

❌ Misconception 3 : Security opposes business. ✅ Correct : Security should empower business rather than hinder it.

Future Development Trends

AI‑Driven Security Analytics : Machine‑learning‑based anomaly detection.

Zero‑Trust Architecture : Never trust any network traffic.

Cloud‑Native Security : Protect containers and micro‑services.

Security Left‑Shift : Embed security early in the development lifecycle.

Recommended Learning Resources

Books: “Linux Security Essentials”, “Network Security Attack & Defense”.

Certifications: CISSP, CEH, OSCP.

Communities: FreeBuf, 安全客, Seebug.

Tools: Kali Linux, Metasploit, Nmap.

Conclusion : Building a complete Linux security protection system is a systematic engineering effort that requires the integration of technology, processes, and personnel. The practical insights shared here aim to help readers avoid common pitfalls and quickly establish an effective security posture.