Detecting and Recovering Linux Server Intrusions: Essential Checks and Commands

As open‑source products become increasingly popular, Linux operations engineers must be able to clearly determine whether a machine has been compromised. The following checklist, based on observations in CentOS 6.9 (applicable to other distributions), lists common signs of intrusion and the corresponding commands.

1. Check for missing or cleared logs. Verify whether log files still exist or have been emptied.

2. Look for suspicious passwd and shadow files. Examine /etc/passwd and /etc/shadow for unexpected entries.

3. Detect modifications to passwd or shadow. Compare current file contents with known good versions.

4. Review recent successful and failed login events. Check /var/log/lastlog for the last successful login and the most recent failed attempts.

5. List all currently logged‑in users. Inspect the /var/run/utmp file.

6. View the complete login history since system creation. Examine /var/log/wtmp.

7. Analyze user connection durations (hours). Also derived from /var/log/wtmp.

8. Detect abnormal network traffic. Use tcpdump to capture packets or iperf to measure throughput.

9. Inspect the security log. Review /var/log/secure for signs of intrusion.

10. Identify suspicious processes and their scripts.

a. Use top to find the PID of an abnormal process.

b. Search the virtual filesystem for the executable file of that PID.

11. Recover deleted important files (e.g., logs) when a process still holds them.

a. Verify that the target file (e.g., /var/log/secure) is missing.

b. Use lsof to see if any process still has the file open.

c. Locate the file descriptor in /proc/<PID>/fd/<fd> (e.g., /proc/1264/fd/4) to access the deleted content.

d. Redirect the file descriptor output to a new file using I/O redirection.

e. Verify that the recovered file now exists and contains the expected data.

These techniques help administrators quickly detect compromise indicators and, when possible, restore critical log files that attackers may have tried to erase.