Detecting and Recovering a Compromised Linux Server: A Practical Checklist

With the rise of open‑source products, Linux operations engineers must be able to clearly identify whether a machine has been compromised. Based on real‑world experience, the article compiles common intrusion scenarios and the corresponding verification commands, primarily demonstrated on CentOS 6.9 (applicable to other distributions).

Check for missing or cleared logs. Verify whether system logs still exist or have been emptied.

Inspect user account files. Look for newly created or altered /etc/passwd and /etc/shadow files.

Review recent successful and failed login events. Examine /var/log/lastlog for the latest login attempts.

List all currently logged‑in users. Check the /var/run/utmp file.

Show all users that have ever logged in since system creation. Review the /var/log/wtmp file.

Analyze user connection durations. Use the same /var/log/wtmp data to see how long each user was connected.

Detect abnormal network traffic. Capture packets with tcpdump or measure throughput using iperf.

Examine the security log. Look into /var/log/secure for suspicious entries.

Identify malicious processes and their scripts.

Use top to find the PID of an abnormal process.

Search the virtual file system ( /proc/<PID>/fd) for the file descriptor that still holds the deleted file.

Recover deleted critical files. When a process keeps a file open, the file remains on disk even after deletion. The file can be accessed via its file descriptor in /proc . For example, if /var/log/secure is removed, locate the process (e.g., rsyslogd with PID 1264) that still holds the file descriptor, then read /proc/1264/fd/4 and redirect the output to a new file to restore the log.

The article includes screenshots illustrating each command and its output, helping readers follow the procedures step by step.