How to Detect and Recover from Linux Server Intrusions: 11 Essential Checks

As open‑source products become more prevalent, Linux operations engineers must be able to clearly determine whether a machine has been compromised. The following eleven common intrusion scenarios and corresponding verification commands, observed on CentOS (applicable to other distributions), are presented for reference.

1. Intruder may delete machine log files

Check whether log files still exist or have been cleared. Example commands are shown in the accompanying image.

2. Intruder may create a new file storing usernames and passwords

Inspect /etc/passwd and /etc/shadow files. Example commands are illustrated in the image.

3. Intruder may modify username and password files

Review the contents of /etc/passwd and /etc/shadow for anomalies. Example commands are provided in the image.

4. View recent successful login events and the last unsuccessful login attempt

Examine the "/var/log/lastlog" file. Example commands are displayed in the image.

5. View all currently logged‑in users

Check the "/var/run/utmp" log file. Example commands are shown in the image.

6. View users who have logged in since the machine was created

Inspect the "/var/log/wtmp" file. Example commands are shown in the image.

7. View each user's total connection time (hours)

Again, refer to the "/var/log/wtmp" file. Example commands are illustrated in the image.

8. If abnormal traffic is observed

Use tcpdump to capture network packets or the iperf tool to analyze traffic.

9. Examine the "/var/log/secure" log file

Search for attacker information. Example commands are displayed in the image.

10. Query the script file associated with an abnormal process

a. Use the top command to find the PID of the suspicious process. b. Locate the executable file in the virtual filesystem ( /proc).

11. Recover deleted important files after a confirmed intrusion

When a process still holds an open file descriptor, the file remains on disk even after deletion. The lsof utility can reveal these descriptors via the /proc filesystem. Steps:

Identify that /var/log/secure is missing.

Use lsof to check if any process (e.g., PID 1264 – rsyslogd) still has the file open (fd 4).

Access the descriptor through /proc/1264/fd/4 to read the file contents.

Redirect the output to a new file using I/O redirection.

Verify that /var/log/secure is restored.

These recovery techniques are especially useful for log files and databases.