How to Detect and Recover From Linux Server Intrusions: A Step‑by‑Step Guide

As open‑source products become increasingly popular, Linux operations engineers must be able to clearly identify whether a compromised machine has been intruded. The author shares several common intrusion scenarios observed on CentOS 6.9 and how to verify them.

1. Attackers may delete the machine's log files

Check whether log information still exists or has been cleared. Example commands:

2. Attackers may create a new file storing usernames and passwords

Inspect /etc/passwd and /etc/shadow files. Example commands:

3. Attackers may modify the username and password files

Examine the contents of /etc/passwd and /etc/shadow to detect changes. Example commands:

4. View recent successful login events and the last unsuccessful login event

Check the /var/log/lastlog file. Example commands:

5. View all users currently logged into the machine

Inspect the /var/run/utmp file. Example commands:

6. View users who have logged in since the machine was created

Check the /var/log/wtmp file. Example commands:

7. View each user's total connection time (hours)

Again examine the /var/log/wtmp file. Example commands:

8. If abnormal traffic is observed

Use tcpdump to capture packets or iperf to assess traffic.

9. Examine the /var/log/secure log file

Attempt to discover attacker information. Example commands:

10. Identify scripts associated with suspicious processes

Use top to find the PID of the suspicious process.

11. Recover deleted important files after confirming intrusion

When a process keeps a deleted file open, the file remains on disk and can be accessed via its file descriptor in /proc. Use lsof to locate open descriptors, then read the file through /proc/<pid>/fd/<fd> and redirect the output to a new file.

Example steps:

Verify that /var/log/secure is missing.

Use lsof to see if any process still has the file open.

Identify the PID (e.g., 1264) and file descriptor (e.g., 4) from the lsof output.

Read the file via /proc/1264/fd/4 and redirect it to a new file.

Confirm that /var/log/secure now exists.