How to Detect and Recover From Linux Server Intrusions: Essential Checks

As open‑source products become more prevalent, Linux operations engineers need clear methods to determine whether a host has been compromised. The following checks were performed on a CentOS 6.9 system and are applicable to other Linux distributions.

1. Verify log integrity

Attackers may delete log files. Check whether logs still exist or have been cleared, using appropriate commands.

2. Inspect /etc/passwd and /etc/shadow

Attackers might create new user accounts. Review these files for unexpected entries.

3. Detect modifications to user files

Check the contents of /etc/passwd and /etc/shadow for unauthorized changes.

4. Review recent successful and failed logins

Examine /var/log/lastlog to see the last successful login and recent failed attempts.

5. List currently logged‑in users

Inspect the /var/run/utmp file for all active sessions.

6. Examine historical login records

Check /var/log/wtmp for users who have logged in since the system was created.

7. Analyze user connection durations

Use /var/log/wtmp to see how long each user has been connected (in hours).

8. Detect abnormal network traffic

If unusual traffic is observed, capture packets with tcpdump or measure bandwidth with iperf.

9. Search the security log

Review /var/log/secure for clues about the attacker’s activity.

10. Identify suspicious processes and their scripts

Use top to find the PID of an abnormal process, then locate its executable in the virtual file system (e.g., /proc/<PID>/fd/).

11. Recover deleted critical files

If important files such as logs have been removed, they may still be open by a running process. Use lsof to find processes holding the deleted file descriptor, then read the file via /proc/<PID>/fd/<FD> and redirect the output to a new file to restore it.

Note: When a process keeps a file open after deletion, the file remains on disk and can be recovered through its file descriptor, even though it is no longer visible in the directory tree.