How to Detect and Investigate Linux Server Intrusions: Essential Log Checks

As open‑source products become more prevalent, Linux operations engineers must be able to clearly determine whether a machine has been compromised. The following checklist, based on experience with CentOS 6.9 (applicable to other distributions), details common intrusion indicators and how to verify them.

1. Intruder may delete machine log information

Check whether log files still exist or have been cleared. Example commands are shown in the accompanying image.

2. Intruder may create a new file storing usernames and passwords

Inspect /etc/passwd and /etc/shadow for unexpected entries. Command examples are illustrated below.

3. Intruder may modify username and password files

Review the contents of /etc/passwd and /etc/shadow to detect tampering. Example commands are provided.

4. Check recent successful login events and the last unsuccessful login

Examine the /var/log/lastlog file. Example commands are shown.

5. View all currently logged‑in users

Inspect the /var/run/utmp file. Example commands are illustrated.

6. View users who have logged in since the machine was created

Check the /var/log/wtmp file. Example commands are provided.

7. View connection time (hours) for all users

Also based on /var/log/wtmp. Example output is shown.

8. If abnormal traffic is detected on the machine

Use tcpdump to capture packets or tools like iperf to analyze traffic.

9. Can view /var/log/secure log file

Examine this log to discover attacker information. Example commands are illustrated.

10. Query the script file corresponding to an abnormal process

Use top to find the PID of the suspicious process, then locate its executable in the virtual filesystem.

11. If confirmed the machine is compromised and important files have been deleted, attempt file recovery

When a process keeps a file open, the file remains on disk even after deletion. The process can still read/write via its file descriptor, though the file is invisible in the filesystem.

The /proc directory reflects kernel and process information in memory. Each process has a directory named by its PID (e.g., /proc/1234) containing files that expose file descriptors, symbolic links to disk files, and other details. Tools like lsof use this information to list open files.

If a deleted file is still opened by a running process, its contents can be recovered by reading the corresponding descriptor under /proc/<pid>/fd/<fd> and redirecting the output to a new file.

For example, if /var/log/secure was deleted, you can:

Confirm the file is missing.

Run lsof to find a process (e.g., rsyslogd with PID 1264) that still holds the file descriptor.

Inspect /proc/1264/fd/4 to view the file’s data.

Redirect the data to a new file using I/O redirection.

Verify that /var/log/secure now exists.