How to Detect If Your Linux Server Has Been Compromised: 10 Essential Checks

With the rise of open‑source products, it is crucial for Linux operations engineers to clearly determine whether an abnormal machine has been compromised. Based on personal experience, the following common indicators of a compromised machine are provided for reference.

Background: The cases below were observed on a CentOS 6.9 system; similar steps apply to other Linux distributions.

1. Attackers may delete log files

Check whether log files still exist or have been cleared. Example commands are shown in the accompanying image.

2. Attackers may create a new file containing usernames and passwords

Inspect /etc/passwd and /etc/shadow files. Example commands are illustrated in the image.

3. Attackers may modify the username and password files

Compare the contents of /etc/passwd and /etc/shadow to detect tampering. Commands are shown in the image.

4. Review recent successful login events and the last failed login attempt

Check the /var/log/lastlog file. Example commands are displayed in the image.

5. View all currently logged‑in users

Inspect the /var/run/utmp log file. Commands are illustrated in the image.

6. View all users who have ever logged in since the machine was created

Check the /var/log/wtmp file. Example commands are shown in the image.

7. View the total login time (in hours) for all users

Again, the /var/log/wtmp file provides this information. Commands are displayed in the image.

8. If the machine generates abnormal traffic

Use tcpdump to capture network packets or tools like iperf to analyze traffic.

9. Examine the /var/log/secure log file

Search for attacker information. Example commands are illustrated in the image.

10. Identify the script file associated with an abnormal process

a. Use top to find the PID of the suspicious process.

b. Locate the executable file in the virtual file‑system.

11. Recover deleted important files when intrusion is confirmed

1) If a process still holds an opened file descriptor, the file remains on disk even after deletion; it can be read via the descriptor.

2) The /proc directory mirrors kernel and process information in memory. Each process has a directory named by its PID (e.g., /proc/1234) containing file descriptors and links to the original files. Tools like lsof can list these descriptors.

3) When a file is deleted but a process still accesses it, lsof can be used to retrieve the file content from /proc.

Example: If the attacker deleted /var/log/secure, the recovery steps are:

a) Verify that /var/log/secure is missing.

b) Use lsof to see if any process still has the file open.

c) The output shows PID 1264 (rsyslogd) with file descriptor 4 pointing to the deleted file. The file can be accessed via /proc/1264/fd/4.

d) Redirect the content to a new file, e.g., cat /proc/1264/fd/4 > /var/log/secure.

e) Verify that /var/log/secure now exists. This method is useful for many applications, especially log files and databases.

Article reprinted from Devops技术栈 (copyright belongs to the original author, removal upon request).