Detecting and Recovering Linux Server Intrusions: Essential Commands

Detecting and Recovering Linux Server Intrusions

As open‑source products become more prevalent, Linux operations engineers must be able to determine whether a machine has been compromised. The following checklist, based on experience with CentOS 6.9 (applicable to other distributions), outlines typical intrusion indicators and recovery steps.

1. Missing or cleared logs – Verify that log files still exist and have not been emptied. Example commands are shown in the accompanying screenshots.

2. Suspicious passwd/shadow files – Attackers may create a new file containing usernames and passwords. Inspect /etc/passwd and /etc/shadow for anomalies.

3. Modified passwd/shadow files – Compare current contents with known good copies to detect tampering.

4. Recent successful and failed login events – Review /var/log/lastlog for the last successful login and the most recent failed attempts.

5. Currently logged‑in users – Check /var/run/utmp to see who is logged in now.

6. Historical login records – Examine /var/log/wtmp for a list of users who have logged in since the system was created.

7. User session durations – Use the same /var/log/wtmp file to determine how long each user has been connected.

8. Abnormal network traffic – Capture packets with tcpdump or measure throughput with iperf to spot unusual flows.

9. Examine the secure log – Look at /var/log/secure for clues about the attacker’s activity.

10. Identify malicious processes – Use top to find suspicious PIDs, then locate the corresponding executable in the virtual file system.

11. Recover deleted critical files – If important files (e.g., logs) have been removed, they may still be open by a running process. The lsof command can reveal which processes hold file descriptors to deleted files. By accessing the descriptor via /proc/<PID>/fd/<fd> and redirecting its output, the file can be restored.

Example recovery of /var/log/secure:

Confirm the file is missing.

Run lsof | grep /var/log/secure to find the process (e.g., PID 1264, rsyslogd) holding the deleted file.

Inspect /proc/1264/fd/4 to view the file’s contents.

Redirect the output to a new file, e.g., cat /proc/1264/fd/4 > /var/log/secure.

Verify that the restored file now exists and contains the expected log entries.

These techniques are valuable for many applications, especially when dealing with log files and databases that may have been targeted by attackers.

Author: 铭的随记 Source: http://www.cnblogs.com/stonehe/p/7562374.html