How to Detect and Recover From Linux Server Intrusions: Practical Steps

Background

With the growing popularity of open‑source products, Linux operations engineers must be able to quickly determine whether a machine has been compromised. The following procedures were tested on CentOS 6.9 and are applicable to other Linux distributions.

Common Compromise Indicators

Log files may be deleted or cleared. Verify the presence and integrity of logs such as /var/log/lastlog, /var/log/secure, and others.

Attackers may create or modify /etc/passwd and /etc/shadow. Examine these files for unexpected entries.

Check recent successful and failed login events using /var/log/lastlog.

Inspect currently logged‑in users via /var/run/utmp.

Review historical login records from /var/log/wtmp, including total connection time per user.

Monitor abnormal network traffic with tcpdump or iperf.

Search /var/log/secure for suspicious activity.

Process Inspection

Identify abnormal processes and their associated scripts:

Use top to find the PID of suspicious processes.

Locate the executable file in the virtual filesystem, e.g., /proc/<em>PID</em>/fd.

Recovering Deleted Files with lsof

If a critical log such as /var/log/secure has been removed, it can be recovered as long as a process still holds an open file descriptor.

Confirm the file is missing.

Run lsof | grep /var/log/secure to see which process (e.g., PID 1264 rsyslogd) still has the file open.

Inspect the descriptor path, e.g., /proc/1264/fd/4, which points to the deleted file’s data.

Redirect the descriptor’s contents to a new file: cat /proc/1264/fd/4 > /var/log/secure.

Verify that /var/log/secure is restored and contains the expected logs.

These steps are valuable for recovering deleted log files or databases that remain open by running processes.

Illustrative Commands (Images)