How to Detect and Recover from Linux Server Intrusions: Essential Checks

As open‑source products become increasingly popular, Linux operations engineers must be able to determine whether a machine has been compromised. Drawing on personal experience, the author lists common indicators of a hacked system (examined on CentOS 6.9, but applicable to other distributions).

1. Attackers may delete log files

Verify whether log files still exist or have been cleared; example commands are shown.

2. Attackers may create a new file containing usernames and passwords

Inspect

/etc/passwd

and

/etc/shadow

files; example commands are provided.

3. Attackers may modify username and password files

Review the contents of

/etc/passwd

and

/etc/shadow

to detect unauthorized changes; example commands are shown.

4. Check recent successful login events and the last failed login

Examine the

/var/log/lastlog

file; example commands are provided.

5. View all currently logged‑in users

Read the

/var/run/utmp

file; example commands are shown.

6. List users who have logged in since the system was created

Inspect the

/var/log/wtmp

file; example commands are shown.

7. Show total connection time (hours) for each user

Again use the

/var/log/wtmp

file; example commands are displayed.

8. Detect abnormal network traffic

Capture packets with

tcpdump

or measure bandwidth using

iperf

.

9. Examine the /var/log/secure file for attacker clues

Run the shown commands to search for suspicious entries.

10. Identify scripts associated with suspicious processes

a) Use

top

to find the PID of the abnormal process.

b) Search the virtual file system for the executable file of that PID.

11. Recover deleted important files (e.g., /var/log/secure )

1) When a process keeps a file open, the file remains on disk even after deletion; it is invisible to the filesystem but accessible via the process's file descriptor.

2) The

/proc

directory mirrors kernel and process information in memory; each process has a directory like

/proc/1234

containing file descriptors and symbolic links to the actual files.

3) If a deleted file is still opened by a process,

lsof

can reveal the file descriptor, allowing recovery from

/proc/<pid>/fd/<fd>

.

Example recovery steps for a deleted

/var/log/secure

file:

a) Verify the file is missing.

b) Use

lsof

to check if any process still holds the file open.

c) The output shows that PID 1264 (rsyslogd) has file descriptor 4 open to the deleted file.

d) Read the content via

/proc/1264/fd/4

and redirect it to a new file.

e) After redirection, the

/var/log/secure

file reappears and can be used normally again.