How to Detect and Recover from Linux Server Intrusions: Essential Commands and Techniques

As a Linux operations engineer, being able to quickly determine whether a machine has been compromised is crucial. The following checklist, demonstrated on a CentOS 6.9 system, shows typical intrusion indicators and the commands to verify them.

1. Check for deleted or cleared log files

[root@hlmcen69n3 ~]# ll -h /var/log/*
-rw-------. 1 root root 2.6K Jul  7 18:31 /var/log/anaconda.ifcfg.log
-rw-------. 1 root root 23K Jul  7 18:31 /var/log/anaconda.log
-rw-------. 1 root root 26K Jul  7 18:31 /var/log/anaconda.program.log
-rw-------. 1 root root 63K Jul  7 18:31 /var/log/anaconda.storage.log

[root@hlmcen69n3 ~]# du -sh /var/log/*
8.0K /var/log/anaconda
4.0K /var/log/anaconda.ifcfg.log
24K  /var/log/anaconda.log
28K  /var/log/anaconda.program.log
64K  /var/log/anaconda.storage.log

2. Look for suspicious user/password files

[root@hlmcen69n3 ~]# ll /etc/pass*
-rw-r--r--. 1 root root 1373 Sep 15 11:36 /etc/passwd
-rw-r--r--. 1 root root 1373 Sep 15 11:36 /etc/passwd-

[root@hlmcen69n3 ~]# ll /etc/sha*
-rw-------. 1 root root 816 Sep 15 11:36 /etc/shadow
-rw-------. 1 root root 718 Sep 15 11:36 /etc/shadow-

3. Verify integrity of /etc/passwd and /etc/shadow

[root@hlmcen69n3 ~]# more /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin

[root@hlmcen69n3 ~]# more /etc/shadow
root:*LOCK*:14600:::::
bin:*:17246:0:99999:7::
daemon:*:17246:0:99999:7::

4. Review recent successful and failed login events

[root@hlmcen69n3 ~]# lastlog
Username          Port  From             Latest
root                                 **Never logged in**
bin                                  **Never logged in**
daemon                               **Never logged in**

5. List currently logged‑in users

[root@hlmcen69n3 ~]# who
stone    pts/0        2017-09-20 16:17 (X.X.X.X)
test01   pts/2        2017-09-20 16:47 (X.X.X.X)

6. Show all users that have ever logged in (wtmp)

[root@hlmcen69n3 ~]# last
test01   pts/1        X.X.X.X   Wed Sep 20 16:50 still logged in
test01   pts/2        X.X.X.X   Wed Sep 20 16:47 - 16:49 (00:02)
stone    pts/1        X.X.X.X   Wed Sep 20 16:46 - 16:47 (00:01)
stone    pts/0        X.X.X.X   Wed Sep 20 16:17

7. Check total connection time per user (wtmp)

[root@hlmcen69n3 ~]# ac -dp
stone                 11.98
Sep 15  total        11.98
stone                 67.06
Sep 18  total        67.06
stone                  1.27
test01                 0.24
Today   total         1.50

8. Capture abnormal network traffic

Use tcpdump to capture packets or iperf to measure bandwidth when unusual traffic is observed.

9. Search the secure log for intrusion clues

[root@hlmcen69n3 ~]# cat /var/log/secure | grep -i "accepted password"
Sep 20 12:47:20 hlmcen69n3 sshd[37193]: Accepted password for stone from X.X.X.X port 15898 ssh2
Sep 20 16:17:47 hlmcen69n3 sshd[38206]: Accepted password for stone from X.X.X.X port 9140 ssh2
Sep 20 16:46:04 hlmcen69n3 sshd[38511]: Accepted password for stone from X.X.X.X port 2540 ssh2
Sep 20 16:47:16 hlmcen69n3 sshd[38605]: Accepted password for test01 from X.X.X.X port 10790 ssh2
Sep 20 16:50:04 hlmcen69n3 sshd[38652]: Accepted password for test01 from X.X.X.X port 28956 ssh2

10. Identify the script behind an abnormal process

First, find the PID with top (see image).

Then locate the executable via the /proc filesystem:

[root@hlmcen69n3 ~]# ll /proc/1850/ | grep -i exe
lrwxrwxrwx. 1 root root 0 Sep 15 12:31 exe -> /usr/bin/python

[root@hlmcen69n3 ~]# ll /usr/bin/python
-rwxr-xr-x. 2 root root 9032 Aug 18 2016 /usr/bin/python

11. Recover a deleted critical file (e.g., /var/log/secure)

If a file has been removed but a process still holds it open, the data can be recovered from the process’s file descriptor.

# Verify the file is missing
[root@hlmcen69n3 ~]# ll /var/log/secure
ls: cannot access /var/log/secure: No such file or directory

# Find the process that still has it open
[root@hlmcen69n3 ~]# lsof | grep /var/log/secure
rsyslogd 1264 root 4w REG 8,1 3173904 /var/log/secure (deleted)

# Read the file descriptor and redirect to a new file
[root@hlmcen69n3 ~]# cat /proc/1264/fd/4 > /var/log/secure

# Verify the restored file
[root@hlmcen69n3 ~]# ll /var/log/secure
-rw-r--r--. 1 root root 3173904 Sep 20 17:24 /var/log/secure
[root@hlmcen69n3 ~]# head /var/log/secure
Sep 17 03:28:15 hlmcen69n3 sshd[13288]: reverse mapping checking getaddrinfo for 137-64-15-51.rev.cloud.scaleway.com [51.15.64.137] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 17 03:28:15 hlmcen69n3 unix_chkpwd[13290]: password check failed for user (root)
Sep 17 03:28:15 hlmcen69n3 sshd[13288]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=51.15.64.137  user=root

Recovering deleted log files or databases in this way can be invaluable during incident response.