Essential Linux Commands for Incident Response and System Forensics

1. Identify a running process by filename

# pidof filename

2. Find processes using a file or TCP/UDP port

# fuser -n tcp port

3. View file metadata (modification time, size, etc.)

# stat filename

4. List loaded kernel modules

# lsmod

5. Check RPC services

# rpcinfo -p

6. Verify if a network interface is in promiscuous mode

# dmesg | grep eth0

7. Verify integrity of system binaries (similar to md5sum)

# rpm -Vf /bin/ls

Use rpm -Vf /bin/ps – no output means the file is unchanged; otherwise it reports modifications. If the RPM database is compromised, compare against a trusted repository, e.g.,

rpm -Vvp ftp://mirror.site/dir/RedHat/RPMS/fileutils-3.16-10.i386.rpm

.

Common binaries to verify:

/usr/bin/chfn

/usr/bin/chsh

/bin/login

/bin/ls

/usr/bin/passwd

/bin/ps

/usr/bin/top

/usr/sbin/in.rshd

/bin/netstat

/sbin/ifconfig

/usr/sbin/syslogd

/usr/sbin/inetd

/usr/sbin/tcpd

/usr/bin/killall

/sbin/pidof

/usr/bin/find

8. When the machine is confirmed compromised

1. Create a disk image on an external backup drive. 2. Mount a read‑only rescue CD containing static versions of common tools (ls, ps, netstat, etc.). 3. Use nc to stream command output to a remote host.

9. Record a baseline of file hashes

find /sbin -type f | xargs md5sum > 1st

Later verify with:

md5sum -c 1st | grep OK

10. Minimize writes on an infected host

On a clean machine run: nc -L -p 1234 > some_audit_output.log (uppercase L for persistent listening). On the compromised host send output with: command | nc 192.168.20.191 1234 or use script > /mnt/export.log and finish with Ctrl+D.

11. Locate suspicious processes

1. netstat -anp – note unusual entries. 2. Enter the process directory, e.g., cd /proc/3299. 3. ls -la to see the executable path. 4. Inspect the fd subdirectory for open file handles. 5. ps -awx to review the process list again.

12. Recover deleted logs

Search for undeleted logs (history, sniffer logs) and examine /proc/*/fd entries that point to deleted files. Use a statically compiled lsof | grep deleted to list them, note the inode number, then recover with SleuthKit: df /var → identify the device (e.g., sda1) icat /dev/sda1 149743 (replace 149743 with the inode).

13. Detect hidden processes

Compare the list of PIDs from ps -ef with entries in /proc:

ps -ef | awk '{print $2}' | sort -n | uniq > 1

ls /proc | sort -n | uniq > 2

diff 1 2

14. Emergency toolset (tct)

Contains utilities such as icat for data recovery. For remote imaging, run on a clean host: nc -L -p 1234 > abc.img On the compromised host:

dd if=/dev/hdb5 count 20000 bs=1024 | nc 192.168.0.1 1234 -w 3

If the image is large, split across multiple ports and concatenate the parts.

15. Inspect dynamic library dependencies

ldd /bin/ls

shows direct dependencies; indirect ones may be missed. Example output:

libtermcap.so.2 => /lib/libtermcap.so.2 (0x40022000)
libc.so.6 => /lib/tls/libc.so.6 (0x42000000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

16. Using strace to trace system calls

Example to trace open calls of /bin/ls: strace -e open /bin/ls > /dev/null To capture all calls to a running process and write to a file: strace -o out -ff -p PID Filter the output for open entries:

grep open out* | grep -v -e No -e null -e denied | grep WR

17. Forward system logs to a remote log server

Edit /etc/syslog.conf and add: *.* @192.168.20.163 Restart syslog and verify with a failed login attempt or with tcpdump port 514.

18. Time‑based file discovery

If the intrusion started at a known time (e.g., 09‑27), create a reference file: touch -t 09270000 /tmp/a Then find files newer than that:

find / \( -newer /tmp/a -o -cnewer /tmp/a \) -ls

19. Full‑disk cloning

dd if=/dev/sda of=/dev/sdb bs=1024

– clone entire disk. For partition cloning:

dd if=/dev/sda1 of=/abc bs=1024

20. Searching for specific strings

Example to locate files containing a phrase: find /tmp -type f -exec grep "no exist" {} \; -print Search for executables named *crond in /etc/rc.d and inspect them:

find /etc/rc.d -name '*crond' -exec file {} \;

21. Generating a core dump for analysis

Send a SIGSEGV to a process to produce a core file, then use strings or a custom C program to reconstruct the executable.