Recover Deleted Linux Log Files with lsof and /proc

When a Linux system is compromised or a log file is accidentally deleted, the file may still exist on disk as long as a process keeps it open, even though it is invisible in the filesystem.

Linux exposes open file information through the /proc filesystem; each process has a directory named by its PID, and the fd subdirectory contains entries that are symbolic links to the files opened by that process.

The lsof command can list open files and reveal which ones are marked as "(deleted)". By locating the PID of the process that still holds the deleted file, you can read the file’s contents via the corresponding descriptor, e.g., /proc/2699/fd/1.

Example: after confirming that syslogd (PID 2699) has /var/log/messages open and deleted, the following commands retrieve the data:

# lsof | grep /var/log/messages
syslogd   2699 root 1w REG 8,2 480817 330592 /var/log/messages (deleted)

# cat /proc/2699/fd/1 | head -n 5
Jan 13 08:59:02 station90 syslogd 1.4.1: restart.
Jan 13 10:44:22 station90 syslogd 1.4.1: restart.
Jan 13 10:44:22 station90 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jan 13 10:44:22 station90 kernel: Linux version 2.6.18-164.el5 ([email protected]) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Tue Aug 18 15:51:48 EDT 2009
Jan 13 10:44:22 station90 kernel: Command line: ro root=LABEL=/ rhgb quiet

Finally, redirect the descriptor’s content to a new file to restore the log: cat /proc/2699/fd/1 > /var/log/messages Touching the target file beforehand is optional. This technique is useful for recovering deleted logs, database transaction logs, and other critical files while the owning process remains running.