Why Deleted Files Still Fill Your Linux Disk and How Filebeat Fixes It

On a Red Hat Linux application server we encountered a situation where the

/opt/applog

filesystem reported usage above the threshold despite the actual file size being far smaller. The root cause was that many log files had been deleted but their disk space remained allocated because the processes that opened them kept the file handles open.

In Linux every resource is represented as a file, and each running program holds a file descriptor that points to the underlying file. To discover which files are still open we can use the

lsof

command, which lists all open files and their associated processes.

<code>lsof
COMMAND      PID   USER   FD   TYPE   DEVICE   SIZE/OFF   NODE   NAME
...
filebeat 111442 app 1r REG 253,3 209715229 1040407 /opt/applog/E.20171016.info.012.log
filebeat 111442 app 2r REG 253,3 209715254 385080 /opt/applog/E.20171015.info.001.log (deleted)
...</code>

The columns show the command name, process ID, user, file descriptor, type, device, size, inode, and the full path of the opened file. Notice the

(deleted)

tag in the

NAME

column – the file has been removed from the directory tree but the process (in this case

filebeat

) still holds it open, preventing the space from being reclaimed.

Filebeat is part of the ELK (Elasticsearch‑Logstash‑Kibana) stack, a popular open‑source log‑collection and analysis platform. Elasticsearch provides distributed search, Logstash collects and transforms logs, and Kibana offers a web UI for visualization. Filebeat acts as a lightweight agent that tails log files and forwards them to Logstash or Elasticsearch.

In our environment a scheduled job deletes logs older than 12 hours. When the job removes a file that Filebeat is still reading, the file becomes "deleted" but its handle stays open, causing the filesystem to appear full.

Solution 1 : Stop the Filebeat process (

kill -9 $(pidof filebeat)

) to force the kernel to release the held space. This is a quick fix but does not prevent the problem from recurring.

Solution 2 : Adjust Filebeat’s configuration to close idle file handles automatically. Two relevant settings are:

close_older: 1h

– closes a file handle if the file has not been modified for the specified duration.

force_close_files: false

– when set to true, Filebeat closes a file as soon as its name changes (e.g., on rotation or deletion). The recommended approach is to keep this false and lower

ignore_older

, but for our case we enable it.

<code>close_older: 30m
force_close_files: true</code>

With these settings Filebeat will close handles for logs that are older than 30 minutes or whose names have changed, allowing the kernel to reclaim the space promptly.

By combining proper log‑rotation policies with Filebeat’s automatic handle‑closing features, we can ensure that deleted log files no longer consume disk space.