Comprehensive Filebeat Tuning Parameters to Reduce High Resource Usage

Overview

Filebeat can consume excessive resources due to directory scanning, harvesters, parsers, queues, or downstream back‑pressure. Effective tuning requires evidence from logs, metrics, configuration, and process sampling while preserving data integrity, latency, throughput, and stability. The steps below apply to Filebeat 7.x and 8.x; replace <log directory> and <output address> with real values.

Identification

1. Verify version, startup flags, and config path

filebeat version
sudo systemctl show filebeat -p MainPID -p ExecStart -p FragmentPath --no-pager
sudo filebeat export config -c /etc/filebeat/filebeat.yml | sed -n '1,120p'

Exported config confirms the merged effective configuration, avoiding edits to files that are not loaded.

2. Continuously sample process resources

PID="$(systemctl show -p MainPID --value filebeat)"
ps -p "$PID" -o pid,etime,%cpu,%mem,rss,nlwp,cmd
pidstat -rud -p "$PID" 1 10

Sampling should cover peak log‑write periods; a single CPU snapshot does not represent long‑term trends.

3. Check retries and queue evidence

sudo journalctl -u filebeat --since '30 min ago' --no-pager \
  | grep -Ei 'error|retry|queue|publisher|harvester|registrar' | tail -n 200

Persistent retries, connection errors, or a full queue indicate downstream back‑pressure that must be addressed first.

4. Count open files and registry entries

sudo lsof -nP -p "$PID" | awk '$5=="REG" {print $9}' | sort | uniq -c | sort -nr | head
sudo ls -lh /var/lib/filebeat/registry/ 2>/dev/null || true

Do not delete the registry; it stores offsets and its removal can cause duplicate collection or data loss.

5. Distinguish active from historic files

sudo find <log directory> -type f -mmin -5 -printf '%s %p
' | head -n 50
sudo find <log directory> -type f | wc -l

When many historic files exist, narrow the scan range to active files.

Preparation

6. Backup and syntax check

#!/usr/bin/env bash
set -euo pipefail
CONFIG="/etc/filebeat/filebeat.yml"
BACKUP="/var/backups/filebeat/filebeat.yml.$(date +%Y%m%d-%H%M%S)"
sudo install -d -m 0750 /var/backups/filebeat
sudo cp -a "$CONFIG" "$BACKUP"
sudo filebeat test config -c "$CONFIG"

Keep the backup in version control; this step does not restart the service.

7. Verify output connectivity

sudo filebeat test output -c /etc/filebeat/filebeat.yml
sudo ss -tpn | grep -F filebeat || true

The test only checks connection and authentication; it does not guarantee downstream throughput.

Input Configuration

8. Use filestream and exclude rotated files

filebeat.inputs:
- type: filestream
  id: app-json
  paths: [<log directory>/*.json]
  prospector.scanner.exclude_files:
    - '\.gz$'
    - '\.[0-9]{1,8}$'

New deployments should prefer filestream. The id must be stable and unique; the same file cannot be matched by both log and filestream inputs.

9. Reduce new‑file scan frequency

filebeat.inputs:
- type: filestream
  id: app-json
  paths: [<log directory>/*.json]
  prospector.scanner.check_interval: 10s

Increasing check_interval delays detection of new files but does not affect reading of active files; verify business‑acceptable latency first.

10. Adjust idle polling for legacy log input

filebeat.inputs:
- type: log
  paths: [<log directory>/*.log]
  scan_frequency: 10s
  backoff: 1s
  max_backoff: 10s
  backoff_factor: 2

These settings apply only to versions still supporting the log input. Larger max_backoff reduces idle polling but lowers real‑time visibility.

11. Limit harvesters per input

filebeat.inputs:
- type: filestream
  id: app-limited
  paths: [<log directory>/*.log]
  harvester_limit: 128

Harvester limits protect file handles and memory but cause active files exceeding the limit to wait; set according to acceptable delay.

12. Verify service file‑descriptor limit

sudo systemctl show filebeat -p LimitNOFILE --no-pager
sudo -u filebeat sh -c 'ulimit -n'
sudo lsof -p "$PID" 2>/dev/null | wc -l

The nofile limit must cover harvesters, network connections, registry, and safety margin.

13. Set systemd nofile limit

# /etc/systemd/system/filebeat.service.d/limits.conf
[Service]
LimitNOFILE=65536

After editing, run daemon-reload and restart; ensure downstream redundancy and disk capacity can handle any backlog.

14. Close silent log files

filebeat.inputs:
- type: log
  paths: [<log directory>/*.log]
  close_inactive: 5m
  close_renamed: true
  close_removed: true
  ignore_older: 24h

close_inactive

must be larger than backoff; ignore_older does not delete files or registry entries.

15. Verify closed file handles

sudo find <log directory> -type f -name '*.log' -mmin +5 -printf '%p
' | head
sudo lsof -nP -p "$PID" | grep '<log directory>' | head

Files older than close_inactive that remain open indicate an issue; continuously written files should stay open.

Parsing

16. Set multiline rule for Java stack traces

parsers:
- multiline:
    type: pattern
    pattern: '^[[:space:]]+(at |\.{3}[0-9]+ more)'
    negate: false
    match: after

The rule matches only continuation lines; validate with real, sanitized samples to avoid swallowing subsequent normal logs.

17. Short‑term processor debugging

sudo filebeat -e -c /etc/filebeat/filebeat.yml \
  -d 'input,harvester,processors' 2>&1 | head -n 160

Debugging adds overhead and should be used only on test nodes or briefly for troubleshooting; do not enable permanently.

18. Drop health‑check events

processors:
- drop_event:
    when:
      or:
      - contains: {message: '/healthz'}
      - contains: {message: '/readyz'}

drop_event

is irreversible; ensure it does not affect audit, alerting, or security analysis and monitor the drop rate.

19. Decode only confirmed JSON messages

processors:
- decode_json_fields:
    fields: ["message"]
    target: ""
    overwrite_keys: false
    add_error_key: true

If the pipeline already outputs structured fields, avoid redundant decoding; add_error_key helps detect format drift.

20. Enumerate high‑cost parsers

sudo grep -R --line-number -E 'decode_json_fields|dissect|grok|script|multiline' \
  /etc/filebeat /etc/filebeat/modules.d 2>/dev/null

Prioritize optimization based on actual hit counts; moving parsing elsewhere does not eliminate its cost.

Output Configuration

21. Set bounded memory queue

queue.mem:
  events: 4096
  flush.min_events: 512
  flush.timeout: 5s

events

counts events, not bytes; large logs can inflate memory usage. Verify semantics for the specific version.

22. Use capacity‑controlled disk queue

queue.disk:
  path: "/var/lib/filebeat/diskqueue"
  max_size: 10GB
  segment_size: 1GB
  read_ahead: 512
  write_ahead: 2048

Disk queues relieve short‑term back‑pressure but increase I/O; capacity must stay below partition free space and alert thresholds.

23. Check disk‑queue partition

sudo df -h /var/lib/filebeat
sudo df -ih /var/lib/filebeat
sudo du -sh /var/lib/filebeat/* 2>/dev/null || true

Changing disk‑queue size is high‑risk; confirm quota, inode availability, alerts, and a gradual rollout plan.

24. Tune Elasticsearch bulk sending

output.elasticsearch:
  hosts: ["https://<output address>:9200"]
  worker: 2
  bulk_max_size: 1600
  compression_level: 3
  backoff.init: 1s
  backoff.max: 60s

Increasing worker and bulk_max_size can boost throughput but may cause downstream rejections; higher compression raises CPU usage.

25. Tune Logstash output concurrency

output.logstash:
  hosts: ["<output address>:5044"]
  loadbalance: true
  workers: 2
  compression_level: 3
  pipelining: 2

Load‑balancing requires multiple hosts; pipelining adds in‑flight batches.

26. Collect publish‑failure evidence

sudo journalctl -u filebeat --since '15 min ago' --no-pager \
  | grep -Ei 'publish|retry|connection|error' | tail -n 100

HTTP 429/503, TLS errors, or connection resets should be addressed by checking the target, authentication, or network first.

Validation

27. Enable local stats endpoint only

http.enabled: true
http.host: 127.0.0.1
http.port: 5066
monitoring.enabled: false

Do not expose the stats port publicly; field names vary by version, so use the actual endpoint.

28. Read internal stats

curl -fsS http://127.0.0.1:5066/stats | jq .
curl -fsS http://127.0.0.1:5066/?pretty

Focus on harvester, queue, publish success/failure rates, and trends rather than single instantaneous values.

29. Compare publish success vs. failure rates

# Use actual exporter metrics
sum(rate(filebeat_output_events_total[5m]))
sum(rate(filebeat_output_events_failed_total[5m]))

First verify the exporter’s metric names, labels, and units; do not assume uniformity across Filebeat versions.

Rollback

30. Single‑node restart verification

#!/usr/bin/env bash
set -euo pipefail
sudo filebeat test config -c /etc/filebeat/filebeat.yml
sudo filebeat test output -c /etc/filebeat/filebeat.yml
sudo systemctl daemon-reload
sudo systemctl restart filebeat
sleep 10
sudo systemctl is-active --quiet filebeat
sudo journalctl -u filebeat -n 80 --no-pager

Run on a single node first; verify service liveness, downstream delivery, latency, and application‑log backlog.

31. Restore configuration from backup

#!/usr/bin/env bash
set -euo pipefail
BACKUP_FILE="<absolute backup path>"
sudo test -r "$BACKUP_FILE"
sudo cp -a "$BACKUP_FILE" /etc/filebeat/filebeat.yml
sudo filebeat test config -c /etc/filebeat/filebeat.yml
sudo systemctl restart filebeat

Rollback restores configuration only; it cannot undo dropped events or duplicated sends.

Acceptance and Final Guidance

When file‑handle count is high, first tighten input scopes; when retries or backlog persist, address downstream issues; when parsing cost is high, optimize rules and drop low‑value events. Acceptance checks must include resource usage, end‑to‑end latency, downstream errors, and disk free space. Any changes involving restarts, disk queues, or drop rules require gradual rollout, backups, and must keep log integrity as the rollback condition.