Master Filebeat 7.7: From Basics to Advanced Log Shipping

Filebeat Overview

Filebeat 7.7.0 is a lightweight log shipper that monitors specified log files, reads new entries, and forwards events to Elasticsearch or Logstash for indexing.

Relationship with Beats

Filebeat is one of the six Beats tools. Beats are lightweight data shippers written in Go that consume far fewer resources than Logstash. The family includes Packetbeat, Metricbeat, Filebeat, Winlogbeat, Auditbeat, and Heartbeat.

Filebeat Architecture

Filebeat consists of two main components: inputs (which discover log files) and harvesters (which read each file line‑by‑line). Harvesters keep files open, track the last read offset, and write state information to a registry file so that processing can resume after restarts.

Filebeat guarantees at‑least‑once delivery by persisting each event’s delivery status; if an output is unavailable, events are retried until acknowledged.

Installation

Download the tarball filebeat-7.7.0-linux-x86_64.tar.gz, extract it, and use the provided filebeat.reference.yml as a reference for all configuration options.

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.0-linux-x86_64.tar.gz
tar -xzvf filebeat-7.7.0-linux-x86_64.tar.gz

Basic Commands

export

– export environment variables run – start Filebeat (default) test – test configuration keystore – manage secret storage modules – manage module configurations setup – initialize environment

Inputs and Outputs

Supported input types include log, syslog, TCP, UDP, Docker, Kafka, etc. The most common is log. Supported outputs include Elasticsearch, Logstash, Kafka, Redis, file, console, and Elastic Cloud.

Keystore Usage

Use the keystore to store sensitive values such as passwords, referenced in the config as ${ES_PWD}.

filebeat keystore create
filebeat keystore add ES_PWD
filebeat keystore remove ES_PWD
filebeat keystore list

Example Configuration – Logstash Output

# Filebeat inputs
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/logs/es_aaa_index_search_slowlog.log
    - /var/logs/es_bbb_index_search_slowlog.log
    - /var/logs/es_ccc_index_search_slowlog.log
    - /var/logs/es_ddd_index_search_slowlog.log

# Logstash output
output.logstash:
  hosts: ["192.168.110.130:5044", "192.168.110.131:5044", "192.168.110.132:5044", "192.168.110.133:5044"]
  loadbalance: true

Example Configuration – Elasticsearch Output

# Filebeat inputs (same as above)

# Elasticsearch output
output.elasticsearch:
  hosts: ["192.168.110.130:9200", "192.168.110.131:9200"]
  username: "elastic"
  password: "${ES_PWD}"

Running Filebeat

Start Filebeat with ./filebeat -e. Logs will be indexed into an index named filebeat-%{[beat.version]}-%{+yyyy.MM.dd} in Elasticsearch.

Filebeat Modules

Modules simplify collection for common services. The Elasticsearch module can parse ES slow‑log queries. Typical steps:

Configure filebeat.yml (set setup.kibana.host, credentials, etc.).

Enable the module: ./filebeat modules enable elasticsearch.

Load the module dashboards: ./filebeat setup -e.

Start Filebeat.

After setup, Kibana will display parsed slow‑log entries.