How to Use Filebeat for Efficient Log Collection and Multi‑Target Output

Filebeat Log Collection

Filebeat Overview

Filebeat ships with pre‑built modules that contain the ingest pipeline, Elasticsearch template, Filebeat prospectors and Kibana dashboards needed to collect, parse, enrich and visualize various log formats.

Modules provide a lightweight, single‑purpose log shipper that runs on servers without Java and can forward logs to Logstash, Elasticsearch, Redis and other destinations.

ELK Architecture

Filebeat Installation

Download the package from the official site: https://www.elastic.co/downloads/beats/filebeat

# 1. Download

# 2. Install
yum localinstall -y filebeat-5.6.16-x86_64.rpm

# 3. Backup original config
cp /etc/filebeat/filebeat.yml{,.bak}

Filebeat output to a file

# vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/blog.zls.com_access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: blog_ngx_log

- input_type: log
  paths:
    - /var/log/nginx/www.zls.com_access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: www_ngx_log

output.file:
  path: "/tmp"
  filename: "zls_filebeat.txt"

# Start Filebeat
systemctl start filebeat

Filebeat output to Logstash

Filebeat configuration

# vi /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/blog.zls.com_access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: ngx_zls

output.logstash:
  hosts: ["10.0.0.84:6666"]
  enabled: true
  worker: 1
  compression_level: 3
  # loadbalance: true

Logstash configuration

input{
  beats{
    port => "6666"
    codec => "json"
  }
}
filter{
  json{
    source => "message"
    remove_field => "message"
  }
}
output{
  elasticsearch{
    hosts => ["10.0.0.81:9200"]
    index => "%{type}-%{+yyyy.MM.dd}"
    codec => "json"
  }
}

Filebeat output to Elasticsearch

# vi /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/blog.zls.com_access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: ngx_zls

output.elasticsearch:
  hosts: ["10.0.0.81:9200"]
  index: "nginx_es-%{+yyyy.MM.dd}"

# Restart Filebeat
systemctl stop filebeat
rm -f /var/lib/filebeat/registry
systemctl start filebeat

Filebeat output to Redis

Filebeat configuration

# vi /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/blog.zls.com_access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: blog.zls.com_json

- input_type: log
  paths:
    - /var/log/nginx/www.zls.com_access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: www.zls.com_json

output.redis:
  hosts: ["10.0.0.52:6379"]
  key: "nginx_log"
  db: 6
  timeout: 5
  # password: zls

Logstash configuration

# vi /etc/logstash/conf.d/ngx_redis_es.conf
input{
  redis{
    data_type => "list"
    key => "nginx_log"
    host => "10.0.0.52"
    port => "6379"
    db => "6"
    codec => "json"
  }
}
filter{
  json{
    source => "message"
    remove_field => ["message"]
  }
}
output{
  elasticsearch{
    hosts => ["10.0.0.82:9200"]
    index => "%{type}-%{+yyyy.MM.dd}"
  }
}

Filebeat output to multiple targets

# vi /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /usr/local/nginx/logs/access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: ngx_log

- input_type: log
  paths:
    - /usr/local/tomcat/logs/tomcat_access_log.*.log
  exclude_lines: ["^DBG","^$"]
  document_type: tc_log

output.redis:
  hosts: ["10.0.0.54:6379"]
  key: "tn"
  db: 2
  timeout: 5
  password: zls

output.file:
  path: "/tmp"
  filename: "zls.txt"
  worker: 1
  compression_level: 3
  loadbalance: true