Master Filebeat: Collect and Ship Logs to Elasticsearch, Logstash, and Redis

Filebeat Log Collection

Filebeat Overview

Filebeat comes with pre‑built modules that contain the configurations needed to collect, parse, enrich and visualize various log file formats. Each module consists of one or more file sets, which include ingest node pipelines, Elasticsearch templates, Filebeat prospectors configuration and Kibana dashboards.

Filebeat is a lightweight, single‑purpose log shipper that can run on servers without Java and forward logs to Logstash, Elasticsearch, Redis, and other destinations.

ELK Architecture

Filebeat Installation and Deployment

Download from the official site: https://www.elastic.co/downloads/beats/filebeat

# 1. Download
# 2. Install
yum localinstall -y filebeat-5.6.16-x86_64.rpm

# 3. Backup original config
cp /etc/filebeat/filebeat.yml{,.bak}

Filebeat output to a file

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/blog.zls.com_access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: blog_ngx_log

- input_type: log
  paths:
    - /var/log/nginx/www.zls.com_access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: www_ngx_log

output.file:
  path: "/tmp"
  filename: "zls_filebeat.txt"

# start filebeat
systemctl start filebeat

Filebeat to Logstash

Filebeat configuration

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/blog.zls.com_access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: ngx_zls

output.logstash:
  hosts: ["10.0.0.84:6666"]
  enabled: true
  worker: 1
  compression_level: 3

Logstash configuration

input{
  beats{
    port => "6666"
    codec => "json"
  }
}
filter{
  json{
    source => "message"
    remove_field => "message"
  }
}
output{
  elasticsearch{
    hosts => ["10.0.0.81:9200"]
    index => "%{type}-%{+yyyy.MM.dd}"
    codec => "json"
  }
}

Filebeat to Elasticsearch

output.elasticsearch:
  hosts: ["10.0.0.81:9200"]
  index: "nginx_es-%{+yyyy.MM.dd}"

Filebeat to Redis

output.redis:
  hosts: ["10.0.0.52:6379"]
  key: "nginx_log"
  db: 6
  timeout: 5

Logstash configuration for Redis

input{
  redis{
    data_type => "list"
    key => "nginx_log"
    host => "10.0.0.52"
    port => "6379"
    db => "6"
    codec => "json"
  }
}
filter{
  json{
    source => "message"
    remove_field => ["message"]
  }
}
output{
  elasticsearch{
    hosts => ["10.0.0.82:9200"]
    index => "%{type}-%{+yyyy.MM.dd}"
  }
}

Filebeat to multiple destinations

filebeat.prospectors:
- input_type: log
  paths:
    - /usr/local/nginx/logs/access_json.log
  exclude_lines: ["^DBG","^$"]
  document_type: ngx_log
- input_type: log
  paths:
    - /usr/local/tomcat/logs/tomcat_access_log.*.log
  exclude_lines: ["^DBG","^$"]
  document_type: tc_log

output.redis:
  hosts: ["10.0.0.54:6379"]
  key: "tn"
  db: 2
  timeout: 5
  password: zls

output.file:
  path: "/tmp"
  filename: "zls.txt"
  worker: 1
  compression_level: 3
  loadbalance: true