Master Log Collection with Filebeat and Graylog: A Complete Deployment Guide

Filebeat Tool Introduction

Filebeat is a log shipper that monitors specified log directories or files, reads new entries, and forwards them to Elasticsearch, Logstash, or Graylog.

Filebeat Workflow

When enabled, Filebeat starts prospectors to detect log files, launches a harvester for each file, reads new lines, sends them to the spooler, which aggregates events and forwards them to the configured output (e.g., Graylog).

Filebeat Configuration Example

# 配置输入来源的日志信息
# 我们合理将其配置到了 inputs.d 目录下的所有 yml 文件
filebeat.config.inputs:
  enabled: true
  path: ${path.config}/inputs.d/*.yml
  # 若收取日志格式为 json 的 log 请开启此配置
  # json.keys_under_root: true

# 配置 Filebeat 需要加载的模块
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

# 配置将日志信息发送那个地址上面
output.logstash:
  hosts: ["11.22.33.44:5500"]

processors:
  - add_host_metadata: ~
  - rename:
      fields:
        - from: "log"
          to: "message"
  - add_fields:
      target: ""
      fields:
        token: "0uxxxxaM-1111-2222-3333-VQZJxxxxxwgX "

# 收集的数据类型
- type: log
  enabled: true
  # 日志文件的路径地址
  paths:
    - /var/log/supervisor/app_escape_worker-stderr.log
    - /var/log/supervisor/app_escape_prod-stderr.log
  symlinks: true
  # 包含的关键字信息
  include_lines: ["WARNING", "ERROR"]
  # 打上数据标签
  tags: ["app", "escape", "test"]
  # 防止程序堆栈信息被分行识别
  multiline.pattern: '^\[?[0-9]...{3}'
  multiline.negate: true
  multiline.match: after

- type: log
  enabled: true
  ...

Modules for specific services such as iptables, PostgreSQL, and Nginx can be configured similarly.

Graylog Service Overview

Graylog is an open‑source log aggregation, analysis, and alerting platform that uses Elasticsearch for storage and MongoDB for configuration. It offers a simple web UI, inputs, extractors, streams, and pipelines for processing logs.

Graylog Architecture

Typical deployment includes three components: Elasticsearch (log storage), MongoDB (configuration), and Graylog (web UI and API).

Inputs collect logs, extractors transform fields, streams route logs to index sets, and pipelines enable custom processing such as discarding high‑level messages.

rule "discard debug messages"
when
  to_long($message.level) > 6
then
  drop_message();
end

Service Installation and Deployment

Deploy Filebeat via package managers, Docker, or source compilation. Example commands for Ubuntu deb, Docker run, and Docker‑compose configurations for Graylog, Elasticsearch, and MongoDB are provided.

# Ubuntu (deb)
$ curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.1-amd64.deb
$ sudo dpkg -i filebeat-7.8.1-amd64.deb
$ sudo systemctl enable filebeat
$ sudo service filebeat start

# Docker run Filebeat
docker run -d --name=filebeat --user=root \
  --volume "./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume "/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume "/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker.elastic.co/beats/filebeat:7.8.1 filebeat -e -strict.perms=false \
  -E output.elasticsearch.hosts=["elasticsearch:9200"]

# Graylog docker‑compose.yml (excerpt)
version: "3"
services:
  mongo:
    restart: on-failure
    container_name: graylog_mongo
    image: "mongo:3"
    volumes:
      - "./mongodb:/data/db"
    networks:
      - graylog_network
  elasticsearch:
    restart: on-failure
    container_name: graylog_es
    image: "elasticsearch:6.8.5"
    volumes:
      - "./es_data:/usr/share/elasticsearch/data"
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx5120m"
    networks:
      - graylog_network
  graylog:
    restart: on-failure
    container_name: graylog_web
    image: "graylog/graylog:3.3"
    ports:
      - "9000:9000"
      - "5044:5044"
      - "12201:12201"
      - "12201:12201/udp"
      - "1514:1514"
      - "1514:1514/udp"
    environment:
      - GRAYLOG_PASSWORD_SECRET=...
      - GRAYLOG_ROOT_PASSWORD_SHA2=...
      - GRAYLOG_HTTP_EXTERNAL_URI=http://11.22.33.44:9000/
    depends_on:
      - mongo
      - elasticsearch
    networks:
      - graylog_network
networks:
  graylog_network:
    driver: bridge

Graylog UI Features

The Graylog web interface provides dashboards, search, stream management, and alerting capabilities, illustrated by several screenshots.