Master Log Collection with Filebeat and Graylog: A Step‑by‑Step Guide

When deploying many services across test and production environments, centralized log collection becomes essential.

Graylog, which uses Elasticsearch for storage and MongoDB for configuration, offers a simple, extensible interface, making it an attractive alternative to Nginx‑based or ELK solutions.

Filebeat Overview

1. Filebeat log shipper

Filebeat monitors specified log directories or files, reads new entries, and forwards them to Elasticsearch, Logstash, or Graylog.

2. Filebeat workflow

After installation, Filebeat starts prospectors that detect configured paths, spawns harvesters for each file, sends harvested events to a spooler, and finally forwards the batch to the configured output (e.g., Graylog).

3. Why choose Filebeat

Filebeat is lightweight compared to Logstash and works well on machines with limited resources.

Filebeat Configuration

The main configuration resides in /etc/filebeat/filebeat.yml (or equivalent on macOS/Windows). Inputs are defined under inputs.d/*.yml, allowing separate configurations per service.

# Configure input sources
filebeat.config.inputs:
  enabled: true
  path: ${path.config}/inputs.d/*.yml
# Configure modules
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
output.logstash:
  hosts: ["11.22.33.44:5500"]
processors:
  - add_host_metadata: ~
  - rename:
      fields:
        - from: "log"
          to: "message"
  - add_fields:
      target: ""
      fields:
        token: "0uxxxxaM-1111-2222-3333-VQZJxxxxxwgX"

Example input definition (inputs.d/example.yml):

- type: log
  enabled: true
  paths:
    - /var/log/supervisor/app_escape_worker-stderr.log
    - /var/log/supervisor/app_escape_prod-stderr.log
  symlinks: true
  include_lines: ["WARNING","ERROR"]
  tags: ["app","escape","test"]
  multiline.pattern: '^\[?[0-9]...{3}'
  multiline.negate: true
  multiline.match: after

Graylog components include Inputs, Extractors, Streams, Index Sets, and Pipelines. Inputs collect data, Extractors transform fields, Streams route logs to specific indices, and Pipelines allow custom processing such as discarding messages with level > 6.

rule "discard debug messages"
when
  to_long($message.level) > 6
then
  drop_message();
end

Sidecar agents (Filebeat, Winlogbeat, NXLog) can pull configuration from Graylog and forward logs, supporting various output formats.

Deployment

Filebeat can be installed via DEB/RPM packages, Docker, or compiled from source. Example Docker command:

docker run -d --name=filebeat --user=root \
  --volume "./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume "/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume "/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker.elastic.co/beats/filebeat:7.8.1 filebeat -e -strict.perms=false \
  -E output.elasticsearch.hosts=["elasticsearch:9200"]

Graylog can be deployed with Docker Compose. After generating a password secret and root password hash, the following docker‑compose.yml starts MongoDB, Elasticsearch, and Graylog services:

version: "3"
services:
  mongo:
    restart: on-failure
    container_name: graylog_mongo
    image: "mongo:3"
    volumes:
      - "./mongodb:/data/db"
    networks: [graylog_network]

  elasticsearch:
    restart: on-failure
    container_name: graylog_es
    image: "elasticsearch:6.8.5"
    volumes:
      - "./es_data:/usr/share/elasticsearch/data"
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - ES_JAVA_OPTS=-Xms512m -Xmx5120m
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 12g
    networks: [graylog_network]

  graylog:
    restart: on-failure
    container_name: graylog_web
    image: "graylog/graylog:3.3"
    ports:
      - 9000:9000
      - 5044:5044
      - 12201:12201
      - 12201:12201/udp
      - 1514:1514
      - 1514:1514/udp
    volumes:
      - "./graylog_journal:/usr/share/graylog/data/journal"
    environment:
      - GRAYLOG_PASSWORD_SECRET=zscMb65...FxR9ag
      - GRAYLOG_ROOT_PASSWORD_SHA2=77e29e0f...557515f
      - GRAYLOG_HTTP_EXTERNAL_URI=http://11.22.33.44:9000/
      - GRAYLOG_TIMEZONE=Asia/Shanghai
      - GRAYLOG_ROOT_TIMEZONE=Asia/Shanghai
    networks: [graylog_network]
    depends_on: [mongo, elasticsearch]

networks:
  graylog_network:
    driver: bridge

Docker containers can send logs to Graylog via the GELF driver, for example:

docker run --rm=true \
  --log-driver=gelf \
  --log-opt gelf-address=udp://11.22.33.44:12201 \
  --log-opt tag=myapp \
  myapp:0.0.1

Graylog’s web UI provides search, stream management, and pipeline editing capabilities.