Master Log Collection: Deploy Filebeat & Graylog for Centralized Logging

Filebeat Tool Introduction

Filebeat is a lightweight log shipper that monitors specified log directories or files, reads new entries continuously, and forwards the data to destinations such as Elasticsearch, Logstash, or Graylog.

Filebeat Workflow

When Filebeat starts, it launches one or more prospectors to watch the configured paths. Each prospector spawns a harvester for each discovered file. Harvesters read the latest content, send events to the spooler, which aggregates them, and finally the aggregated data is sent to the configured output (e.g., Graylog).

Why Choose Filebeat

Compared with Logstash, Filebeat is more lightweight, making it ideal for environments with limited resources or simple log collection needs.

Filebeat Configuration File

The core of Filebeat configuration resides in /etc/filebeat/filebeat.yml (or equivalent paths on macOS/Windows). The file defines inputs, modules, and output settings.

# Configure input sources (inputs.d/*.yml)
filebeat.config.inputs:
  enabled: true
  path: ${path.config}/inputs.d/*.yml
  # Enable JSON parsing if needed
  # json.keys_under_root: true

# Load modules
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

# Output to Graylog (Logstash protocol)
output.logstash:
  hosts: ["11.22.33.44:5500"]

processors:
  - add_host_metadata: ~
  - rename:
      fields:
        - from: "log"
          to: "message"
  - add_fields:
      target: ""
      fields:
        token: "0uxxxxaM-1111-2222-3333-VQZJxxxxxwgX"

An example of an inputs.d YAML file shows how to collect specific log files, apply tags, and enable multiline handling.

# Log type definition
- type: log
  enabled: true
  paths:
    - /var/log/supervisor/app_escape_worker-stderr.log
    - /var/log/supervisor/app_escape_prod-stderr.log
  symlinks: true
  include_lines: ["WARNING", "ERROR"]
  tags: ["app", "escape", "test"]
  multiline.pattern: '^\[?[0-9]...{3}'
  multiline.negate: true
  multiline.match: after

# Additional log types can be added similarly
- type: log
  enabled: true
  ...

Filebeat also supports modules for common services such as PostgreSQL, Redis, and iptables.

# iptables module
- module: iptables
  log:
    enabled: true
    var.paths: ["/var/log/iptables.log"]
    var.input: "file"

# postgresql module
- module: postgresql
  log:
    enabled: true
    var.paths: ["/path/to/log/postgres/*.log*"]

# nginx module
- module: nginx
  access:
    enabled: true
    var.paths: ["/path/to/log/nginx/access.log*"]
  error:
    enabled: true
    var.paths: ["/path/to/log/nginx/error.log*"]

Graylog Service Introduction

Graylog is an open‑source log aggregation, analysis, and alerting platform. It combines Elasticsearch for storage, MongoDB for configuration, and provides a web UI for searching and visualizing logs.

Graylog’s architecture consists of three main components: Elasticsearch (IO‑intensive storage), MongoDB (configuration storage), and Graylog server (CPU‑intensive web interface and API).

Graylog processes logs through Inputs, Extractors, Streams, and Index Sets. Inputs collect data, Extractors transform fields, Streams route logs to specific Index Sets, and Pipelines allow advanced processing such as discarding messages above a certain level.

rule "discard debug messages"
when
  to_long($message.level) > 6
then
  drop_message();
end

Service Installation and Deployment

Deploy Filebeat and Graylog using Docker, Docker‑Compose, or native packages. Example commands for installing Filebeat on Ubuntu:

# Ubuntu (deb)
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.1-amd64.deb
sudo dpkg -i filebeat-7.8.1-amd64.deb
sudo systemctl enable filebeat
sudo service filebeat start

Docker command to run Filebeat:

docker run -d --name=filebeat --user=root \
  --volume "./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume "/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume "/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker.elastic.co/beats/filebeat:7.8.1 filebeat -e -strict.perms=false \
  -E output.elasticsearch.hosts=["elasticsearch:9200"]

Graylog can be deployed via Docker‑Compose. After generating a password secret and root password hash, create a docker-compose.yml file with MongoDB, Elasticsearch, and Graylog services, then start with docker-compose up -d. The Graylog web UI is accessible on port 9000.

version: "3"
services:
  mongo:
    restart: on-failure
    container_name: graylog_mongo
    image: "mongo:3"
    volumes:
      - "./mongodb:/data/db"
    networks:
      - graylog_network

  elasticsearch:
    restart: on-failure
    container_name: graylog_es
    image: "elasticsearch:6.8.5"
    volumes:
      - "./es_data:/usr/share/elasticsearch/data"
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx5120m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - graylog_network

  graylog:
    restart: on-failure
    container_name: graylog_web
    image: "graylog/graylog:3.3"
    ports:
      - "9000:9000"
      - "5044:5044"
      - "12201:12201"
      - "12201:12201/udp"
      - "1514:1514"
      - "1514:1514/udp"
    volumes:
      - "./graylog_journal:/usr/share/graylog/data/journal"
    environment:
      - GRAYLOG_PASSWORD_SECRET=zscMb65...FxR9ag
      - GRAYLOG_ROOT_PASSWORD_SHA2=77e29e0f...557515f
      - GRAYLOG_HTTP_EXTERNAL_URI=http://11.22.33.44:9000/
      - GRAYLOG_TIMEZONE=Asia/Shanghai
      - GRAYLOG_ROOT_TIMEZONE=Asia/Shanghai
    networks:
      - graylog_network
    depends_on:
      - mongo
      - elasticsearch

networks:
  graylog_network:
    driver: bridge

Graylog’s GELF input can receive structured events from Docker containers using the gelf log driver.

# Docker run with GELF driver
docker run --rm=true \
  --log-driver=gelf \
  --log-opt gelf-address=udp://11.22.33.44:12201 \
  --log-opt tag=myapp \
  myapp:0.0.1

Docker‑Compose example for a Redis service sending logs to Graylog via GELF:

version: "3"
services:
  redis:
    restart: always
    image: redis
    container_name: "redis"
    logging:
      driver: gelf
      options:
        gelf-address: udp://11.22.33.44:12201
        tag: "redis"
  ...

Graylog UI Features

The Graylog web interface provides dashboards, search, stream management, alerting, and system configuration. Screenshots illustrate the main navigation, stream creation, pipeline editor, and index set settings.