Centralized Log Collection with Filebeat and Graylog: Configuration, Deployment, and Integration Guide

When an organization runs many services across test and production environments, centralized log collection becomes essential. The article compares using Nginx for log exposure versus a dedicated log collection system like ELK, and recommends Graylog as a lightweight alternative that stores logs in Elasticsearch and uses MongoDB for configuration.

Filebeat Tool Introduction

① Filebeat log shipping service – Filebeat monitors specified log directories or files, reads new entries continuously, and forwards them to Elasticsearch, Logstash, or Graylog.

② Filebeat workflow – Upon starting, Filebeat launches prospectors to watch log paths; each discovered file is processed by a harvester that reads new lines and sends them to a spooler, which batches events before sending them to the configured output (e.g., Graylog).

③ Filebeat diagram – The tool is lighter than Logstash, making it suitable for machines with limited resources.

Filebeat Configuration File

The main configuration resides at /etc/filebeat/filebeat.yml for RPM/DEB installations. Inputs are defined in the inputs.d directory, allowing separate configurations per service.

# Configure input sources
filebeat.config.inputs:
  enabled: true
  path: ${path.config}/inputs.d/*.yml
# Enable JSON parsing if needed
# json.keys_under_root: true

# Load modules
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

# Output to Graylog (via Logstash input)
output.logstash:
  hosts: ["11.22.33.44:5500"]

processors:
  - add_host_metadata: ~
  - rename:
      fields:
        - from: "log"
          to: "message"
  - add_fields:
      target: ""
      fields:
        token: "0uxxxxaM-1111-2222-3333-VQZJxxxxxwgX "

A sample inputs.d YAML file shows how to collect specific log files, filter lines containing "WARNING" or "ERROR", and add tags:

# Log type definition
- type: log
  enabled: true
  paths:
    - /var/log/supervisor/app_escape_worker-stderr.log
    - /var/log/supervisor/app_escape_prod-stderr.log
  symlinks: true
  include_lines: ["WARNING", "ERROR"]
  tags: ["app", "escape", "test"]
  multiline.pattern: '^\[?[0-9]...{3}'
  multiline.negate: true
  multiline.match: after

Graylog Service Introduction

① Graylog log monitoring system – An open‑source platform similar to ELK but simpler to deploy, consisting of Elasticsearch (storage), MongoDB (configuration), and Graylog (web UI and API).

② Graylog workflow – Inputs collect logs, extractors transform fields, streams route events to specific index sets, and pipelines allow custom processing such as dropping debug messages.

Example pipeline rule to discard messages with level > 6:

rule "discard debug messages"
when
  to_long($message.level) > 6
then
  drop_message();
end

Graylog also supports Sidecar agents (Filebeat, NXLog, Winlogbeat) for lightweight log collection from Linux and Windows hosts.

Service Installation and Deployment

Deploy Filebeat via DEB/RPM, Docker, or source compilation. Example DEB installation:

# Ubuntu (deb)
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.1-amd64.deb
sudo dpkg -i filebeat-7.8.1-amd64.deb
sudo systemctl enable filebeat
sudo service filebeat start

Deploy Graylog using Docker‑Compose. First generate a 16‑character password_secret and a SHA‑256 hash for the admin password, then create a docker-compose.yml file:

version: "3"
services:
  mongo:
    restart: on-failure
    container_name: graylog_mongo
    image: "mongo:3"
    volumes:
      - "./mongodb:/data/db"
    networks:
      - graylog_network

  elasticsearch:
    restart: on-failure
    container_name: graylog_es
    image: "elasticsearch:6.8.5"
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - ES_JAVA_OPTS=-Xms512m -Xmx5120m
    volumes:
      - "./es_data:/usr/share/elasticsearch/data"
    networks:
      - graylog_network

  graylog:
    restart: on-failure
    container_name: graylog_web
    image: "graylog/graylog:3.3"
    ports:
      - 9000:9000   # Web UI
      - 5044:5044   # Filebeat input
      - 12201:12201 # GELF TCP
      - 12201:12201/udp
      - 1514:1514   # Syslog TCP
      - 1514:1514/udp
    volumes:
      - "./graylog_journal:/usr/share/graylog/data/journal"
    environment:
      - GRAYLOG_PASSWORD_SECRET=zscMb65...FxR9ag
      - GRAYLOG_ROOT_PASSWORD_SHA2=77e29e0f...557515f
      - GRAYLOG_HTTP_EXTERNAL_URI=http://11.22.33.44:9000/
      - GRAYLOG_TIMEZONE=Asia/Shanghai
    networks:
      - graylog_network
    depends_on:
      - mongo
      - elasticsearch

networks:
  graylog_network:
    driver: bridge

After starting the stack, access http:// :9000 to log in to Graylog.

Spring Boot Integration with Graylog

Add the logback-gelf dependency:

<dependency>
  <groupId>de.siegmar</groupId>
  <artifactId>logback-gelf</artifactId>
  <version>3.0.0</version>
</dependency>

Create logback.xml with a GELF UDP appender pointing to the Graylog host and port (e.g., 12201 ), enabling compression and configuring fields such as app_name :

<appender name="GELF" class="de.siegmar.logbackgelf.GelfUdpAppender">
  <graylogHost>ip</graylogHost>
  <graylogPort>12201</graylogPort>
  <maxChunkSize>508</maxChunkSize>
  <useCompression>true</useCompression>
  <encoder class="de.siegmar.logbackgelf.GelfEncoder">
    <includeRawMessage>false</includeRawMessage>
    <includeMarker>true</includeMarker>
    <includeMdcData>true</includeMdcData>
    <includeLevelName>true</includeLevelName>
    <staticField>app_name:austin</staticField>
  </encoder>
</appender>

After restarting the Spring Boot application, logs appear in Graylog’s Search view, where you can query by fields, levels, or keywords using the built‑in search syntax.

The article concludes with a brief note that more advanced configuration options are available in the component’s GitHub documentation.