Centralized Log Collection with Filebeat and Graylog: Installation, Configuration, and Usage

When a company runs many services across test and production environments, collecting logs becomes a critical requirement. The article compares two approaches—exposing logs through Nginx versus using a dedicated log‑collection stack such as ELK—and concludes that Graylog, which stores data in Elasticsearch and caches in MongoDB, offers a simpler yet effective solution.

Filebeat Overview

Filebeat is a lightweight log shipper that monitors specified log files or directories, reads new entries, and forwards them to destinations like Elasticsearch, Logstash, or Graylog. It starts one or more prospectors to watch paths, spawns a harvester for each file, and sends harvested events to a spooler before delivering them to the configured output.

Because Filebeat is more lightweight than Logstash, it is recommended for environments with modest resource constraints.

Filebeat Configuration Example

# Configure input sources (inputs.d directory)
filebeat.config.inputs:
  enabled: true
  path: ${path.config}/inputs.d/*.yml
# Enable JSON parsing if needed
# json.keys_under_root: true
# Load modules
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
# Output to Graylog (via Logstash input)
output.logstash:
  hosts: ["11.22.33.44:5500"]
processors:
  - add_host_metadata: ~
  - rename:
      fields:
        - from: "log"
          to: "message"
  - add_fields:
      target: ""
      fields:
        token: "0uxxxxaM-1111-2222-3333-VQZJxxxxxwgX "

An example inputs.d file shows how to collect logs from specific paths, filter by keywords, add tags, and handle multiline patterns:

# Log type definition
- type: log
  enabled: true
  paths:
    - /var/log/supervisor/app_escape_worker-stderr.log
    - /var/log/supervisor/app_escape_prod-stderr.log
  symlinks: true
  include_lines: ["WARNING", "ERROR"]
  tags: ["app", "escape", "test"]
  multiline.pattern: '^\[?[0-9]...{3}'
  multiline.negate: true
  multiline.match: after
- type: log
  enabled: true
  ...

Graylog Service Overview

Graylog is an open‑source log aggregation, analysis, and alerting platform. Its core components are Elasticsearch (for storage and search), MongoDB (for configuration), and the Graylog server (web UI and API). The system uses Inputs to receive logs, Extractors to parse fields, Streams to route logs, and Index Sets to store them. Pipelines allow custom processing, such as discarding messages with a level greater than 6:

rule "discard debug messages" when
  to_long($message.level) > 6
then
  drop_message();
end

Sidecar agents (available for Linux and Windows) can pull configuration from Graylog and forward logs from Filebeat, Winlogbeat, or NXLog, supporting various output formats like GELF, JSON, and CEF.

Installation & Deployment

Filebeat can be installed via DEB/RPM packages, compiled from source, or run as a Docker container. Example commands for Ubuntu:

# Ubuntu (deb)
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.1-amd64.deb
sudo dpkg -i filebeat-7.8.1-amd64.deb
sudo systemctl enable filebeat
sudo service filebeat start

Docker deployment example:

docker run -d --name=filebeat --user=root \
  --volume "./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume "/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume "/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker.elastic.co/beats/filebeat:7.8.1 filebeat -e -strict.perms=false \
  -E output.elasticsearch.hosts=["elasticsearch:9200"]

Graylog is typically deployed with Docker‑Compose. After generating a 16‑character password_secret and a SHA‑256 hash of the admin password, the following docker‑compose.yml defines MongoDB, Elasticsearch, and Graylog services with appropriate ports (9000 for UI, 5044 for Filebeat, 12201 for GELF, 1514 for Syslog) and environment variables:

version: "3"
services:
  mongo:
    restart: on-failure
    container_name: graylog_mongo
    image: "mongo:3"
    volumes:
      - "./mongodb:/data/db"
    networks:
      - graylog_network
  elasticsearch:
    restart: on-failure
    container_name: graylog_es
    image: "elasticsearch:6.8.5"
    volumes:
      - "./es_data:/usr/share/elasticsearch/data"
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - ES_JAVA_OPTS=-Xms512m -Xmx5120m
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 12g
    networks:
      - graylog_network
  graylog:
    restart: on-failure
    container_name: graylog_web
    image: "graylog/graylog:3.3"
    ports:
      - 9000:9000
      - 5044:5044
      - 12201:12201
      - 12201:12201/udp
      - 1514:1514
      - 1514:1514/udp
    volumes:
      - "./graylog_journal:/usr/share/graylog/data/journal"
    environment:
      - GRAYLOG_PASSWORD_SECRET=zscMb65...FxR9ag
      - GRAYLOG_ROOT_PASSWORD_SHA2=77e29e0f...557515f
      - GRAYLOG_HTTP_EXTERNAL_URI=http://11.22.33.44:9000/
      - GRAYLOG_TIMEZONE=Asia/Shanghai
      - GRAYLOG_ROOT_TIMEZONE=Asia/Shanghai
    networks:
      - graylog
    depends_on:
      - mongo
      - elasticsearch
networks:
  graylog_network:
    driver: bridge

GELF (Graylog Extended Log Format) inputs accept structured events and can be used directly from Docker containers by setting the Docker log driver to gelf and specifying the Graylog address.

# Docker container with GELF logging
docker run --rm=true \
  --log-driver=gelf \
  --log-opt gelf-address=udp://11.22.33.44:12201 \
  --log-opt tag=myapp \
  myapp:0.0.1

Spring Boot Integration

Add the Logback GELF appender dependency:

<dependency>
  <groupId>de.siegmar</groupId>
  <artifactId>logback-gelf</artifactId>
  <version>3.0.0</version>
</dependency>

Create logback.xml with a GELF appender pointing to the Graylog host and port, enable compression, and define static fields such as the application name:

<appender name="GELF" class="de.siegmar.logbackgelf.GelfUdpAppender">
  <graylogHost>ip</graylogHost>
  <graylogPort>12201</graylogPort>
  <maxChunkSize>508</maxChunkSize>
  <useCompression>true</useCompression>
  <encoder class="de.siegmar.logbackgelf.GelfEncoder">
    <includeRawMessage>false</includeRawMessage>
    <includeMarker>true</includeMarker>
    <includeMdcData>true</includeMdcData>
    <includeLevelName>true</includeLevelName>
    <shortPatternLayout class="ch.qos.logback.classic.PatternLayout">
      <pattern>%m%nopex</pattern>
    </shortPatternLayout>
    <fullPatternLayout class="ch.qos.logback.classic.PatternLayout">
      <pattern>%d - [%thread] %-5level %logger{35} - %msg%n</pattern>
    </fullPatternLayout>
    <staticField>app_name:austin</staticField>
  </encoder>
</appender>

After updating the IP address, restart the Spring Boot application and the logs appear in Graylog's Search UI. The article also lists common Graylog search syntaxes (wildcard, exact match, field queries, boolean combinations).

Overall, the guide provides a complete end‑to‑end solution for centralized log collection, processing, and visualization using Filebeat and Graylog, suitable for both on‑premise and containerized deployments.