Log Collection Architecture and Implementation Using Filebeat, Logstash, and Kafka

Log collection is a common requirement for gathering service logs and event logs for offline ETL or real‑time analysis, with many open‑source tools such as Flume, Logstash, and Filebeat available. In the 360 commercial environment, the primary scenarios involve collecting logs from the ad‑delivery engine and middleware services, where logs are first persisted locally before being collected.

Technology selection : Because the ad‑delivery engine is resource‑sensitive, Filebeat was chosen over JVM‑based tools for its lower resource consumption and suitable feature set. For log aggregation, Logstash was selected to complement Filebeat, as both belong to the Elastic stack and integrate easily.

Filebeat version 7.14 (stable) is used with Kafka SASL/SCRAM authentication, while Logstash also uses version 7.14. The current setup primarily uses Filebeat's log input and kafka output, leveraging Filebeat’s extensibility to support additional inputs and outputs.

Filebeat Overview

Filebeat relies on the libbeat library, which provides a publisher to connect inputs and a series of processors for event enrichment. libbeat implements retry and acknowledgment mechanisms to ensure log collection completeness.

Key configuration areas include processors (e.g., add_fields, rename, drop_event), inputs, modules for various open‑source components, and an HTTP endpoint for metrics.

Meta‑persistence mechanism : Filebeat writes metadata such as input type, source file path, offset, timestamps, and inode/device identifiers to data/registry/filebeat/log.json. This file enables tracking of completed, in‑progress, and failed log collections, and can be edited or deleted to trigger re‑collection.

Back‑pressure mechanism : When outputs (e.g., Kafka) continuously fail, Filebeat’s retry logic fills the internal queue; once the queue is full, input collection stops, providing natural back‑pressure and ensuring at‑least‑once delivery semantics.

Design Goals

Unified operations monitoring to reduce operational cost.

Support both real‑time and offline log consumption.

Pipeline with back‑pressure, at‑least‑once semantics, and log replay capability.

Cross‑IDC disaster recovery with dynamic agent configuration.

Design方案

One‑click agent deployment script and an agent manager for metrics reporting and dynamic configuration.

Kafka as the unified log collection destination.

Logstash as an aggregation layer to reduce pressure on Kafka when many agents are present.

Monitoring via Prometheus and Grafana by exposing Filebeat and Logstash metrics as Prometheus format.

Productization into the Ultron platform for project‑level management.

The overall architecture consists of four layers: agent layer (Filebeat + agent manager), aggregation layer (optional Logstash), DataBus layer (Kafka for unified transport), and heterogeneous transmission layer (custom Hamal2 framework for downstream processing).

Filebeat → Kafka (Direct)

For scenarios with few agents, Filebeat directly writes to Kafka using the Kafka output. The meta‑persistence and back‑pressure mechanisms ensure that if Kafka is unavailable, Filebeat stops collecting, preserving log integrity.

Example filebeat.yml configuration:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /root/test*.log
  fields:
    topic_name: test
- type: log
  enabled: true
  paths:
    - /root/test2*.log
  fields:
    topic_name: test2

output.kafka:
  version: 2.0.0
  hosts: ["xxx:9092", "xxx:9092", "xxx:9092"]
  topic: '%{[fields.topic_name]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: lz4
  max_message_bytes: 10000000
  sasl.mechanism: SCRAM-SHA-256
  username: xxx
  password: xxx
  worker: 1
  refresh_frequency: 5m
  codec.format:
    string: '%{[message]}'

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

http.enabled: true
http.host: your host
http.port: 5066

Filebeat → Logstash → Kafka

When many agents are present, an intermediate Logstash layer is added to offload connections from Kafka. Logstash’s persistent queues and back‑pressure mechanisms further guarantee at‑least‑once delivery.

Logstash persistent queue configuration:

queue.type: persisted
queue.max_bytes: 8gb
queue.max_events: 3000000
path.queue: /path/to/queuefile

Sample Filebeat configuration for Logstash output:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /root/test*.log
  fields:
    topic_name: test
    kafka_cluster: cluster1
- type: log
  enabled: true
  paths:
    - /root/test2*.log
  fields:
    topic_name: test2
    kafka_cluster: cluster2

output.logstash:
  hosts: ["logstash.k8s.domain:5044"]

http.enabled: true
http.host: your host
http.port: 5066

Corresponding Logstash pipeline configuration:

input {
  beats {
    port => "5044"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

output {
  stdout { codec => rubydebug { metadata => true } }
  if [fields][kafka_cluster] == "xxx" {
    kafka {
      codec => plain { format => '{ message:"%{message}