Scaling Logstash Architecture: From Simple Setup to High‑Availability with Filters, Filebeat, ES Cluster, and Message Queues

Current Environment Logstash version 5.2 is used. The author reflects on earlier advice to add a message queue to the ELK+Filebeat logging center and discusses the need for extensible architecture as business requirements grow.

Overview As Logstash use cases become more complex, the architecture must evolve. The article walks through incremental enhancements, starting from a simple setup that writes data to an Elasticsearch (ES) cluster, and expands to more sophisticated designs.

Minimal Architecture The simplest deployment consists of a single Logstash instance directly connected to a single ES node, using an INPUT plugin to collect data and an OUTPUT plugin to write to ES. The OUTPUT defaults to stdout, while FILTER is optional.

Introducing Filters Filters parse unstructured log data, extract useful fields, and enrich records (e.g., geolocation from IP, Grok pattern parsing). Adding filters impacts performance; heavy regex processing can be parallelized using the -w option (e.g., bin/logstash -w 8 to run eight filter threads).

Introducing Filebeat Filebeat, a lightweight Go‑based log shipper, forwards logs to Logstash via the Beats protocol. Using the Beats input plugin reduces the resource load on Logstash instances, making the architecture suitable for resource‑constrained environments.

Connecting to an ES Cluster Instead of a single ES node, Logstash should communicate with an ES cluster over HTTP. Data is sent as JSON via the REST API, optionally secured with X‑Pack Security (SSL or HTTP basic). Multiple ES endpoints provide load balancing and high availability. Direct connections to master nodes should be avoided; use client or data nodes instead.

Using a Message Queue for Throughput Peaks When Logstash ingestion outpaces ES write capacity, a message queue (e.g., Redis, Kafka, RabbitMQ) can buffer data, preventing bottlenecks and data loss. The pipeline splits into two stages: a transport stage that pushes logs into the queue, and a storage stage that consumes from the queue, applies filters, and writes to ES.

Multi‑Connection High Availability Deploy a load balancer in front of a Logstash cluster so that if one Logstash instance fails, others continue processing. Each instance should expose multiple INPUT channels (e.g., RSS, file) to avoid single‑point failures; adding more Logstash nodes enables horizontal scaling.

Logstash Expansion Checklist

INPUT layer: appropriate input plugins for each data source.

Message queue: buffer and provide fault tolerance.

FILTER layer: parse and enrich data.

Indexing layer: write processed data to ES.

Each layer can be scaled by adding compute resources. Monitor performance regularly; if input becomes a bottleneck, increase queue capacity, or add more Logstash output instances to boost ES write throughput.