ELK Stack Common Deployment Architectures and Practical Solutions

Overview

ELK (Elasticsearch, Logstash, Kibana, Beats) is currently the most popular centralized logging solution, offering real‑time collection, storage, and visualization of logs. The article outlines typical ELK architectures and related problem‑solving techniques.

ELK Components

Filebeat : a lightweight data shipper that can replace Logstash on application servers and output to Kafka, Redis, etc.

Logstash : a more heavyweight data collection engine with many plugins for filtering, parsing, and formatting.

Elasticsearch : a distributed search engine built on Apache Lucene, providing centralized storage, analysis, and powerful search/aggregation.

Kibana : a web‑based visualization platform for exploring data stored in Elasticsearch.

Common ELK Deployment Architectures

2.1 Logstash as Log Collector

This classic architecture deploys a Logstash instance on each application server to collect, filter, and format logs before sending them to Elasticsearch for storage and Kibana for visualization. Drawbacks include high resource consumption on application servers.

2.2 Filebeat as Log Collector

This architecture replaces Logstash on the application side with Filebeat, a lightweight shipper that typically works together with Logstash. It is the most widely used deployment today.

2.3 Architecture with Cache Queue

Based on the Filebeat approach, this design introduces a Redis (or other message queue) between Filebeat and Logstash, buffering data to improve reliability and balance load under high‑volume scenarios.

2.4 Summary of the Three Architectures

Due to resource overhead, the pure Logstash collector is rarely used today. The Filebeat‑based deployment is the most common, while the cache‑queue architecture is only needed for specific high‑throughput requirements.

Problems and Solutions

Multiline Log Merging

Logs that span multiple lines need to be merged during ingestion. Use the multiline plugin in Filebeat or Logstash, depending on the deployment.

Multiline configuration in Filebeat

pattern: '\['
negate: true
match: after

Explanation: lines that do not match the pattern are appended to the previous line.

Multiline configuration in Logstash

Set what => "previous" (equivalent to Filebeat’s after) or what => "next" (equivalent to before). Use predefined patterns such as %{LOGLEVEL}\s*\] from the Logstash patterns repository.

Replacing Kibana Timestamp with Log Timestamp

By default Kibana shows the ingestion time. Use the grok filter to extract the original timestamp and the date filter to rewrite the @timestamp field.

Filtering Logs by System Module in Kibana

Add a field (e.g., log_from or document_type) to identify the source module, then filter in Kibana or create separate Elasticsearch indices per module.

Conclusion

The article presented three ELK deployment architectures, highlighted the advantages of the Filebeat‑based approach, and offered practical solutions for multiline merging, timestamp correction, and module‑based filtering, demonstrating ELK’s versatility for log analysis, monitoring, and beyond.