ELK Stack: Common Deployment Architectures, Configuration Tips, and Troubleshooting

ELK (Elasticsearch, Logstash, Kibana, and Beats) has become the most popular centralized logging solution, providing real‑time collection, storage, and visualization of logs.

Common ELK Deployment Architectures

1. Logstash as Log Collector

This classic architecture deploys a Logstash instance on each application server to collect logs, filter/format them, and forward to Elasticsearch for storage and Kibana for visualization. The main drawback is high resource consumption on the application servers.

2. Filebeat as Log Collector

Replacing Logstash with the lightweight Filebeat reduces server load. Filebeat is usually paired with Logstash for further processing. This is the most widely used architecture today.

3. Architecture with Cache Queue

Based on the second architecture, a Redis (or other message queue) is introduced. Filebeat sends data to Redis, and Logstash reads from Redis, improving data safety and balancing load under large‑scale log volumes.

Problems and Solutions

Multiline Log Merging

Logs that span multiple lines need to be merged. The solution is to use the multiline plugin in Filebeat or Logstash, depending on the deployment architecture.

pattern: '\['
negate: true
match: after

Replacing Kibana Timestamp with Log Timestamp

By default Kibana shows the ingestion time. Use the grok filter and date plugin in Logstash to extract the timestamp from the log message.

Filtering Logs by System Module

Add a field (e.g., log_from) to identify the source module, or create separate Elasticsearch indices per module. Then use Kibana filters or index patterns to view specific module logs.

Summary

The article presented three ELK deployment architectures, highlighted their use cases, and provided concrete solutions for common issues such as multiline merging, timestamp replacement, and module‑based log filtering. The Filebeat‑based architecture is currently the most popular, while the cache‑queue approach is useful for very high‑volume scenarios. ELK can also serve for application and server resource monitoring beyond simple log aggregation.