Mastering ELK: Choose the Right Log Architecture and Solve Common Issues

1. Overview

ELK has become the most popular centralized logging solution, composed of Beats, Logstash, Elasticsearch, and Kibana, providing real‑time log collection, storage, and visualization.

Filebeat: lightweight data shipper that can replace Logstash on application servers and supports output to Kafka, Redis, etc.

Logstash: heavier data collection engine with many plugins, supporting filtering, parsing, and formatting of logs.

Elasticsearch: distributed search engine based on Apache Lucene, offering centralized storage, analysis, search, and aggregation.

Kibana: web UI for visualizing Elasticsearch data with rich charting capabilities.

2. Common ELK Deployment Architectures

2.1 Logstash as Log Collector

This classic architecture deploys a Logstash instance on each application server, which collects, filters, formats logs and forwards them to Elasticsearch; Kibana visualizes them. Drawback: Logstash consumes significant resources, increasing server load.

2.2 Filebeat as Log Collector

Same as above but replaces Logstash with Filebeat on the application side. Filebeat is lightweight, usually paired with Logstash, and is the most common deployment today.

2.3 Adding a Cache Queue

Based on the second architecture, a Redis (or other) queue is inserted between Filebeat and Logstash. Filebeat sends data to the queue; Logstash reads from it. This mitigates load pressure and improves data safety for large volumes.

2.4 Summary of the Three Architectures

Because of its resource consumption, the first architecture is rarely used. The second (Filebeat + Logstash) is the most popular. The third with a message queue is only needed for very high‑volume scenarios; Logstash can signal Filebeat to throttle when busy.

3. Problems and Solutions

Multiline Log Merging

Issue: Logs that span multiple lines need to be merged.

Solution: Use the multiline plugin in Filebeat or Logstash.

Configuration differs per architecture: in the Logstash‑centric setup configure multiline in Logstash; in the Filebeat‑centric setup configure it in Filebeat.

Filebeat multiline configuration example:

pattern: '['<br/>negate: true<br/>match: after

Explanation: lines not matching the pattern are appended to the previous line.

Replacing Kibana’s @timestamp with Log Time

Issue: Kibana shows the ingestion time instead of the log’s original timestamp.

Solution: Use the grok filter to extract the timestamp and the date filter to convert it.

Example Logstash filter snippet (illustrative):

Define a custom pattern, e.g.,

CUSTOMER_TIME %{YEAR}%{MONTHNUM}%{MONTHDAY}\s+%{TIME}

, and reference it in the Logstash configuration.

Pattern file illustration:

Filtering Logs by System Module in Kibana

Issue: Logs from different system modules are mixed; need to view a specific module.

Solution: Add a field (e.g.,

log_from

) to identify the module or create separate Elasticsearch indices per module.

Example Filebeat configuration adding

log_from

field:

Logstash output configuration using the document type to build index names:

4. Conclusion

The article presented three ELK deployment architectures, highlighted their trade‑offs, and provided practical solutions for multiline merging, timestamp correction, and module‑based filtering. The Filebeat‑based architecture is currently the most widely adopted, and ELK can also serve monitoring and resource‑management scenarios.