Mastering ELK: Compare Logstash vs Filebeat Architectures and Solve Common Logging Challenges

Overview

ELK is a widely used centralized logging stack composed of Beats, Logstash, Elasticsearch, and Kibana. It provides real‑time log collection, storage, and visualization.

Common Deployment Architectures

Logstash as Log Collector

Deploy a Logstash instance on each application server to collect, filter, format, and forward logs to Elasticsearch; Kibana visualizes the data. This architecture consumes significant resources on the application hosts.

Filebeat as Log Collector

Replace Logstash on the application side with the lightweight Filebeat. Filebeat forwards logs to Logstash (or directly to Elasticsearch) and is the most common deployment pattern.

Cache Queue Between Filebeat and Logstash

Insert a Redis (or other message‑queue) layer between Filebeat and Logstash to buffer data, improve reliability, and balance the load of Logstash and Elasticsearch under high‑throughput scenarios.

Architecture Summary

The Logstash‑only architecture is rarely used because of its resource consumption. The Filebeat‑plus‑Logstash architecture is the dominant choice. Adding a cache queue is optional and only required for specific high‑throughput or reliability requirements.

Typical Problems and Solutions

Multiline Log Merging

Logs that span multiple lines must be merged into a single event.

Filebeat Multiline Configuration

pattern : regular expression that identifies the start of a new log entry.

negate : false merges lines matching the pattern to the previous line; true merges non‑matching lines.

match : after appends the line to the previous line’s end; before prepends it.

<code>pattern: '\[' negate: true match: after</code>

This example appends lines that do **not** start with [ to the previous line.

Logstash Multiline Configuration

In Logstash use the multiline filter. The what option previous corresponds to Filebeat’s after, and next corresponds to before. A typical pattern is:

pattern => "%{LOGLEVEL}\s*]"
what => "previous"

Built‑in patterns such as LOGLEVEL are defined in the Logstash patterns repository: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

Replacing Kibana Time Field with Log Timestamp

Kibana shows the ingestion time by default. Use Logstash grok and date filters to extract the timestamp from the log message.

Example log line:

[DEBUG][20170811 10:07:31,359][DefaultBeanDefinitionDocumentReader:106] Loading bean definitions

Create a custom pattern file (e.g., customer_patterns) containing:

CUSTOMER_TIME %{YEAR}%{MONTHNUM}%{MONTHDAY}\s+%{TIME}

Reference the pattern in the Logstash filter:

Alternatively embed the pattern directly:

Filtering Logs by System Module

Add a custom field (e.g., log_from) to identify the originating module, then filter in Kibana, or create separate Elasticsearch indices per module.

Filebeat example adding log_from:

Logstash output example using document_type to route to different indices:

Conclusion

The three ELK deployment patterns—Logstash‑only, Filebeat‑plus‑Logstash, and Filebeat‑plus‑Redis‑plus‑Logstash—address different resource and reliability requirements. Multiline merging, timestamp extraction, and module‑level filtering are implemented via Filebeat or Logstash configuration, demonstrating ELK’s flexibility for log analysis and broader monitoring use cases.