Best ELK Deployment Architectures: Logstash vs Filebeat and Multiline Tips

1. Overview

ELK has become the most popular centralized log solution, consisting of Beats, Logstash, Elasticsearch, and Kibana, providing real‑time log collection, storage and visualization.

Filebeat : a lightweight data shipper that can replace Logstash on application servers, supports output to Kafka, Redis, etc.

Logstash : a heavier data collection engine with many plugins, capable of filtering, analyzing and formatting logs.

Elasticsearch : a distributed search engine built on Apache Lucene, offering centralized storage, analysis, search and aggregation.

Kibana : a web‑based visualization platform for exploring data in Elasticsearch.

2. Common ELK Deployment Architectures

2.1 Logstash as Log Collector

This classic architecture deploys a Logstash instance on each application server to collect logs, filter/format them, and send them to Elasticsearch for storage, with Kibana for visualization. The drawback is that Logstash consumes significant server resources, increasing load on the application host.

2.2 Filebeat as Log Collector

Similar to the first design, but replaces Logstash on the application side with the lightweight Filebeat. Filebeat usually works together with Logstash and is currently the most widely used architecture.

2.3 Adding a Caching Queue

Based on the second architecture, this design introduces a Redis (or other) message queue. Filebeat sends data to Redis, and Logstash reads from Redis, which helps handle large log volumes, improves data safety and balances load between Logstash and Elasticsearch.

2.4 Summary of the Three Architectures

Because of its resource consumption, the first architecture is rarely used today. The second architecture is the most popular. The third architecture is only needed when additional queueing is required for very high data volumes.

3. Issues and Solutions

Problem: How to merge multiline log entries?

Logs that span multiple lines need to be combined into a single event. The solution is to use the multiline plugin in Filebeat or Logstash.

Note: In the first architecture the multiline plugin is configured in Logstash; in the second architecture it is configured in Filebeat, without additional Logstash configuration.

1. Multiline configuration in Filebeat

pattern

: regular expression. negate: false (default) merges lines matching the pattern with the previous line; true does the opposite. match: after appends to the previous line’s end, before prepends to the previous line’s start.

pattern: ‘[‘ negate: true match: after

This configuration merges lines that do not match the pattern to the end of the previous line.

2. Multiline configuration in Logstash

In Logstash the what attribute set to previous corresponds to Filebeat’s after, and next corresponds to before. Example pattern: %{LOGLEVEL}\s*], where LOGLEVEL is a built‑in Logstash pattern.

Problem: How to replace Kibana’s @timestamp with the timestamp inside the log message?

By default Kibana shows the ingestion time. Use the grok filter together with the date plugin to extract and rewrite the timestamp.

Example: for a log line

[DEBUG][20170811 10:07:31,359][DefaultBeanDefinitionDocumentReader:106] Loading bean definitions

, define a custom pattern such as CUSTOMER_TIME %{YEAR}%{MONTHNUM}%{MONTHDAY}\s+%{TIME} and reference it in Logstash.

Alternatively, embed the custom pattern directly: (?<CUSTOMER_TIME>...).

Problem: How to view logs of a specific system module in Kibana?

Add a field that identifies the module, or create separate Elasticsearch indices per module.

Using the second architecture, add a log_from field in Filebeat and filter on it in Kibana.

Or configure separate indices based on document_type and set the index option in Logstash output, e.g., %{type} to create module‑specific indices.

4. Conclusion

This article introduced three ELK deployment architectures, highlighted that the Filebeat‑based design is currently the most popular, and provided practical solutions for multiline merging, timestamp correction, and module‑based filtering. ELK can also serve for distributed log querying, monitoring, and other use cases.

Source: https://my.oschina.net/feinik/blog/1580625