Mastering ELK: Deploy Architectures, Multiline Logs, and Kibana Tips

Overview

ELK is the most popular centralized log solution, composed of Beats, Logstash, Elasticsearch, and Kibana, providing real‑time collection, storage, and visualization.

Filebeat : lightweight data shipper that can replace Logstash on application servers; supports output to Kafka, Redis, etc.

Logstash : heavier data collector with many plugins, capable of filtering, parsing, and formatting logs.

Elasticsearch : distributed search engine based on Apache Lucene, offering centralized storage, analysis, search, and aggregation.

Kibana : web UI for visualizing Elasticsearch data with rich charts.

Common ELK Deployment Architectures

2.1 Logstash as Log Collector

This classic architecture deploys a Logstash instance on each application server, which collects, filters, formats logs and forwards them to Elasticsearch; Kibana visualizes them. Drawback: Logstash consumes considerable resources, increasing server load.

2.2 Filebeat as Log Collector

Same as above but replaces Logstash on the application side with Filebeat, which is lightweight. Filebeat is usually paired with Logstash and is the most common deployment today.

2.3 Adding a Cache Queue

Based on the second architecture, a Redis (or other) queue is inserted between Filebeat and Logstash. Filebeat sends data to Redis; Logstash reads from Redis. This solves high‑volume log collection, balances load, and improves data safety.

2.4 Summary of the Three Architectures

Because of its resource consumption, the first architecture is rarely used. The second (Filebeat + Logstash) is the most popular. The third adds a message queue only when needed for very large data volumes; Logstash can signal Filebeat to throttle when busy.

Issues and Solutions

Problem: How to merge multiline log events?

Logs that span multiple lines need to be combined. Solution: use the multiline plugin in Filebeat or Logstash.

Configuration differs by architecture: in the first architecture configure multiline in Logstash; in the second configure it in Filebeat.

1. Multiline configuration in Filebeat

pattern: regular expression

negate: false (default) merges lines matching pattern with the previous line; true does the opposite

match: after (append to previous line) or before (prepend)

pattern: '[' negate: true match: after

This merges lines that do not match the pattern to the end of the previous line.

2. Multiline configuration in Logstash

In Logstash,

what

= previous corresponds to Filebeat’s after, and

what

= next corresponds to before.

Example pattern:

%{LOGLEVEL}\s*]

using the built‑in LOGLEVEL pattern.

Problem: How to replace Kibana’s @timestamp with the timestamp inside the log message?

Solution: use the

grok

filter together with the

date

plugin in Logstash.

Define a custom pattern, e.g.

CUSTOMER_TIME %{YEAR}%{MONTHNUM}%{MONTHDAY}\s+%{TIME}

, then reference it in the Logstash configuration.

Problem: How to view logs of a specific system module in Kibana?

Solution: add a field that identifies the module or create separate Elasticsearch indices per module.

Example: add a

log_from

field in Filebeat and filter on it in Kibana, or use

document_type

to route logs to different indices.

Update Logstash output to set

index => "%{type}"

so each

document_type

creates its own index.

Conclusion

The article presented three ELK deployment architectures, highlighted their trade‑offs, and provided practical solutions for multiline merging, timestamp replacement, and module‑level filtering in Kibana. The Filebeat + Logstash architecture is currently the most widely adopted.