ELK Stack Deployment Architectures and Common Configuration Solutions

The ELK stack (Elasticsearch, Logstash, Kibana, and Beats) is a popular centralized logging solution that provides real‑time collection, storage, and visualization of logs. Filebeat is a lightweight data shipper that can replace Logstash on application servers and can output to Kafka, Redis, etc.

2.1 Logstash as log collector – Each application server runs a Logstash instance to collect, filter, and format logs before sending them to Elasticsearch for storage and Kibana for visualization. This approach consumes more resources on the application servers.

2.2 Filebeat as log collector – Replaces Logstash with the lightweight Filebeat shipper, usually paired with Logstash for further processing. This is the most commonly used architecture because of its low resource footprint.

2.3 Architecture with a cache queue – Builds on the Filebeat‑based design by inserting a message queue such as Kafka between Filebeat and Logstash. The queue balances load and improves reliability when handling large volumes of log data.

Summary of the three architectures – The Logstash‑only design is rarely used today due to its high resource consumption. The Filebeat‑based architecture is the prevailing choice, while the Kafka‑enhanced version is optional and only needed for specific high‑throughput scenarios.

Problem 1: Multiline log merging – Logs that span multiple lines need to be combined into a single event. The solution is to use the multiline plugin in Filebeat or Logstash. Example Filebeat configuration:

filebeat.prospectors:
- paths:
  - /home/project/elk/logs/test.log
  input_type: log
  multiline:
    pattern: '^\['
    negate: true
    match: after
output:
  logstash:
    hosts: ["localhost:5044"]

Key multiline options:

pattern – regular expression to match lines

negate – true merges lines that do NOT match the pattern

match – "after" appends to the previous line, "before" prepends

Logstash multiline configuration example:

input {
  beats { port => 5044 }
}
filter {
  multiline {
    pattern => "%{LOGLEVEL}\s*]"
    negate => true
    what => "previous"
  }
}
output {
  elasticsearch { hosts => "localhost:9200" }
}

Problem 2: Replace Kibana time field with log timestamp – By default Kibana shows the ingestion time. Use the grok and date filters to extract the timestamp from the log message and overwrite @timestamp:

filter {
  multiline { pattern => "%{LOGLEVEL}\s*]%{YEAR}%{MONTHNUM}%{MONTHDAY}\s+%{TIME}]" negate => true what => "previous" }
  grok { match => ["message", "(?<customer_time>%{YEAR}%{MONTHNUM}%{MONTHDAY}\s+%{TIME})"] }
  date { match => ["customer_time", "yyyyMMdd HH:mm:ss,SSS"] target => "@timestamp" }
}

Alternatively, define a custom pattern file (e.g., CUSTOMER_TIME %{YEAR}%{MONTHNUM}%{MONTHDAY}\s+%{TIME}) and reference it in the grok filter.

Problem 3: View logs of specific system modules in Kibana – Add a field (e.g., log_from) or use different document types/indexes to distinguish modules. Example Filebeat configuration adding log_from:

filebeat.prospectors:
- paths:
  - /home/project/elk/logs/account.log
  input_type: log
  multiline:
    pattern: '^\['
    negate: true
    match: after
  fields:
    log_from: account
- paths:
  - /home/project/elk/logs/customer.log
  input_type: log
  multiline:
    pattern: '^\['
    negate: true
    match: after
  fields:
    log_from: customer
output:
  logstash:
    hosts: ["localhost:5044"]

Or set document_type and configure Logstash output to create separate Elasticsearch indexes:

output {
  elasticsearch { hosts => "localhost:9200" index => "%{type}" }
}

Conclusion – The article reviews three ELK deployment architectures, highlights the advantages of the Filebeat‑based approach, and provides concrete configuration snippets to solve common logging challenges such as multiline merging, timestamp handling, and module‑level filtering.