ELK Stack Common Deployment Architectures and Practical Solutions

The ELK stack (Beats, Logstash, Elasticsearch, Kibana) is a popular centralized logging solution. This article explains the role of each component and presents three common deployment architectures:

Logstash as collector : Deploy Logstash on each application server to collect, filter, and forward logs to Elasticsearch; resource‑intensive.

Filebeat as collector : Use the lightweight Filebeat agent on application servers, often together with Logstash; the most widely used setup.

Architecture with a caching queue : Insert a Kafka (or other) queue between Filebeat and Logstash to handle high‑volume log streams and balance load.

The article then discusses three practical problems and their solutions:

1. Multiline log merging

Logs that span multiple lines need to be combined using the multiline plugin in Filebeat or Logstash. Configuration differs based on the chosen architecture.

filebeat.prospectors:
  -
    paths:
      - /home/project/elk/logs/test.log
    input_type: log
    multiline:
      pattern: '^\['
      negate: true
      match: after
output:
  logstash:
    hosts: ["localhost:5044"]

Key parameters: pattern: regular expression to identify the start of a new log entry. negate: when true, lines not matching the pattern are merged with the previous line. match (or what in Logstash): after merges to the end of the previous line; before merges to the beginning.

2. Replacing Kibana’s @timestamp with the log’s own timestamp

Use the grok filter to extract the timestamp from the log message and the date filter to set @timestamp accordingly.

filter {
  grok {
    match => ["message", "(?<customer_time>%{YEAR}%{MONTHNUM}%{MONTHDAY}\s+%{TIME})"]
  }
  date {
    match => ["customer_time", "yyyyMMdd HH:mm:ss,SSS"]
    target => "@timestamp"
  }
}

3. Viewing logs of specific system modules in Kibana

Add a custom field (e.g., log_from) or use different document types/indexes to distinguish modules, then filter in Kibana based on that field.

filebeat.prospectors:
  -
    paths:
      - /home/project/elk/logs/account.log
    input_type: log
    multiline:
      pattern: '^\['
      negate: true
      match: after
    fields:
      log_from: account
  -
    paths:
      - /home/project/elk/logs/customer.log
    input_type: log
    multiline:
      pattern: '^\['
      negate: true
      match: after
    fields:
      log_from: customer
output:
  logstash:
    hosts: ["localhost:5044"]

In Logstash output, the index can be set dynamically using the document type:

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "%{type}"
  }
}

In summary, the second architecture (Filebeat + Logstash) is currently the most popular for ELK deployments, and the provided configurations address common challenges such as multiline logs, correct timestamp handling, and module‑level log segregation.