Mastering ELK: Choose the Right Log Architecture and Solve Common Issues

Overview

ELK has become the most popular centralized logging solution. It consists of Beats, Logstash, Elasticsearch, and Kibana, which together provide real‑time log collection, storage, and visualization.

Filebeat is a lightweight data shipper that can replace Logstash on application servers and can output to Kafka, Redis, etc.

Logstash is a heavier data collector with many plugins, supporting rich data sources and allowing filtering, analysis, and formatting of logs.

Elasticsearch is a distributed search engine based on Apache Lucene, offering clustered storage, analysis, and powerful search/aggregation.

Kibana is the visualization layer that lets users explore Elasticsearch data through rich charts.

Common ELK Deployment Architectures

2.1 Logstash as Log Collector

Each application server runs a Logstash instance that collects, filters, and formats logs before sending them to Elasticsearch. This approach is resource‑intensive and adds load to the application servers.

2.2 Filebeat as Log Collector

Filebeat replaces Logstash on the application side. It is lightweight and usually works together with Logstash, making it the most widely used architecture.

2.3 Adding a Cache Queue (Kafka)

On top of the Filebeat‑Logstash setup, a Kafka queue is introduced. Filebeat sends data to Kafka, and Logstash reads from Kafka, improving data safety and balancing load under high‑volume scenarios.

2.4 Summary of the Three Architectures

The first architecture is rarely used due to its resource consumption. The second architecture is the current default. The third architecture is optional and only needed when handling very large log volumes.

Problems and Solutions

Multiline Log Merging

Logs that span multiple lines need to be merged. Use the multiline plugin in Filebeat or Logstash.

Filebeat configuration example:

filebeat.prospectors:
- paths:
  - /home/project/elk/logs/test.log
  input_type: log
  multiline:
    pattern: '^\['
    negate: true
    match: after
output:
  logstash:
    hosts: ["localhost:5044"]

Logstash configuration example:

input {
  beats {
    port => 5044
  }
}
filter {
  multiline {
    pattern => "%{LOGLEVEL}\s*\]"
    negate => true
    what => "previous"
  }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
  }
}

Replace Kibana Time Field with Log Timestamp

Use the grok filter to extract the timestamp and the date filter to replace @timestamp.

filter {
  grok {
    match => ["message", "(?<customer_time>%{YEAR}%{MONTHNUM}%{MONTHDAY}\s+%{TIME})"]
  }
  date {
    match => ["customer_time", "yyyyMMdd HH:mm:ss,SSS"]
    target => "@timestamp"
  }
}

Filter Logs by System Module

Add a field (e.g., log_from) in Filebeat to identify the source module, then filter in Kibana.

filebeat.prospectors:
- paths:
  - /home/project/elk/logs/account.log
  input_type: log
  multiline:
    pattern: '^\['
    negate: true
    match: after
  fields:
    log_from: account
- paths:
  - /home/project/elk/logs/customer.log
  input_type: log
  multiline:
    pattern: '^\['
    negate: true
    match: after
  fields:
    log_from: customer
output:
  logstash:
    hosts: ["localhost:5044"]

Alternatively, set document_type and use it to create separate Elasticsearch indices:

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "%{type}"
  }
}

Conclusion

The article introduced three ELK deployment architectures, highlighted that the Filebeat‑based architecture is currently the most popular, and provided solutions for multiline merging, timestamp correction, and module‑based filtering. ELK is not only suitable for centralized log search but also for application and server monitoring.