ELK Stack Overview: Elasticsearch, Logstash, Kibana, and Filebeat Architecture and Deployment Guide

The ELK stack (Elasticsearch, Logstash, Kibana, and Filebeat) is a distributed search and analytics engine built on Apache Lucene that offers high scalability, reliability, and easy management for real‑time storage, search, and analysis of large volumes of data.

Elasticsearch serves as the searchable data store; Logstash collects logs from various sources, filters, parses, and forwards them; Kibana visualizes the data stored in Elasticsearch; Filebeat is a lightweight log shipper that can send logs directly to Logstash or Elasticsearch.

Architecture options :

Architecture 1 – Multiple Logstash instances on each server for a fully distributed setup. This consumes significant CPU and memory on each node, so it is not recommended for most scenarios.

Architecture 2 – Introduces a message queue (Kafka, Redis, RabbitMQ, etc.) between Logstash instances. Logstash before the queue only forwards logs, while the downstream Logstash performs parsing, reducing per‑node resource usage.

Architecture 3 – Simplifies Architecture 2 by sending logs directly to the queue and letting a single Logstash instance consume and process them.

Architecture 4 – Replaces Logstash on each server with Beats (e.g., Filebeat). Beats are lightweight, consume negligible CPU/memory, and can automatically throttle when downstream load is high, making this the most efficient design.

Deployment prerequisites : Install JDK 8, download the appropriate packages for your OS, or use Docker images. After extracting the packages, start each component with the provided binaries.

Logstash command:

bin/logstash -f logstash.conf   # logstash.conf is a user‑created pipeline configuration file

Basic Logstash pipeline structure:

# input
input {
}
# filter
filter {
}
# output
output {
}

Elasticsearch start command: bin/elasticsearch If running as root is prohibited, create a dedicated user and adjust directory permissions:

groupadd elsearch
useradd elsearch -g elsearch -p elsearch
chown -R elsearch:elsearch elasticsearch

Kibana start command: bin/kibana Filebeat start command: filebeat -e -c filebeat.yml Key Filebeat configuration to ship all /var/log/*.log files to Logstash on port 5044:

filebeat.prospectors:
- input_type: log
  paths:
  - /var/log/*.log
output.logstash:
  hosts: ["localhost:5044"]

Logstash example configuration (Filebeat input, JSON codec, conditional type assignment, date parsing, and Elasticsearch output):

input {
  beats {
    port => 5044
    codec => "json"
  }
}
filter {
  if [logtype] {
    mutate { replace => { type => "%{logtype}" } }
  } else {
    mutate { replace => { type => "unknow" } }
  }
  date {
    match => [ "createTime", "yyyy-MM-dd HH:mm:ss" ]
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-%{type}-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

The configuration uses Filebeat as the Logstash input, parses incoming JSON logs, sets the type field based on the optional logtype attribute, normalizes timestamps, and indexes the data into Elasticsearch with a date‑based index name.

In production, Kibana provides visual dashboards for the indexed logs, as shown in the included screenshots.