ELK Stack (Elasticsearch, Logstash, Kibana) Overview, Architecture, Installation, and Configuration Guide (Version 7.7.0)

The article introduces the ELK stack (Elasticsearch, Logstash, Kibana) and its optional Filebeat component, focusing on version 7.7.0, and explains why centralized logging is essential for large distributed systems.

ELK Overview

Elasticsearch is a distributed search and analytics engine built on Lucene; Logstash is a flexible data‑processing pipeline; Kibana provides visualization; Filebeat is a lightweight shipper for forwarding logs to Elasticsearch or Logstash.

Why Use ELK

Centralized log collection, storage, and analysis improve troubleshooting, performance monitoring, and security auditing across dozens or hundreds of servers, overcoming the limitations of grep/awk on individual machines.

Core Log System Features

Collect: ingest logs from multiple sources.

Transport: parse, filter, and forward logs.

Store: persist log data.

Analyze: UI‑driven analytics.

Alert: error reporting and monitoring.

ELK Architecture Diagrams

Several deployment models are described:

Beats + Elasticsearch + Kibana (simple, entry‑level).

Beats + Logstash + Elasticsearch + Kibana (adds Logstash for buffering, richer processing, and reliability).

Beats + Cache/Message Queue + Logstash + Elasticsearch + Kibana (introduces Redis/Kafka/RabbitMQ for decoupling and scaling).

Filebeat Installation

Download the tarball, extract, and configure filebeat.yml (inputs, outputs, multiline handling). Example commands:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.0-linux-x86_64.tar.gz
 tar -xzvf filebeat-7.7.0-linux-x86_64.tar.gz

Start with ./filebeat -e.

Logstash Installation

Download, extract, and run a simple HelloWorld pipeline:

tar -zxvf logstash-7.7.0.tar.gz
 ./bin/logstash -e 'input { stdin { } } output { stdout {} }'

Configuration files define inputs, filters (e.g., grok for Apache logs), and outputs to Elasticsearch.

Elasticsearch Installation

Create a dedicated user, extract the tarball, and set up directories ( $ES_HOME, bin, conf, data, logs, etc.). Configure JVM options in jvm.options (e.g., -Xms2g, -Xmx2g, GC settings) and enable security with certificates generated via elasticsearch-certutil. Sample elasticsearch.yml includes cluster name, node roles, network host, discovery seeds, and X‑Pack security settings.

After starting Elasticsearch ( ./bin/elasticsearch -d), set built‑in passwords with ./bin/elasticsearch-setup-passwords interactive and verify access via http://<host>:9200/.

Kibana Installation

Extract the package, edit kibana.yml (server port, host, Elasticsearch hosts, credentials), and launch with ./bin/kibana. Access the UI at http://<host>:5601/ and log in using the Elasticsearch credentials.

Example Pipeline with Kafka

Filebeat is configured to output to a Kafka topic, Logstash consumes from Kafka, parses Apache logs with grok, and indexes into Elasticsearch. Sample snippets:

# filebeat.yml (output.kafka)
output.kafka:
  hosts: ["192.168.110.130:9092"]
  topic: 'filebeat_test'
  compression: gzip

# logstash/apache.conf
input { kafka { bootstrap_servers => "192.168.110.130:9092" topics => ["filebeat_test"] group_id => "test123" } }
filter { json { source => "message" } grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_field => "message" } }
output { elasticsearch { hosts => ["192.168.110.130:9200"] index => "test_kafka" user => "elastic" password => "${ES_PWD}" } stdout { codec => rubydebug } }

Running Logstash with ./bin/logstash -f conf.d/apache.conf ingests the data, which can then be visualized in Kibana.

The article concludes with screenshots of the Elasticsearch index and Kibana dashboards, confirming successful end‑to‑end log collection, processing, and visualization.