Comprehensive Guide to ELK Stack (Elasticsearch, Logstash, Kibana) Installation, Configuration, and Architecture

This article introduces the ELK stack (Elasticsearch, Logstash, Kibana) and Beats (Filebeat) as a unified solution for centralized log collection, processing, storage, and visualization, using version 7.7.0.

ELK Overview

ELK consists of three open‑source projects: Filebeat for lightweight log shipping, Logstash for flexible data pipelines, Elasticsearch as a distributed search and analytics engine, and Kibana for visualizing data stored in Elasticsearch.

Why Use ELK

Centralized log management simplifies troubleshooting across many servers, enables real‑time analysis, and supports scaling, high availability, and secure transmission.

Core Log System Features

Collect: ingest logs from multiple sources.

Transport: parse, filter, and forward logs.

Store: persist log data.

Analyze: UI‑driven analysis with Kibana.

Alert: error reporting and monitoring.

Typical Architectures

1. Beats + Elasticsearch + Kibana (simple, entry‑level).

2. Beats + Logstash + Elasticsearch + Kibana (adds Logstash for richer processing).

3. Beats + Cache/Message Queue (e.g., Kafka) + Logstash + Elasticsearch + Kibana (adds buffering and decoupling).

Installation Steps

Filebeat

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.0-linux-x86_64.tar.gz
 tar -xzvf filebeat-7.7.0-linux-x86_64.tar.gz

Configure filebeat.yml (inputs, output to Kafka or Elasticsearch) and start with ./filebeat -e.

Logstash

curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-7.7.0.tar.gz
 tar -zxvf logstash-7.7.0.tar.gz

Create conf.d/apache.conf with Kafka input, JSON and Grok filters, and Elasticsearch output, then run: nohup ./bin/logstash -f conf.d/apache.conf & Elasticsearch

curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.7.0-linux-x86_64.tar.gz
 tar -zxvf elasticsearch-7.7.0-linux-x86_64.tar.gz
 ln -s elasticsearch-7.7.0 es

Set system limits (ulimit, vm.max_map_count, swappiness), create elasticsearch.yml with cluster settings, enable X‑pack security, and start with ./bin/elasticsearch -d. Generate certificates using

./bin/elasticsearch-certutil ca -out config/elastic-certificates.p12 -pass "password"

and configure TLS in elasticsearch.yml.

Kibana

curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-7.7.0-linux-x86_64.tar.gz
 tar -zxvf kibana-7.7.0-linux-x86_64.tar.gz

Edit kibana.yml (server.host, elasticsearch.hosts, credentials) and start with ./bin/kibana. Access via http://<host>:5601 and log in with the Elasticsearch user.

System Tuning Commands

ulimit -n 65535
 echo "* soft nofile 65535" >> /etc/security/limits.conf
 echo "* hard nofile 65535" >> /etc/security/limits.conf
 swapoff -a
 sysctl -w vm.swappiness=1
 sysctl -w vm.max_map_count=262144

Verification

Check cluster health, create indices, and view data in Kibana dashboards. Install the Elasticsearch Head plugin via Node.js or Chrome extension for additional UI inspection.

References

Official ELK documentation.

Sample log file: logstash-tutorial.log.

Related articles on Filebeat and Logstash usage.