ELK Stack Overview, Architecture, Installation and Configuration Guide (Version 7.7.0)

This article primarily introduces the composition, principles, and practical usage of the ELK stack (Elasticsearch, Logstash, Kibana, and Filebeat) based on version 7.7.0.

ELK Introduction

ELK stands for the three open‑source frameworks Elasticsearch, Logstash, and Kibana (collectively known as the Elastic Stack). Filebeat, a lightweight shipper from the Beats family, can replace Logstash for data collection.

Filebeat forwards and centralizes log data, monitors specified log files, reads new content, and forwards events to Elasticsearch or Logstash.

Logstash is a free, open‑source server‑side data processing pipeline that can ingest data from multiple sources, transform it, and ship it to your preferred storage.

Elasticsearch is the distributed search and analytics engine at the core of the Elastic Stack, providing near‑real‑time search for structured and unstructured data.

Kibana is an open‑source analytics and visualization platform for Elasticsearch, offering dashboards and a web UI for log analysis.

Why Use ELK

Logs (system, application, security) help operators understand server health, diagnose errors, and monitor performance. Traditional tools like grep and awk become inefficient when dealing with logs across dozens or hundreds of machines, making a centralized solution essential.

A complete log system should provide:

Collection: ingest logs from diverse sources.

Transport: reliably parse, filter, and forward logs.

Storage: persist log data.

Analysis: UI‑driven exploration.

Alerting: error reporting and monitoring.

ELK Architecture Analysis

Three main deployment patterns are described:

Beats + Elasticsearch + Kibana – a simple entry‑level setup using Filebeat as the shipper.

Beats + Logstash + Elasticsearch + Kibana – adds Logstash for richer processing, buffering, and protocol support.

Beats + Cache/MQ (Redis, Kafka, RabbitMQ) + Logstash + Elasticsearch + Kibana – introduces a message queue for decoupling and scaling.

ELK Deployment

Download URLs for each component are provided (official Elastic site and Chinese mirrors). Installation uses tar.gz packages on Linux.

Filebeat Installation

Download and extract:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.0-linux-x86_64.tar.gz
tar -xzvf filebeat-7.7.0-linux-x86_64.tar.gz

Configuration file filebeat.yml defines inputs, harvesters, and outputs. Start with:

./filebeat -e

Logstash Installation

Download, extract, and run a simple HelloWorld pipeline:

tar -zxvf logstash-7.7.0.tar.gz
./bin/logstash -e 'input { stdin { } } output { stdout {} }'

Elasticsearch Installation

Elasticsearch is a distributed document store built on Lucene, offering horizontal scalability, full‑text search, near‑real‑time analytics, high availability, and a RESTful API.

Key system settings (ulimit, vm.max_map_count, memory lock, etc.) are shown to optimize performance.

Configuration elasticsearch.yml includes cluster name, node roles, network settings, discovery hosts, and X‑Pack security with TLS certificates generated via ./bin/elasticsearch-certutil.

Kibana Installation

Download, extract, and edit kibana.yml to point to the Elasticsearch cluster and set authentication credentials.

Start Kibana with ./bin/kibana and access via http://<em>host</em>:5601.

Example Analysis

A complete pipeline is demonstrated using Filebeat to ship Apache logs to Kafka, Logstash to parse JSON and Grok patterns, and Elasticsearch for storage, visualized in Kibana.

Configuration snippets for filebeat.yml, Logstash apache.conf, and the corresponding startup commands are provided.

Resulting indices and visualizations are shown with screenshots of Elasticsearch and Kibana dashboards.

Original article link: https://www.cnblogs.com/zsql/p/13164414.html