Mastering ELK Stack: Step-by-Step Installation, Configuration, and Usage

ELK Overview

What is ELK?

ELK is a combination of three open‑source projects—Elasticsearch, Logstash, and Kibana—offered by Elastic as an enterprise‑grade solution for log collection, analysis, and visualization. Its main advantages include flexible processing, simple JSON‑based configuration, high‑performance real‑time search, linear cluster scaling, and a rich web UI.

Elasticsearch

Elasticsearch is a distributed, open‑source search engine built on Apache Lucene. It provides real‑time indexing, full‑text search, and RESTful JSON APIs. Key features are distributed architecture, automatic shard and replica management, high availability, and easy scalability.

Logstash

Logstash is a log‑collection and processing pipeline. It runs in a client‑server model, gathering logs from many sources, filtering/modifying them, and forwarding the results to Elasticsearch.

Kibana

Kibana offers a web UI for visualizing data stored in Elasticsearch, allowing users to create dashboards, charts, and search logs interactively.

ELK Workflow

1. Deploy Logstash on each log‑generating server. 2. Logstash formats logs and sends them to an Elasticsearch cluster. 3. Elasticsearch indexes and stores the data. 4. Kibana queries the cluster and displays the results.

Installation and Deployment

Prerequisites

# systemctl disable --now firewalld
# setenforce 0
# yum -y install java
# hostnamectl set-hostname node1

Install Elasticsearch

Download and install the RPM

# cd /data
# rpm -ivh elasticsearch-6.7.2.rpm

Backup and edit configuration

# cp -a *.yml bak/
# vim elasticsearch.yml
cluster.name: my-elk-cluster
node.name: node1
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["192.168.10.10:9300", "192.168.10.20:9300"]

System limits tuning

# vim /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc 32000
* hard nproc 32000
* soft memlock unlimited
* hard memlock unlimited

systemd configuration

# vim /etc/systemd/system.conf
DefaultLimitNOFILE=65536
DefaultLimitNPROC=32000
DefaultLimitMEMLOCK=infinity

Kernel parameter

# vim /etc/sysctl.conf
vm.max_map_count=262144
# sysctl -p

Start Elasticsearch

# systemctl enable --now elasticsearch.service
# ss -natp | grep 9200

Install Elasticsearch‑head plugin

Elasticsearch‑head requires Node.js and PhantomJS.

Compile Node.js

# yum install gcc gcc-c++ make -y
# tar zxvf node-v8.2.1.tar.gz
# cd node-v8.2.1
# ./configure && make && make install

Install PhantomJS

# tar jxvf phantomjs-2.1.1-linux-x86_64.tar.bz2
# ln -s /opt/phantomjs-2.1.1-linux-x86_64/bin/phantomjs /usr/bin

Deploy elasticsearch‑head

# unzip elasticsearch-head-master.zip
# cd elasticsearch-head-master
# npm install
# npm install --registry=https://registry.npmmirror.com
# vim /etc/elasticsearch/elasticsearch.yml   (add http.cors.enabled: true and http.cors.allow-origin: "*")
# systemctl restart elasticsearch
# cd elasticsearch-head-master
# npm run start &

Test elasticsearch‑head

Access http://<server_ip>:9100 in a browser.

Install Logstash

# yum -y install java
# yum -y install httpd && systemctl start httpd
# rpm -ivh logstash-6.7.2.rpm
# systemctl enable --now logstash.service
# ln -s /usr/share/logstash/bin/logstash /usr/bin/

Logstash command‑line usage

# logstash -e 'input { stdin{} } output { stdout{} }'

Logstash pipeline example (inline)

logstash -e 'input { stdin{} } output { elasticsearch { hosts=>["192.168.10.10:9200","192.168.10.20:9200"]} }'

Logstash configuration file structure

input : defines data sources such as file, beats, kafka, redis, stdin.

filter : optional processing (grok, date, mutate, multiline, etc.).

output : typically Elasticsearch or stdout.

Sample Logstash config (system‑log.conf)

input {
  file { path => "/var/log/messages" type => "system" start_position => "beginning" }
  file { path => "/var/log/yum.log" type => "yum" start_position => "beginning" }
}
output {
  if [type] == "system" {
    elasticsearch { hosts => ["192.168.91.100:9200","192.168.91.101:9200"] index => "system-%{+YYYY.MM.dd}" }
  }
}

Run Logstash with the config

# chmod +r /var/log/messages
# logstash -f system-log.conf

Install Kibana

# rpm -ivh kibana-6.7.2-x86_64.rpm

Configure Kibana

# vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.91.100:9200","http://192.168.91.101:9200"]
kibana.index: ".kibana"
logging.dest: /var/log/k.log
i18n.locale: "zh-CN"
# chown kibana:kibana /var/log/k.log

Start Kibana

# systemctl enable --now kibana.service
# ss -nap | grep 5601

Access Kibana

Open http://<kibana_ip>:5601 in a browser to view dashboards and verify the ELK stack is working.