Deploy ELK Stack: Complete Guide to Elasticsearch, Logstash & Kibana Setup

ELK Overview

The ELK log analysis system combines Logstash, Elasticsearch, and Kibana to provide a searchable, analyzable, and visualized logging solution that can ingest data from any source and format.

Components

Elasticsearch (ES) stores and indexes log data in a cluster; Logstash collects logs and forwards them to ES; Kibana visualizes the data in a user‑friendly interface.

Log Processing Steps

1. Centralize logs with Beats
2. Format logs with Logstash and output to Elasticsearch
3. Index and store formatted data in Elasticsearch
4. Visualize data with Kibana

Elasticsearch Details

Elasticsearch is a distributed, real‑time full‑text search engine written in Java, designed for cloud environments. Core concepts include:

Near‑real‑time (NRT) indexing (≈1 s delay).

Cluster: one or more nodes sharing data and providing indexing/search services.

Node: a single server that stores data and participates in the cluster.

Index (type): logical partition similar to a database table.

Shards & replicas: split indices into multiple primary shards and optional replica shards for scalability and fault tolerance.

Logstash Details

Logstash is a powerful data‑processing tool that can ingest, filter, transform, and output data. Main components:

Shipper – monitors local log files and collects new entries.

Indexer – stores logs locally.

Broker – connects multiple shippers and indexers.

Search and storage – enables searching and storing events.

Web interface – provides a web‑based UI.

Kibana Details

Kibana is an open‑source analytics and visualization platform for Elasticsearch. It allows searching, viewing, and creating advanced charts on indexed data.

Deploying the ELK Stack

Environment

node1 192.168.118.13
node2 192.168.118.14
apache 192.168.118.128

1. Disable firewall and SELinux

systemctl stop firewalld
systemctl disable firewalld
setenforce 0

2. Configure Elasticsearch hosts

echo '192.168.118.13 node1' >> /etc/hosts
echo '192.168.118.14 node2' >> /etc/hosts
java -version   # install with yum if missing

3. Install Elasticsearch

cd /opt
rpm -ivh elasticsearch-5.5.0.rpm
systemctl daemon-reload
systemctl enable elasticsearch.service
cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak
# edit /etc/elasticsearch/elasticsearch.yml:
cluster.name: my-elk-cluster
node.name: node1
path.data: /data/elk_data
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["node1","node2"]
mkdir -p /data/elk_data
chown elasticsearch:elasticsearch /data/elk_data
systemctl start elasticsearch
netstat -antp | grep 9200
# verify in browser: http://192.168.118.13:9200

4. Install elasticsearch‑head plugin

# Install Node.js
yum -y install gcc gcc-c++ make
tar zxvf node-v8.2.1.tar.gz
cd node-v8.2.1 && ./configure && make && make install
# Install PhantomJS
tar jxf phantomjs-2.1.1-linux-x86_64.tar.bz2 -C /usr/local/src/
cp /usr/local/src/phantomjs-2.1.1-linux-x86_64/bin/phantomjs /usr/local/bin
# Install elasticsearch‑head
tar zxf elasticsearch-head.tar.gz -C /usr/local/src/
cd /usr/local/src/elasticsearch-head && npm install
# Enable CORS in Elasticsearch
echo -e '
http.cors.enabled: true
http.cors.allow-origin: "*"' >> /etc/elasticsearch/elasticsearch.yml
systemctl restart elasticsearch.service
# Start head
cd /usr/local/src/elasticsearch-head && npm run start &

5. Install Logstash

# Install Apache and Java on the log source host
yum -y install httpd
systemctl start httpd
yum -y install java
# Install Logstash
cd /opt
rpm -ivh logstash-5.5.1.rpm
systemctl start logstash.service
systemctl enable logstash.service
ln -s /usr/share/logstash/bin/logstash /usr/local/bin/
# Test Logstash
logstash -e 'input { stdin{} } output { stdout{} }'
logstash -e 'input { stdin{} } output { stdout{ codec=>rubydebug } }'
logstash -e 'input { stdin{} } output { elasticsearch { hosts=>["192.168.118.13:9200"] } }'
# Configure file input for Apache logs (example in original guide)
vim /etc/logstash/conf.d/system.conf
systemctl restart logstash.service

6. Install Kibana

cd /opt
rpm -ivh kibana-5.5.1-x86_64.rpm
cp /etc/kibana/kibana.yml /etc/kibana/kibana.yml.bak
# edit kibana.yml:
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.221.20:9200"
kibana.index: ".kibana"
systemctl start kibana.service
systemctl enable kibana.service
# Access Kibana at http://<node_ip>:5601

Error troubleshooting

Common issues can be diagnosed by checking Elasticsearch cluster health ( http://<node_ip>:9200/_cluster/health?pretty) and Kibana logs. Screenshots of successful queries and index creation are shown in the original guide.

For the full original guide and additional screenshots, see the source link.