Mastering ELK Stack: Step-by-Step Installation and Configuration Guide

What is ELK Stack?

ELK Stack is the combination of three open‑source projects—Elasticsearch, Logstash and Kibana—commonly used together for real‑time data retrieval, analysis and visualization. All three are maintained by Elastic.co.

Why Choose ELK Stack?

Flexible processing: Elasticsearch provides real‑time full‑text indexing without pre‑programming.

Easy configuration: JSON API for Elasticsearch and Ruby‑DSL for Logstash.

High‑performance search: Queries are executed in real time with near‑second response.

Linear cluster scaling for both Elasticsearch and Logstash.

Rich front‑end: Kibana lets users create dashboards with a few clicks.

Install JDK

# download and install JRE 8u73
wget 'http://sdlc-esd.Oracle.com/ESD6/JSCDL/jdk/8u73-b02/jre-8u73-linux-x64.rpm' -O jre-8u73-linux-x64.rpm
yum install jre-8u73-linux-x64.rpm -y
java -version

Elasticsearch 2.2.0 Installation

Import the public key and create a YUM repository:

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/elasticsearch.repo <<EOF
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=https://packages.elastic.co/elasticsearch/2.x/CentOS
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
EOF
yum install elasticsearch
chkconfig --add elasticsearch   # SysV init
systemctl daemon-reload
systemctl enable elasticsearch.service
mkdir /opt/elasticsearch.path.data
sed -i 's|#path.data:.*|path.data: /opt/elasticsearch.path.data|' /etc/elasticsearch/elasticsearch.yml
sed -i 's|#network.host:.*|network.host: 0.0.0.0|' /etc/elasticsearch/elasticsearch.yml
service elasticsearch start

Kibana 4.4.1 Installation

Kibana 4.4.1 requires Elasticsearch 2.2.

wget https://download.elastic.co/kibana/kibana/kibana-4.4.1-linux-x64.tar.gz
tar zxvf kibana-4.4.1-linux-x64.tar.gz -C /opt/
vim /opt/kibana-4.4.1-linux-x64/config/kibana.yml
# set
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.174.68:9200"
nohup ./bin/kibana -l /var/log/kibana4.4.1.log &

Logstash Installation

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/logstash.repo <<EOF
[logstash-2.2]
name=Logstash repository for 2.2.x packages
baseurl=http://packages.elastic.co/logstash/2.2/CentOS
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
EOF
yum install logstash -y

Testing Logstash

JAVA_HOME=/usr/java/jre1.8.0_73 /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'

Log4j Integration

Add a SocketAppender to your Log4j configuration:

log4j.rootLogger=logstash
log4j.appender.logstash=org.apache.log4j.net.SocketAppender
log4j.appender.logstash.port=56789
log4j.appender.logstash.remoteHost=192.168.174.68

Create Logstash pipeline:

input {
  log4j {
    mode => "server"
    host => "0.0.0.0"
    port => 56789
    type => "log4j"
  }
}
output {
  elasticsearch {
    hosts => "192.168.174.68:9200"
  }
}

Access Kibana at http://192.168.174.68:5601 to view collected logs.

Problem & Solution

When starting Elasticsearch you may encounter:

Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.data' (/opt/elasticsearch.path.data/elasticsearch)

Fix by granting write permission:

chmod 777 /opt/elasticsearch.path.data/