Mastering ELK: Step‑by‑Step Installation and Integration of Logstash, Redis, Elasticsearch, and Kibana

1. Introduction

ELK consists of three open‑source components: Elasticsearch, Logstash, and Kibana. Elasticsearch is a distributed search engine featuring zero‑configuration, automatic discovery, sharding, replica mechanisms, a RESTful API, and multi‑source data handling. Logstash collects, parses, and stores logs, while Kibana offers a web UI for visualizing Logstash and Elasticsearch data.

2. Logstash

Logstash requires a Java runtime environment. Install JDK with # yum -y install java-1.8.0, verify the version, then download Logstash (

# wget https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz

), extract it, and add its bin directory to PATH:

# echo "export PATH=$PATH:/usr/local/logstash-1.5.4/bin" > /etc/profile.d/logstash.sh

# . /etc/profile

Common Logstash parameters: -e: specify configuration inline (useful for quick tests). -f: specify a configuration file for production use.

Example of starting Logstash with inline configuration and plain output: # logstash -e "input {stdin{}} output {stdout{}}" Example of JSON output:

# logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'

Using a configuration file ( logstash-simple.conf) that reads from stdin and writes to stdout with the rubydebug codec:

# vim logstash-simple.conf

input { stdin {} }

output { stdout { codec => rubydebug } }

Run Logstash in agent mode with verbose output:

# logstash -f logstash-simple.conf --verbose

Pipeline started {:level=>:info}

Logstash startup completed

3. Redis

Installation steps (example for Redis 2.8.19):

# wget http://download.redis.io/releases/redis-2.8.19.tar.gz

# tar zxf redis-2.8.19.tar.gz

# cd redis-2.8.19

# make

# make install

Run the provided installer script to set default port (6379), configuration file, log file, data directory, and executable path. The script prompts for defaults and applies them.

Test the server: # ./redis-cli -h 192.168.1.104 -p 6379 ping →

PONG

# ./redis-cli -h 192.168.1.104 -p 6379 set name zhengyansheng

→

OK

# ./redis-cli -h 192.168.1.104 -p 6379 get name

→ "zhengyansheng" Monitor real‑time activity:

# ./redis-cli monitor

OK

1444315328.103928 [0 192.168.1.104:56211] "rpush" "logstash:redis" "{\"message\":\"dajihao linux\",...}"

4. Elasticsearch

Installation:

# wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.tar.gz

# tar zxf elasticsearch-1.7.2.tar.gz -C /usr/local/

Edit /usr/local/elasticsearch-1.7.2/config/elasticsearch.yml to disable multicast discovery, set the network host, and enable CORS for Kibana:

discovery.zen.ping.multicast.enabled: false

network.host: 192.168.1.104

http.cors.allow-origin: "/.*/"

http.cors.enabled: true

Start Elasticsearch: # /usr/local/elasticsearch-1.7.2/bin/elasticsearch (foreground) or # /usr/local/elasticsearch-1.7.2/bin/elasticsearch -d (daemon).

Verify listening ports:

# netstat -tnlp | grep java

tcp        0      0 :::9200                 :::*                    LISTEN      7407/java

tcp        0      0 :::9300                 :::*                    LISTEN      7407/java

Logstash to Elasticsearch configuration ( logstash-elasticsearch.conf):

# cat logstash-elasticsearch.conf

input { stdin {} }

output { elasticsearch { host => "192.168.1.104" } stdout { codec => rubydebug } }

Run Logstash with the configuration:

# /usr/local/logstash-1.5.4/bin/logstash agent -f logstash-elasticsearch.conf

Pipeline started {:level=>:info}

Logstash startup completed

{"message":"python linux java c++","@version":"1","@timestamp":"2015-10-08T14:51:56.899Z","host":"0.0.0.0"}

Check that the document was indexed:

# curl http://localhost:9200/_search?pretty

{ "took" : 28, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "logstash-2015.10.08", "_type" : "logs", "_id" : "AVBH7-6MOwimSJSPcXjb", "_score" : 1.0, "_source" : { "message" : "python linux java c++", "@version" : "1", "@timestamp" : "2015-10-08T14:51:56.899Z", "host" : "0.0.0.0" } } ] } }

Install the Kopf plugin for a graphical Elasticsearch UI:

# cd /usr/local/elasticsearch-1.7.2/bin/

# ./plugin install lmenezes/elasticsearch-kopf

If the automatic install fails, download the zip manually, unzip, and move the directory to /usr/local/elasticsearch-1.7.2/plugins/kopf .

5. Kibana

Download and extract Kibana 4.1.2: # wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz # tar zxf kibana-4.1.2-linux-x64.tar.gz -C /usr/local Edit /usr/local/kibana-4.1.2-linux-x64/config/kibana.yml to point to Elasticsearch: elasticsearch_url: "http://192.168.1.104:9200" Start Kibana: # /usr/local/kibana-4.1.2-linux-x64/bin/kibana {"name":"Kibana","hostname":"localhost.localdomain","pid":1943,"level":30,"msg":"No existing kibana index found","time":"2015-10-08T00:39:21.617Z","v":0} {"name":"Kibana","hostname":"localhost.localdomain","pid":1943,"level":30,"msg":"Listening on 0.0.0.0:5601","time":"2015-10-08T00:39:21.637Z","v":0} Kibana listens on port 5601. Open a browser at http://<host>:5601 , create an index pattern logstash-* , and use the Discover tab to browse the indexed log entries.