Mastering ELK: Step‑by‑Step Installation and Integration of Logstash, Redis, Elasticsearch, and Kibana
This guide walks through the complete installation, configuration, and integration of the ELK stack components—Logstash, Redis, Elasticsearch, and Kibana—providing command‑line examples, configuration snippets, testing procedures, and visual verification using Kibana and the Kopf plugin.
1. Introduction
ELK consists of three open‑source components: Elasticsearch, Logstash, and Kibana. Elasticsearch is a distributed search engine featuring zero‑configuration, automatic discovery, sharding, replica mechanisms, a RESTful API, and multi‑source data handling. Logstash collects, parses, and stores logs, while Kibana offers a web UI for visualizing Logstash and Elasticsearch data.
2. Logstash
Logstash requires a Java runtime environment. Install JDK with # yum -y install java-1.8.0, verify the version, then download Logstash (
# wget https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz), extract it, and add its bin directory to PATH:
# echo "export PATH=$PATH:/usr/local/logstash-1.5.4/bin" > /etc/profile.d/logstash.sh# . /etc/profileCommon Logstash parameters: -e: specify configuration inline (useful for quick tests). -f: specify a configuration file for production use.
Example of starting Logstash with inline configuration and plain output: # logstash -e "input {stdin{}} output {stdout{}}" Example of JSON output:
# logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'Using a configuration file ( logstash-simple.conf) that reads from stdin and writes to stdout with the rubydebug codec:
# vim logstash-simple.confinput { stdin {} }output { stdout { codec => rubydebug } }Run Logstash in agent mode with verbose output:
# logstash -f logstash-simple.conf --verbosePipeline started {:level=>:info}Logstash startup completed3. Redis
Installation steps (example for Redis 2.8.19):
# wget http://download.redis.io/releases/redis-2.8.19.tar.gz# tar zxf redis-2.8.19.tar.gz# cd redis-2.8.19# make# make installRun the provided installer script to set default port (6379), configuration file, log file, data directory, and executable path. The script prompts for defaults and applies them.
Test the server: # ./redis-cli -h 192.168.1.104 -p 6379 ping →
PONG# ./redis-cli -h 192.168.1.104 -p 6379 set name zhengyansheng→
OK# ./redis-cli -h 192.168.1.104 -p 6379 get name→ "zhengyansheng" Monitor real‑time activity:
# ./redis-cli monitorOK1444315328.103928 [0 192.168.1.104:56211] "rpush" "logstash:redis" "{\"message\":\"dajihao linux\",...}"4. Elasticsearch
Installation:
# wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.tar.gz# tar zxf elasticsearch-1.7.2.tar.gz -C /usr/local/Edit /usr/local/elasticsearch-1.7.2/config/elasticsearch.yml to disable multicast discovery, set the network host, and enable CORS for Kibana:
discovery.zen.ping.multicast.enabled: falsenetwork.host: 192.168.1.104http.cors.allow-origin: "/.*/"http.cors.enabled: trueStart Elasticsearch: # /usr/local/elasticsearch-1.7.2/bin/elasticsearch (foreground) or # /usr/local/elasticsearch-1.7.2/bin/elasticsearch -d (daemon).
Verify listening ports:
# netstat -tnlp | grep javatcp 0 0 :::9200 :::* LISTEN 7407/javatcp 0 0 :::9300 :::* LISTEN 7407/javaLogstash to Elasticsearch configuration ( logstash-elasticsearch.conf):
# cat logstash-elasticsearch.confinput { stdin {} }output { elasticsearch { host => "192.168.1.104" } stdout { codec => rubydebug } }Run Logstash with the configuration:
# /usr/local/logstash-1.5.4/bin/logstash agent -f logstash-elasticsearch.confPipeline started {:level=>:info}Logstash startup completed{"message":"python linux java c++","@version":"1","@timestamp":"2015-10-08T14:51:56.899Z","host":"0.0.0.0"}Check that the document was indexed:
# curl http://localhost:9200/_search?pretty{ "took" : 28, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "logstash-2015.10.08", "_type" : "logs", "_id" : "AVBH7-6MOwimSJSPcXjb", "_score" : 1.0, "_source" : { "message" : "python linux java c++", "@version" : "1", "@timestamp" : "2015-10-08T14:51:56.899Z", "host" : "0.0.0.0" } } ] } }Install the Kopf plugin for a graphical Elasticsearch UI:
# cd /usr/local/elasticsearch-1.7.2/bin/# ./plugin install lmenezes/elasticsearch-kopfIf the automatic install fails, download the zip manually, unzip, and move the directory to /usr/local/elasticsearch-1.7.2/plugins/kopf .
5. Kibana
Download and extract Kibana 4.1.2: # wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz # tar zxf kibana-4.1.2-linux-x64.tar.gz -C /usr/local Edit /usr/local/kibana-4.1.2-linux-x64/config/kibana.yml to point to Elasticsearch: elasticsearch_url: "http://192.168.1.104:9200" Start Kibana: # /usr/local/kibana-4.1.2-linux-x64/bin/kibana {"name":"Kibana","hostname":"localhost.localdomain","pid":1943,"level":30,"msg":"No existing kibana index found","time":"2015-10-08T00:39:21.617Z","v":0} {"name":"Kibana","hostname":"localhost.localdomain","pid":1943,"level":30,"msg":"Listening on 0.0.0.0:5601","time":"2015-10-08T00:39:21.637Z","v":0} Kibana listens on port 5601. Open a browser at http://<host>:5601 , create an index pattern logstash-* , and use the Discover tab to browse the indexed log entries.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
