Master Real-Time Log Management with the Open-Source ELK Stack

Logs include system, application, and security logs. Centralized logging helps ops and developers monitor server health, detect errors, and assess performance.

Traditional per‑server log access is cumbersome; using a centralized solution like syslog aggregates logs.

For advanced search and analysis, the ELK stack (Elasticsearch, Logstash, Kibana) provides a complete pipeline.

ELK Component Overview

Elasticsearch is a distributed search engine with features such as zero‑configuration, automatic node discovery, sharding, replication, RESTful API, and multi‑source support.

Logstash collects, parses, and stores logs for later use.

Kibana offers a web UI for visualizing and searching logs processed by Logstash and stored in Elasticsearch.

Installation Procedure

1. Install JDK (Java Runtime) required by Logstash. Download the appropriate JRE, extract it, and set JAVA_HOME and PATH environment variables.

# wget http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz
# mkdir /usr/local/java
# tar -zxf jdk-8u45-linux-x64.tar.gz -C /usr/local/java/
# export JAVA_HOME=/usr/local/java/jdk1.8.0_45
# export PATH=$PATH:$JAVA_HOME/bin

2. Install Logstash by downloading and extracting the tarball.

# wget https://download.elastic.co/logstash/logstash/logstash-1.5.2.tar.gz
# tar -zxf logstash-1.5.2.tar.gz -C /usr/local/
# /usr/local/logstash-1.5.2/bin/logstash -e 'input { stdin { } } output { stdout {} }'

3. Install Elasticsearch, extract it, and start the service.

# tar -zxf elasticsearch-1.6.0.tar.gz -C /usr/local/
# /usr/local/elasticsearch-1.6.0/bin/elasticsearch
# nohup /usr/local/elasticsearch-1.6.0/bin/elasticsearch >nohup &
# netstat -anp | grep :9200

4. Verify Logstash can send data to Elasticsearch using a simple configuration file and curl queries.

# cat logstash-es-simple.conf
input { stdin { } }
output {
  elasticsearch { host => "localhost" }
  stdout { codec => rubydebug }
}
# /usr/local/logstash-1.5.2/bin/logstash -f logstash-es-simple.conf
# curl 'http://localhost:9200/_search?pretty'

5. Install the Kopf plugin for Elasticsearch to browse data.

# cd /usr/local/elasticsearch-1.6.0/
# ./plugin -install lmenezes/elasticsearch-kopf

6. Install Kibana, extract it, and launch the web UI (default port 5601). Configure an index pattern (e.g., logstash-*) to view incoming logs.

# tar -zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/
# /usr/local/kibana-4.1.1-linux-x64/bin/kibana

7. Optionally configure Logstash as an indexer to ingest system logs into Elasticsearch.

# cat /usr/local/logstash-1.5.2/logstash-indexer.conf
input {
  file { type => "syslog" path => ["/var/log/messages","/var/log/syslog"] }
  syslog { type => "syslog" port => "5544" }
}
output {
  stdout { codec => rubydebug }
  elasticsearch { host => "localhost" }
}
# /usr/local/logstash-1.5.2/bin/logstash -f logstash-indexer.conf

After completing these steps, the ELK platform is ready for real‑time log collection, search, and visualization.