How to Build a Centralized Log System with the ELK Stack: Step‑by‑Step Installation Guide

Introduction

In everyday life we often need to review past events or trace the cause of problems, which requires logs—textual or visual records of system activity. Modern distributed systems generate massive amounts of log data (approximately 2 EB per day worldwide), making traditional manual log inspection impractical.

A centralized log system addresses this challenge by collecting, transporting, storing, analyzing, and alerting on log data from diverse sources.

Market Products

Various solutions exist, ranging from simple tools like Rsyslog and Syslog‑ng to commercial products such as Splunk. Open‑source alternatives include Facebook’s Scribe, Apache’s Chukwa, LinkedIn’s Kafka, Cloudera’s Fluentd, and the ELK stack.

While Splunk is powerful, its high cost limits adoption; the ELK stack offers a free, open‑source alternative.

ELK Stack Overview

ELK is not a single product but a suite of three open‑source components—Elasticsearch, Logstash, and Kibana—commonly used together and maintained by Elastic.co.

Elasticsearch

Elasticsearch is a real‑time distributed search and analytics engine built on Apache Lucene. It supports full‑text, structured, and analytical queries, offers high availability, clustering, sharding, replication, and a JSON‑friendly API.

Logstash

Logstash is a real‑time data collection engine written in JRuby. It consists of three parts: Shipper (input), Broker (default Redis), and Indexer (output). It can ingest data from virtually any source, integrate with many external applications, and scale elastically.

Kibana

Kibana is a JavaScript‑based web UI for visualizing Elasticsearch data. It provides interactive dashboards, charts, and tables, with the latest version 4.3 (Kibana 4).

ELK Architecture

The typical flow is: Shipper collects data from sources, forwards it to the Broker, the Indexer writes data into Elasticsearch, which creates indices, and Kibana visualizes the results.

Installation and Configuration

The guide uses two virtual machines (Ubuntu 14.04 and CentOS 7.1) with 2 CPU, 4 GB RAM, and 30 GB disk. Required software: Elasticsearch 2.1.0, Logstash 2.1.1, Kibana 4.3.0, Nginx, Logstash‑forwarder, and JDK.

Step 1: Install JDK

$ sudo add-apt-repository ppa:openjdk-r/ppa
$ sudo apt-get update
$ sudo apt-get install openjdk-8-jdk
$ java -version

$ sudo yum install java-1.8.0-openjdk
$ java -version

Step 2: Install Elasticsearch

$ wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.1.0/elasticsearch-2.1.0.tar.gz
$ tar xzvf elasticsearch-2.1.0.tar.gz
$ cd elasticsearch-2.1.0/config
$ vi elasticsearch.yml   # set network.host: localhost
$ ./bin/elasticsearch &
$ curl localhost:9200/

Step 3: Install Kibana

$ wget https://download.elastic.co/kibana/kibana/kibana-4.3.0-linux-x64.tar.gz
$ tar xzvf kibana-4.3.0-linux-x64.tar.gz
$ cd kibana-4.3.0-linux-x64/config
$ vi kibana.yml   # set server.host: "localhost"
$ ./bin/kibana &

Step 4: Install Nginx (reverse proxy)

$ sudo apt-get install nginx apache2-utils   # Ubuntu
$ sudo vi /etc/nginx/sites-available/default   # proxy_pass http://localhost:5601
$ sudo service nginx restart
$ sudo yum install nginx httpd-tools   # CentOS
$ sudo vi /etc/nginx/nginx.conf   # include /etc/nginx/conf.d/*conf
$ sudo vi /etc/nginx/conf.d/kibana.conf   # same proxy configuration
$ sudo systemctl enable nginx
$ sudo systemctl start nginx

Step 5: Install Logstash

$ wget https://download.elastic.co/logstash/logstash/logstash-2.1.1.tar.gz
$ tar xzvf logstash-2.1.1.tar.gz
$ cd logstash-2.1.1/bin
$ ./logstash -e 'input { stdin { } } output { stdout {} }'

Step 6: Configure Logstash Pipeline

# simple.conf
input {
  lumberjack {
    port => 5043
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}
filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] }
}
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

Step 7: Install Logstash‑forwarder

$ echo 'deb http://packages.elastic.co/logstashforwarder/debian stable main' | sudo tee /etc/apt/sources.list.d/logstashforwarder.list
$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install logstash-forwarder
$ sudo mkdir -p /etc/pki/tls/certs
$ sudo scp user@logstash_server:/etc/pki/tls/certs/logstash_forwarder.crt /etc/pki/tls/certs/
$ sudo vi /etc/logstash-forwarder.conf   # configure servers and files
$ sudo service logstash-forwarder start
$ sudo service logstash-forwarder status

Step 8: Final Verification

After starting all components, access Kibana via the Nginx proxy. The initial empty dashboard appears, then after Logstash‑forwarder begins sending data, indices can be created and visualized in the Discover view.

All components work together to provide a functional centralized logging solution.