How to Build a Distributed ELK+Redis Log Collection System on CentOS

ELK Overview

ELK Stack consists of Elasticsearch, Logstash, and Kibana. It provides powerful log monitoring and analysis, useful for tracking nginx access logs or any other log source. In this setup, Logstash agents read nginx logs, push them to a Redis queue, and Logstash indexers consume the queue, store data in Elasticsearch, and display it via Kibana.

System Environment

Two machines are used:

hadoop‑master (192.168.186.128) – runs nginx and a Logstash indexer (installed from source).

hadoop‑slave (192.168.186.129) – runs Logstash agent, Elasticsearch, Redis, and nginx.

Redis Installation

[root@hadoop-slave ~]# wget https://github.com/antirez/redis/archive/2.8.20.tar.gz
[root@hadoop-slave ~]# tar -zxf 2.8.20.tar.gz
[root@hadoop-slave ~]# mv redis-2.8.20/ /usr/local/src/
[root@hadoop-slave src]# cd redis-2.8.20/
[root@hadoop-slave src]# make

After compilation, create directories for configuration, runtime, and data, then copy the default config:

[root@hadoop-slave local]# mkdir -p /usr/local/redis/{conf,run,db}
[root@hadoop-slave local]# cp redis.conf /usr/local/redis/conf/

Start Redis in the background and verify the listening port (6379):

[root@hadoop-slave src]# /usr/local/redis/redis-server /usr/local/redis/conf/redis.conf &
[root@hadoop-slave redis]# netstat -antulp | grep 6379

Elasticsearch Installation

Elasticsearch uses HTTP port 9200 and inter‑node TCP port 9300. Install the 1.7.1 tarball and create a symlink for easy access:

[root@hadoop-slave ~]# wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.tar.gz
[root@hadoop-slave ~]# mkdir /usr/local/elk
[root@hadoop-slave ~]# tar zxf elasticsearch-1.7.1.tar.gz -C /usr/local/elk/
[root@hadoop-slave bin]# ln -s /usr/local/elk/elasticsearch-1.7.1/bin/elasticsearch /usr/bin
[root@hadoop-slave bin]# elasticsearch start

Verify the service with a curl request:

[root@hadoop-slave ~]# curl -X GET http://localhost:9200
{
  "status" : 200,
  "name" : "Wasp",
  "cluster_name" : "elasticsearch",
  "version" : { "number" : "1.7.1", "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19", "build_timestamp" : "2015-07-29T09:54:16Z", "build_snapshot" : false, "lucene_version" : "4.10.4" },
  "tagline" : "You Know, for Search"
}

Logstash Installation

Logstash can be installed from source or via yum. The example uses source on the master node and yum on the slave node.

# Source installation on master
[root@hadoop-master ~]# wget https://download.elastic.co/logstash/logstash/logstash-1.5.3.tar.gz
[root@hadoop-master ~]# tar -zxf logstash-1.5.3.tar.gz -C /usr/local/
[root@hadoop-master logstash-1.5.3]# mkdir /usr/local/logstash-1.5.3/etc

# Yum installation on slave
[root@hadoop-slave ~]# rpm --import https://packages.elasticsearch.org/GPG-KEY-elasticsearch
[root@hadoop-slave ~]# vi /etc/yum.repos.d/CentOS-Base.repo   # add Logstash repo
[root@hadoop-slave ~]# yum install logstash   # installs to /opt

Kibana Installation

[root@hadoop-slave ~]# wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
[root@hadoop-slave ~]# tar -zxf kibana-4.1.1-linux-x64.tar.gz
[root@hadoop-slave ~]# mv kibana-4.1.1-linux-x64 /usr/local/elk
[root@hadoop-slave bin]# ./kibana &

Access the UI at http://192.168.186.129:5601/ (open TCP 5601 if remote access is needed).

Logstash Configuration

Logstash roles are described as shipper (agent), broker (Redis), and indexer. The following configurations illustrate the agent that reads nginx logs and pushes them to Redis, and the indexer that reads from Redis and writes to Elasticsearch.

Agent Configuration (logstash_agent.conf)

input {
  file {
    type => "nginx access log"
    path => ["/usr/local/nginx/logs/host.access.log"]
  }
}
output {
  redis {
    host => "192.168.186.129"
    data_type => "list"
    key => "logstash:redis"
  }
}

Run the agent in background:

nohup /usr/local/logstash-1.5.3/bin/logstash -f /usr/local/logstash-1.5.3/etc/logstash_agent.conf &

Indexer Configuration (logstash_indexer.conf)

input {
  redis {
    host => "192.168.186.129"
    data_type => "list"
    key => "logstash:redis"
    type => "redis-input"
  }
}
filter {
  grok {
    type => "nginx_access"
    match => [
      "message", "%{IPORHOST:http_host} %{IPORHOST:client_ip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:http_status_code} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{NUMBER:time_duration:float} %{NUMBER:time_backend_response:float}",
      "message", "%{IPORHOST:http_host} %{IPORHOST:client_ip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:http_status_code} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{NUMBER:time_duration:float}"
    ]
  }
}
output {
  elasticsearch {
    embedded => false
    protocol => "http"
    host => "localhost"
    port => "9200"
  }
}

Run the indexer in background:

nohup /opt/logstash/bin/logstash -f /opt/logstash/etc/logstash_indexer.conf &

Testing the ELK+Redis Pipeline

Start all components:

/usr/local/redis/redis-server /usr/local/redis/conf/redis.conf &
elasticsearch start -d
nohup /usr/local/logstash-1.5.3/bin/logstash -f /usr/local/logstash-1.5.3/etc/logstash_agent.conf &
nohup /opt/logstash/bin/logstash -f /opt/logstash/etc/logstash_indexer.conf &
nohup /opt/logstash/bin/logstash -f /opt/logstash/etc/logstash_agent.conf &
./kibana &

Generate traffic by refreshing the nginx pages on both machines (http://192.168.186.129/ and http://192.168.186.128/). The access logs are collected, pushed through Redis, indexed in Elasticsearch, and visualized in Kibana.

In Kibana you will see the combined nginx access logs from both hosts. Timezone differences may appear but do not affect log content.