How to Build a Full-Stack ELK Logging Platform with Redis, Filebeat, and Grafana

Environment Configuration

Three servers are used: two 1v2g Alibaba Cloud VMs and a friend’s 1v2g VPS.

Architecture

Node Roles

node1: Elasticsearch 6.4 + Filebeat

node2: Kibana 6.4 + Grafana + Filebeat

node3: Logstash + Nginx + Filebeat + Redis

Elasticsearch runs alone with 1 primary shard and 0 replicas; old indices are deleted weekly.

Log Collection – Redis

Redis is used as the Logstash broker. It provides both input and output plugins; producers write logs, consumers read them.

Redis Message Queue Benefits

1. Prevent Logstash‑Elasticsearch communication failures.
2. Avoid ES overload from massive write operations.
3. Applications can write logs directly to the queue.

If Redis becomes a bottleneck, consider Kafka or Flume.

Compile and Install Redis

wget http://download.redis.io/releases/redis-4.0.11.tar.gz
tar -zxf redis-4.0.11.tar.gz
cp -r redis-4.0.11 /usr/local/
mv redis-4.0.11/ redis
cd /usr/local/redis/
cp redis.conf bin/
make PREFIX=/usr/local/redis install
echo "export PATH=$PATH:/usr/local/redis/bin" >> /etc/profile

Start Redis

Foreground (not recommended): redis-server Daemon mode: set daemonize yes in redis.conf and run: redis-server redis.conf Connect with redis-cli. Shut down safely with redis-cli shutdown.

Nginx Log Format

Replace the default log format with a JSON structure for easier ingestion:

log_format json '{ "@timestamp": "$time_iso8601", "time": "$time_iso8601", "clientip": "$remote_addr", "remote_user": "$remote_user", "body_bytes_sent": "$body_bytes_sent", "request_time": "$request_time", "status": "$status", "host": "$host", "request": "$request", "request_method": "$request_method", "uri": "$uri", "http_referrer": "$http_referer", "http_x_forwarded_for": "$http_x_forwarded_for", "http_user_agent": "$http_user_agent" }';
access_log /var/log/nginx/io.log json;
access_log /var/log/nginx/doc.access.log json;

Reload Nginx: systemctl reload nginx Verify the JSON logs:

tail -f /var/log/nginx/io.log

Filebeat Installation and Configuration

Filebeat is a lightweight log shipper; it reduces the need for Logstash on every host.

Install via RPM

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.0.1-x86_64.rpm
rpm -vi filebeat-7.0.1-x86_64.rpm

Collect Nginx Logs

Define inputs and route each log type to a separate Redis key:

vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
    - /var/log/nginx/io.log
  fields:
    type: www_access
  fields_under_root: true
- type: log
  paths:
    - /var/log/nginx/io.error.log
  fields:
    type: www_error
  fields_under_root: true
- type: log
  paths:
    - /var/log/nginx/doc.access.log
  fields:
    type: doc_access
  fields_under_root: true
setup.template.settings:
  index.number_of_shards: 1
output.redis:
  hosts: ["127.0.0.1:6379"]
  key: "nginx"
  keys:
    - key: "%{[type]}"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Restart Filebeat: systemctl restart filebeat Check Redis for incoming logs with redis-cli. The key length grows as logs are written.

Logstash Configuration (Processing Side)

Install Logstash (requires Java 8):

yum install java
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
vim /etc/yum.repos.d/logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

yum install logstash

Logstash Pipeline

Read from Redis, decode JSON, add timestamps, enrich with geo‑IP, and output to Elasticsearch:

vim /etc/logstash/conf.d/test.conf
input {
  redis {
    host => "127.0.0.1"
    port => 6379
    type => "www_access"
    data_type => "list"
    key => "www_access"
    codec => "json"
  }
  redis {
    host => "127.0.0.1"
    port => 6379
    type => "nginx_error"
    data_type => "list"
    key => "www_error"
  }
  redis {
    host => "127.0.0.1"
    port => 6379
    type => "doc_access"
    data_type => "list"
    key => "doc_access"
    codec => "json"
  }
}
filter {
  if [type] == "www_access" {
    json { source => "message" remove_field => "message" }
    date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] }
    geoip { source => "clientip" fields => ["city_name","country_code2","country_name","region_name","longitude","latitude","ip"] add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"] add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"] }
    mutate { convert => ["[geoip][coordinates]", "float"] }
  } else if [type] =~ "error" {
    grok {}
    date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] }
  } else {
    json { source => "message" remove_field => "message" }
    date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] }
    geoip { source => "clientip" fields => ["city_name","country_code2","country_name","region_name","longitude","latitude","ip"] add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"] add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"] }
    mutate { convert => ["[geoip][coordinates]", "float"] }
  }
}
output {
  if [type] == "www_access" {
    elasticsearch { hosts => ["60.205.177.168:9200"] index => "nginx_access-%{+YYYY.MM.dd}" }
    stdout { codec => rubydebug }
  } else if [type] == "www_error" {
    elasticsearch { hosts => ["60.205.177.168:9200"] index => "nginx_error-%{+YYYY.MM.dd}" }
  } else {
    elasticsearch { hosts => ["60.205.177.168:9200"] index => "nginx_doc-%{+YYYY.MM.dd}" }
    stdout { codec => rubydebug }
  }
}

Start Logstash: logstash -f /etc/logstash/conf.d/test.conf Logs are consumed from Redis and indexed into Elasticsearch.

Elasticsearch Installation (Storage & Analysis)

Elasticsearch also requires Java.

Download and Extract

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
tar -xzf elasticsearch-6.4.0.tar.gz

Configure

vim config/elasticsearch.yml
cluster.name: my-elk
node.name: es1
path.data: /elasticsearch/elasticsearch-6.4.0/data
path.logs: /elasticsearch/elasticsearch-6.4.0/logs/
network.host: 192.168.179.134
http.port: 9200

Start

cd elasticsearch-6.4.0/
./bin/elasticsearch

Run as a background process with -d and ensure proper file permissions (e.g., chown -R elker.elker /elasticsearch).

Verify

netstat -ntlp | grep 9200
curl 192.168.179.134:9200

Common Startup Warnings & Fixes

max file descriptors too low – increase in /etc/security/limits.conf :

* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

max virtual memory areas too low – set in /etc/sysctl.conf :

vm.max_map_count=655360
sysctl -p

Kibana Installation (Visualization)

Download & Extract

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz
tar -xzf kibana-6.4.0-linux-x86_64.tar.gz

Configure

vim kibana-6.4.0-linux-x86_64/config/kibana.yml
server.port: 5601
server.host: "192.168.179.134"
elasticsearch.url: "http://192.168.179.134:9200"

Start

/kibana-6.4.0-linux-x86_64/bin/kibana

Check with netstat -ntlp | grep 5601 and open http://192.168.179.134:5601 in a browser.

Security Warnings & Fixes

Missing encryption key – add to kibana.yml :

xpack.reporting.encryptionKey: "a_random_string"

Insecure session cookies – add:

xpack.security.encryptionKey: "something_at_least_32_characters"

Grafana Installation (Dashboarding)

Install RPM

wget https://dl.grafana.com/oss/release/grafana-5.4.2-1.x86_64.rpm
yum install grafana-5.4.2-1.x86_64.rpm
yum install initscripts -y

Start Service

service grafana-server start
/sbin/chkconfig --add grafana-server
# or systemd
systemctl daemon-reload
systemctl start grafana-server
systemctl enable grafana-server.service

Configure MySQL Backend

Default SQLite can be switched to MySQL by editing /etc/grafana/grafana.ini:

[database]
type = mysql
host = 192.168.179.131:3306
name = grafana
user = root
password = "123456"

Create the database first:

create database grafana character set utf8 collate utf8_general_ci;

If MySQL 8 rejects the default auth plugin, switch to mysql_native_password:

alter user root@'%' identified with mysql_native_password by '123456';
select user, plugin from mysql.user;

Data Visualization in Kibana

Create index patterns for the three log types, then build visualizations such as a bar chart of IP access counts and a geo‑point map using the enriched geoip fields.

Example steps: select index, set clientip as the X‑axis, choose a metric aggregation, and save.

Data Visualization in Grafana

Add Elasticsearch as a data source, then create dashboards like a blog‑traffic graph.

All screenshots in the original article illustrate the UI configuration.

Welcome to discuss and improve the setup together.