Mastering ELK: From Log Collection to Real-Time Visualization with Redis Caching

ELK Practical Guide

After successfully deploying an ELK logging system, this article shows how to use Kibana for visualizing logs and how to extend the stack with Redis as a caching queue.

In Kibana, open the Settings menu, select Index Patterns , and create patterns such as nagios-system-* and nagios-secure-* to match the indices created by Logstash.

Once the index patterns are created, go to Discover to view collected logs, add or filter fields, and see the full log details.

Collecting Application Logs

Edit the Nginx configuration to output logs in JSON format:

[root@elk-master ~]# vim /usr/local/nginx/conf/nginx.conf
log_format json '{"@timestamp":"$time_iso8601","@version":"1","client":"$remote_addr","url":"$uri","status":"$status","domian":"$host","host":"$server_addr","size":"$body_bytes_sent","responsetime":"$request_time","referer":"$http_referer","ua":"$http_user_agent"}';
access_log  logs/elk.access.log  json;

Similarly, configure Apache to emit JSON logs:

# vim /etc/httpd/conf/httpd.conf
LogFormat "{ \"@timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \"@version\": \"1\", \"tags\":[\"apache\"], \"message\": \"%h %l %u %t \"%r\" %>s %b\", \"clientip\": \"%a\", \"duration\": %D, \"status\": %>s, \"request\": \"%U%q\", \"urlpath\": \"%U\", \"urlquery\": \"%q\", \"bytes\": %B, \"method\": \"%m\", \"site\": \"%{Host}i\", \"referer\": \"%{Referer}i\", \"useragent\": \"%{User-agent}i\" }" ls_apache_json
CustomLog logs/access_log ls_apache_json

Logstash Configuration for Log Collection

input {
    file { path => "/var/log/messages" type => "system" start_position => "beginning" }
    file { path => "/var/log/secure" type => "secure" start_position => "beginning" }
    file { path => "/var/log/httpd/access_log" type => "http" start_position => "beginning" }
    file { path => "/usr/local/nginx/logs/elk.access.log" type => "nginx" start_position => "beginning" }
}

output {
    if [type] == "system" {
        elasticsearch { hosts => ["192.168.73.133:9200"] index => "nagios-system-%{+YYYY.MM.dd}" }
    }
    if [type] == "secure" {
        elasticsearch { hosts => ["192.168.73.133:9200"] index => "nagios-secure-%{+YYYY.MM.dd}" }
    }
    if [type] == "http" {
        elasticsearch { hosts => ["192.168.73.133:9200"] index => "nagios-http-%{+YYYY.MM.dd}" }
    }
    if [type] == "nginx" {
        elasticsearch { hosts => ["192.168.73.133:9200"] index => "nagios-nginx-%{+YYYY.MM.dd}" }
    }
}

Run Logstash in the background:

[root@elk-master ~]# nohup logstash -f /etc/logstash/conf.d/elk.conf &

Introducing Redis as a Caching Queue

Using Redis reduces pressure on Elasticsearch when handling high‑volume logs and provides a buffer to prevent data loss.

Install Redis: # yum install -y redis Edit /etc/redis.conf to bind to the appropriate IP, enable daemonize yes, and set a password.

Start Redis: # systemctl restart redis Test the Redis instance with redis-cli -h 192.168.73.133 info.

Redis Output Configuration (logstash → Redis)

# vim /etc/logstash/conf.d/redis-out.conf
output {
    redis {
        host => "192.168.73.133"
        port => "6379"
        password => "root123"
        db => "2"
        data_type => "list"
        key => "nagios_system"
    }
    # repeat similar blocks for http, nginx, secure logs
}

Redis Input Configuration (Redis → Elasticsearch)

# vim /etc/logstash/conf.d/redis-in.conf
input {
    redis {
        type => "system"
        host => "192.168.73.133"
        password => "root123"
        port => "6379"
        db => "2"
        data_type => "list"
        key => "nagios_system"
        batch_count => 1
    }
    # repeat for http, nginx, secure logs
}
output {
    if [type] == "system" {
        elasticsearch { hosts => ["192.168.73.133:9200"] index => "nagios-system-%{+YYYY.MM.dd}" }
    }
    # repeat for other types
}

Run the Redis‑based pipelines similarly with nohup logstash -f /etc/logstash/conf.d/redis-out.conf &.

Production Tips

Classify logs (system, access, error, runtime, device, debug) and define a consistent JSON format.

Standardize paths and keep log formats JSON‑friendly.

Process logs in order: system → error → runtime → access.

Periodically delete old indices, e.g.,

curl -X DELETE http://<em>host</em>:9200/logstash-*-`date +%Y-%m-%d -d "-30 days"`

.