Building a Real-Time Log Analysis Platform with ELK: Installation, Configuration, and Usage

In modern micro‑service environments, logs are scattered across many machines, making troubleshooting difficult; a unified real‑time log analysis platform such as ELK can dramatically improve incident investigation efficiency.

ELK Overview

ELK consists of three open‑source components: Elasticsearch for distributed search and analytics, Logstash for data collection and processing, and Kibana for visualizing data in a browser.

Logstash

Logstash collects logs from various sources, applies filters, and forwards the structured data to a destination. Its pipeline has three stages:

Input – e.g., file { path => "/log/sb-log.log" } or redis { host => "10.140.45.190" } Filter – e.g., Grok patterns to parse timestamps, thread names, log levels, etc.

Output – e.g.,

elasticsearch { hosts => "localhost:9200" index => "logback" }

or

redis { data_type => "channel" key => "logstash_list_0" }

Elasticsearch

Elasticsearch is a distributed RESTful search and analytics engine that supports structured, unstructured, geo, and metric queries, offers fast millisecond‑level responses, and scales from a laptop to petabyte‑scale clusters.

Kibana

Kibana provides a browser‑based UI for creating dashboards and exploring Elasticsearch data without writing code or managing additional infrastructure.

ELK Implementation Diagram

Logstash (Shipper role) runs on each service host, reads log files, and pushes raw events to a Redis channel. A second Logstash instance (Indexer role) pulls from Redis, parses logs with Grok, and indexes them into Elasticsearch. Kibana queries Elasticsearch to display the logs.

Platform Setup

All components can be installed on a single Ubuntu machine for a tutorial environment. Required steps:

Install JDK 1.7+.

Download and extract Logstash, Elasticsearch, and Kibana tarballs.

Installing Logstash

Extract the package and run a simple pipeline to verify installation:

tar -xzvf logstash-7.3.0.tar.gz
cd logstash-7.3.0
bin/logstash -e 'input { stdin {} } output { stdout {} }'

Successful start is indicated by log messages and the ability to echo "Hello Logstash" and receive a JSON event.

Installing Elasticsearch

Extract and start Elasticsearch:

tar -xzvf elasticsearch-7.3.0-linux-x86_64.tar.gz
cd elasticsearch-7.3.0
bin/elasticsearch

If the machine has insufficient memory, adjust config/jvm.options. Run curl http://localhost:9200 to verify the cluster is up.

Installing Kibana

Extract and configure config/kibana.yml with the Elasticsearch host, then start:

tar -xzvf kibana-7.3.0-linux-x86_64.tar.gz
cd kibana-7.3.0-linux-x86_64/bin
./kibana

Access http://<i>ip</i>:5601 to see the Kibana UI.

Configuring Shipper Logstash

Use a Logstash config that reads the Spring Boot log file and sends each line to a Redis channel:

input {
  file { path => ["/log/sb-log.log"] }
}
output {
  redis { host => "10.140.45.190" port => 6379 db => 8 data_type => "channel" key => "logstash_list_0" }
}

Configuring Indexer Logstash

Configure Logstash to read from the same Redis channel, parse the log with Grok, and index into Elasticsearch:

input {
  redis { host => "192.168.142.131" port => 6379 db => 8 data_type => "channel" key => "sb-logback" }
}
filter {
  grok { match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NOTSPACE:threadName}\] %{LOGLEVEL:level}  %{DATA:logger} %{NOTSPACE:applicationName} -(?:.*=%{NUMBER:timetaken}ms|)" } }
}
output {
  stdout {}
  elasticsearch { hosts => "localhost:9200" index => "logback" }
}

Parsing Nginx Logs

Define a Grok pattern for Nginx access logs, e.g.:

%{IPV4:ip} - - \[%{HTTPDATE:time}\] "%{NOTSPACE:method} %{DATA:requestUrl} HTTP/%{NUMBER:httpVersion}" %{NUMBER:httpStatus} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:agent}"

Extend the Indexer configuration with multiple type inputs and conditional filters/output blocks to handle both Spring Boot and Nginx logs.

Viewing Results

After starting Elasticsearch, Kibana, the Shipper and Indexer Logstash instances, invoke the Spring Boot API or generate Nginx traffic. In Kibana’s Discover view, select the appropriate index (e.g., logback or nginx) to see structured log entries.

Running ELK in the Background

Use Supervisor to keep Elasticsearch, Logstash, and Kibana running as daemons and to enable automatic start on boot. Example supervisord.conf sections:

[program:elasticsearch]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
directory=/home/elk/elk/elasticsearch
user=elk
command=/home/elk/elk/elasticsearch/bin/elasticsearch

[program:logstash]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
directory=/home/elk/elk/logstash
user=elk
command=/home/elk/elk/logstash/bin/logstash -f /home/elk/elk/logstash/indexer-logstash.conf

[program:kibana]
environment=LS_HEAP_SIZE=5000m
directory=/home/elk/elk/kibana
user=elk
command=/home/elk/elk/kibana/bin/kibana

Reload Supervisor with sudo supervisorctl reload to start all services.

Conclusion

The article demonstrates how to build an ELK‑based real‑time log analysis platform, install and configure its three components, integrate logs from Spring Boot and Nginx using Logstash and Grok, and manage the stack with Supervisor for production‑grade reliability.