Building and Using an ELK Real‑Time Log Analysis Platform

In the process of troubleshooting production issues, centralized log collection is essential; the ELK stack (Elasticsearch, Logstash, Kibana) provides a unified real‑time log analysis solution.

ELK Overview

ELK consists of three open‑source components: Elasticsearch for distributed search and analytics, Logstash for data collection and processing, and Kibana for visualizing data in a browser.

Logstash

Logstash gathers logs from various sources, applies filters, and forwards the data to a destination such as Elasticsearch.

Input: supports files, syslog, MySQL, message queues, etc.

Filter: parses and transforms data into a structured format.

Output: can send data to Elasticsearch, Redis, and many other targets.

Elasticsearch

Elasticsearch is a distributed RESTful search and analytics engine offering fast queries, aggregation, scalability from a laptop to petabyte‑scale clusters, and resilience in distributed environments.

Kibana

Kibana provides a browser‑based UI for creating dashboards and exploring Elasticsearch data without writing code.

ELK Implementation

The typical workflow is: Logstash (Shipper) collects logs from services and pushes them to Redis; another Logstash instance (Indexer) reads from Redis, parses logs, and stores them in Elasticsearch; Kibana queries Elasticsearch and displays the results.

Platform Setup

All components can be installed on a single Ubuntu machine for a demo.

Install Logstash

tar -xzvf logstash-7.3.0.tar.gz

cd logstash-7.3.0</code><code>bin/logstash -e 'input { stdin {} } output { stdout {} }'

Install Elasticsearch

tar -xzvf elasticsearch-7.3.0-linux-x86_64.tar.gz

cd elasticsearch-7.3.0/</code><code>bin/elasticsearch

Verify with curl http://localhost:9200 and check the JSON response.

Install Kibana

tar -xzvf kibana-7.3.0-linux-x86_64.tar.gz

cd kibana-7.3.0-linux-x86_64/bin</code><code>./kibana

Access http://<em>ip</em>:5601 to confirm the UI loads.

Integrating Spring Boot Logs

Create spring-logback.xml to output logs to /log/sb-log.log, then package and run the application.

# Build</code><code>mvn package -Dmaven.test.skip=true</code><code># Run</code><code>java -jar sb-elk-start-0.0.1-SNAPSHOT.jar

Configure a Logstash Shipper to read the file and send to Redis:

input { file { path => ["/log/sb-log.log"] } }</code><pre><code>output { redis { host => "10.140.45.190" port => 6379 db => 8 data_type => "channel" key => "logstash_list_0" } }

Configure an Indexer Logstash to read from Redis, parse with Grok, and write to Elasticsearch:

input { redis { host => "192.168.142.131" port => 6379 db => 8 data_type => "channel" key => "sb-logback" } }</code><pre><code>filter { grok { match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NOTSPACE:threadName}\] %{LOGLEVEL:level}  %{DATA:logger} %{NOTSPACE:applicationName} -%{GREEDYDATA:msg}" } } }

output { elasticsearch { hosts => "localhost:9200" index => "logback" } }

Integrating Nginx Access Logs

Define a Grok pattern for Nginx logs:

%{IPV4:ip} \- \- \[%{HTTPDATE:time}\] "%{NOTSPACE:method} %{DATA:requestUrl} HTTP/%{NUMBER:httpVersion}" %{NUMBER:httpStatus} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:agent}"

Extend the Indexer configuration to handle both logback and nginx types using conditional blocks.

Running ELK as Daemons

Install supervisor and add programs for Elasticsearch, Logstash, and Kibana:

[program:elasticsearch]</code><code>environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"</code><code>directory=/home/elk/elk/elasticsearch</code><code>user=elk</code><code>command=/home/elk/elk/elasticsearch/bin/elasticsearch

[program:logstash]</code><code>environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"</code><code>directory=/home/elk/elk/logstash</code><code>user=elk</code><code>command=/home/elk/elk/logstash/bin/logstash -f /home/elk/elk/logstash/indexer-logstash.conf

[program:kibana]</code><code>environment=LS_HEAP_SIZE=5000m</code><code>directory=/home/elk/elk/kibana</code><code>user=elk</code><code>command=/home/elk/elk/kibana/bin/kibana

Reload Supervisor with sudo supervisorctl reload to start all components automatically.

Conclusion

The guide demonstrates how to build an ELK real‑time log analysis platform, integrate logs from Spring Boot and Nginx, and keep the services running reliably using Supervisor.