How to Build a Real-Time ELK Log Analysis Platform for Spring Boot and Nginx

ELK Overview

ELK is an open‑source real‑time log analysis platform composed of three components: Elasticsearch for storage and search, Logstash for data collection and processing, and Kibana for visualisation.

Logstash

Logstash collects logs from various sources, applies filters, and forwards the data to a destination such as Elasticsearch or Redis. Its pipeline consists of three stages: input, filter, and output.

Elasticsearch

Elasticsearch is a distributed RESTful search and analytics engine offering full‑text, structured, and geo queries, fast aggregation, high scalability, and flexible data types.

Kibana

Kibana provides a browser‑based UI for creating dashboards and exploring data stored in Elasticsearch.

Implementation Plan

To collect logs from multiple servers, a Logstash shipper runs on each service host, sending logs to a Redis queue. An indexer Logstash instance reads from Redis, parses the logs, and stores them in Elasticsearch. Kibana then visualises the indexed data.

ELK Platform Setup

Prerequisites

Ubuntu machine (all components installed on the same host for the tutorial)

JDK 1.7 or higher

Download the Logstash, Elasticsearch, and Kibana tarballs

Install Logstash

tar -xzvf logstash-7.3.0.tar.gz

Start a simple pipeline to verify the installation:

cd logstash-7.3.0
bin/logstash -e 'input { stdin {} } output { stdout {} }'

Enter Hello Logstash and you should see a JSON event confirming success.

Install Elasticsearch

tar -xzvf elasticsearch-7.3.0-linux-x86_64.tar.gz

cd elasticsearch-7.3.0
bin/elasticsearch

If the process fails due to insufficient memory, edit config/jvm.options to lower the heap size. Running as a non‑root user avoids permission errors.

Verify with:

curl http://localhost:9200

Install Kibana

tar -xzvf kibana-7.3.0-linux-x86_64.tar.gz

Edit config/kibana.yml to point to the Elasticsearch host:

elasticsearch.hosts: "http://ip:9200"
server.host: "0.0.0.0"

cd kibana-7.3.0-linux-x86_64/bin
./kibana

Open http://ip:5601 in a browser; the Kibana landing page indicates a successful start.

Integrating Spring Boot Logs

Create a spring-logback.xml file that defines a ROLLING_FILE appender writing logs to /log/sb-log.log with a pattern that includes timestamp, thread, level, logger, and application name.

Package the Spring Boot application and deploy it on the same Ubuntu host:

# Build
mvn package -DskipTests=true
# Run
java -jar sb-elk-start-0.0.1-SNAPSHOT.jar

Verify the log file contains entries such as:

2019-08-11 18:01:31.602 [http-nio-8080-exec-2] INFO c.i.s.aop.WebLogAspect sb-elk - 接口日志 POST请求测试接口结束调用:耗时=11ms,result=BaseResponse{code=10000, message='操作成功'}

Shipper Logstash (Log File → Redis)

input {
  file { path => "/log/sb-log.log" }
}
output {
  redis {
    host => "10.140.45.190"
    port => 6379
    db => 8
    data_type => "channel"
    key => "logstash_list_0"
  }
}

Indexer Logstash (Redis → Elasticsearch)

input {
  redis {
    host => "192.168.142.131"
    port => 6379
    db => 8
    data_type => "channel"
    key => "sb-logback"
  }
}
filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NOTSPACE:threadName}\] %{LOGLEVEL:level}  %{DATA:logger} %{NOTSPACE:applicationName} -(?:.*=%{NUMBER:timetaken}ms|)" }
  }
}
output {
  elasticsearch { hosts => "localhost:9200" index => "logback" }
  stdout {}
}

The Grok pattern extracts fields such as time, threadName, level, logger, applicationName, and timetaken.

Integrating Nginx Access Logs

Assuming Nginx is installed, its access log resides at /var/log/nginx/access.log. Use a Grok pattern like:

%{IPV4:ip} - - \[%{HTTPDATE:time}\] "%{NOTSPACE:method} %{DATA:requestUrl} HTTP/%{NUMBER:httpVersion}" %{NUMBER:httpStatus} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:agent}"

Indexer Configuration for Multiple Log Types

input {
  redis { type => "logback" ... }
  redis { type => "nginx" ... }
}
filter {
  if [type] == "logback" { ... }
  if [type] == "nginx" { grok { match => { "message" => "..." } } }
}
output {
  if [type] == "logback" { elasticsearch { index => "logback" } }
  if [type] == "nginx" { elasticsearch { index => "nginx" } }
}

Running ELK as Background Services

Use supervisor to keep Elasticsearch, Logstash, and Kibana running after the terminal is closed.

[program:elasticsearch]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
command=/home/elk/elk/elasticsearch/bin/elasticsearch
user=elk

[program:logstash]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
command=/home/elk/elk/logstash/bin/logstash -f /home/elk/elk/logstash/indexer-logstash.conf
user=elk

[program:kibana]
command=/home/elk/elk/kibana/bin/kibana
user=elk

After adding these sections to /etc/supervisor/supervisord.conf, run sudo supervisorctl reload. The services will start automatically on boot and can be managed individually with sudo supervisorctl start|stop|restart [program_name].

Conclusion

By following the steps above you have built a functional ELK stack, integrated logs from a Spring Boot application and Nginx, and configured the components to run continuously in the background, providing a powerful real‑time log analysis solution for micro‑service environments.