Mastering ELK: Build a Real‑Time Log Analysis Platform for Spring Boot & Nginx

In the process of troubleshooting online issues, logs are essential. Modern micro‑service architectures scatter logs across many machines, making centralized, real‑time analysis crucial. This article introduces the open‑source ELK stack and shows how to build and use it.

ELK Overview

ELK consists of three components: Elasticsearch, Logstash, and Kibana.

Logstash

Logstash is a data collection engine with a pipeline architecture: input, filter, and output.

Input: collects data from files, syslog, databases, message queues, etc.

Filter: parses and transforms data, extracting fields.

Output: sends data to Elasticsearch or other destinations.

Elasticsearch

Elasticsearch is a distributed RESTful search and analytics engine that supports structured and unstructured queries, aggregations, high‑speed responses, horizontal scalability, resilience, and flexible data types.

Kibana

Kibana provides a browser‑based UI to visualize Elasticsearch data, create dashboards, and explore logs without writing code.

ELK Implementation Scheme

The typical workflow: Logstash (Shipper) reads logs from services and pushes them to a Redis channel; Logstash (Indexer) reads from Redis, parses the logs, and stores them in Elasticsearch; Kibana queries Elasticsearch and displays the logs.

ELK Platform Setup

Prerequisites: one Ubuntu machine (for the tutorial we install all components on the same host), JDK 1.8+, and download packages for Logstash, Elasticsearch, and Kibana.

Install Logstash

tar -xzvf logstash-7.3.0.tar.gz

Test with a simple pipeline:

cd logstash-7.3.0
bin/logstash -e 'input { stdin {} } output { stdout {} }'

A successful start shows a log entry.

Install Elasticsearch

tar -xzvf elasticsearch-7.3.0-linux-x86_64.tar.gz
cd elasticsearch-7.3.0/bin
./elasticsearch

Common issues: insufficient memory (adjust jvm.options) and running as root (create a dedicated user).

Install Kibana

tar -xzvf kibana-7.3.0-linux-x86_64.tar.gz
cd kibana-7.3.0-linux-x86_64/bin
./kibana

Access http://<ip>:5601 to verify the UI.

Using ELK with Spring Boot

Create a Spring Boot project and add a Logback configuration ( spring-logback.xml) that writes logs to /log/sb-log.log.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="ROLLING_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>/log/sb-log.log</file>
    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
      <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{25} -%msg%n</pattern>
    </encoder>
  </appender>
  <root level="INFO">
    <appender-ref ref="ROLLING_FILE"/>
  </root>
</configuration>

Package and deploy the jar on Ubuntu.

mvn package -DskipTests=true
java -jar sb-elk-start-0.0.1-SNAPSHOT.jar

Configure Shipper Logstash

input {
  file {
    path => ["/log/sb-log.log"]
  }
}
output {
  redis {
    host => "10.140.45.190"
    port => 6379
    db => 8
    data_type => "channel"
    key => "logstash_list_0"
  }
}

Configure Indexer Logstash

input {
  redis {
    host => "192.168.142.131"
    port => 6379
    db => 8
    data_type => "channel"
    key => "sb-logback"
  }
}
filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NOTSPACE:threadName}\] %{LOGLEVEL:level}  %{DATA:logger} %{NOTSPACE:applicationName} -%{NUMBER:timetaken}ms" }
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logback"
  }
  stdout {}
}

Using ELK with Nginx

Nginx access logs are stored in /var/log/nginx/access.log. Add a Grok pattern to parse them:

%{IPV4:ip} - - \[%{HTTPDATE:time}\] "%{NOTSPACE:method} %{DATA:requestUrl} HTTP/%{NUMBER:httpVersion}" %{NUMBER:httpStatus} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:agent}"

Extend the Indexer configuration to handle a second input type (type => "nginx") and apply a corresponding filter and output. The full configuration is available in the GitHub repository.

Running ELK as Background Services

Install Supervisor ( apt-get install supervisor) and add program sections for Elasticsearch, Logstash, and Kibana in /etc/supervisor/supervisord.conf:

[program:elasticsearch]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
directory=/home/elk/elk/elasticsearch
user=elk
command=/home/elk/elk/elasticsearch/bin/elasticsearch

[program:logstash]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
directory=/home/elk/elk/logstash
user=elk
command=/home/elk/elk/logstash/bin/logstash -f /home/elk/elk/logstash/indexer-logstash.conf

[program:kibana]
environment=LS_HEAP_SIZE=5000m
directory=/home/elk/elk/kibana
user=elk
command=/home/elk/elk/kibana/bin/kibana

Reload Supervisor ( sudo supervisorctl reload) to start all components automatically on boot. Use sudo supervisorctl start/stop [program_name] to manage individual services.

Conclusion

The tutorial covered ELK fundamentals, step‑by‑step platform installation, integration with Spring Boot and Nginx logs, and how to keep the stack running as background services. Source code and configuration files are available on GitHub.

References

Elastic official website

Logstash Best Practices

Installing JDK 1.8 on Ubuntu

Grok testing website

Common Grok patterns