How to Build a Real-Time ELK Log Analysis Platform on Ubuntu

ELK Overview

During online incident troubleshooting, log querying is indispensable. In modern micro‑service architectures logs are scattered across many machines, making retrieval difficult. A unified real‑time log analysis platform dramatically improves debugging efficiency.

ELK Introduction

ELK is an open‑source real‑time log analysis stack composed of three parts: Elasticsearch , Logstash and Kibana .

Logstash

Logstash is a data‑collection engine with a real‑time pipeline. It gathers logs from various sources, normalises them, and forwards them to a chosen destination.

Input – collects data from files, syslog, MySQL, message queues, etc.

Filter – parses and transforms data into a structured format.

Output – sends data to Elasticsearch or other targets.

Elasticsearch

Elasticsearch (ES) is a distributed RESTful search and analytics engine with features such as multi‑type queries, aggregation, high speed (millisecond response for billions of records), scalability from a laptop to PB‑scale clusters, resilience, and flexible data type support.

Kibana

Kibana provides a browser‑based UI for visualising Elasticsearch data. It enables quick creation and sharing of dynamic dashboards without writing code or additional infrastructure.

ELK Implementation Scheme

The typical workflow is:

Deploy Logstash (Shipper role) on each micro‑service host to collect log files and push them to a Redis queue.

Deploy another Logstash (Indexer role) to read from Redis, parse with filters, and store into Elasticsearch.

Elasticsearch master‑slave nodes synchronise data.

Kibana reads from Elasticsearch and displays logs to users.

ELK Platform Setup

Prerequisites:

One Ubuntu machine (the tutorial installs all three components on the same host).

JDK 1.7+ installed.

Download the Logstash, Elasticsearch and Kibana tarballs.

Install Logstash

tar -xzvf logstash-7.3.0.tar.gz

Test with a simple pipeline:

cd logstash-7.3.0
bin/logstash -e 'input { stdin {} } output { stdout {} }'

Successful start is indicated by log output (see image).

Install Elasticsearch

tar -xzvf elasticsearch-7.3.0-linux-x86_64.tar.gz

Start Elasticsearch:

cd elasticsearch-7.3.0
bin/elasticsearch

Common issues:

Insufficient memory – adjust config/jvm.options to match available RAM.

Running as root – create a non‑root user and start Elasticsearch with that account.

Verify start with curl http://localhost:9200 which returns JSON information about the node.

Install Kibana

tar -xzvf kibana-7.3.0-linux-x86_64.tar.gz

Edit config/kibana.yml to point to the Elasticsearch host, then start:

cd kibana-7.3.0-linux-x86_64/bin
./kibana

Access http://<i>ip</i>:5601; a successful UI indicates Kibana is running.

Using ELK with Spring Boot

Create a Spring Boot project and add a spring-logback.xml configuration that defines a ROLLING_FILE appender writing logs to /log/sb-log.log.

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
    ...
    <appender name="ROLLING_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{25} ${appName} -%msg%n</pattern>
        </encoder>
    </appender>
    ...
</configuration>

Package and deploy the jar on Ubuntu:

# Build
mvn package -Dmaven.test.skip=true
# Deploy
java -jar sb-elk-start-0.0.1-SNAPSHOT.jar

Log file /log/sb-log.log should contain entries such as:

2019-08-11 18:01:31.602 [http-nio-8080-exec-2] INFO  c.i.s.aop.WebLogAspect sb-elk -接口日志 POST请求测试接口结束调用:耗时=11ms,result=BaseResponse{code=10000, message='操作成功'}

Configure Shipper Logstash

input {
    file { path => ["/log/sb-log.log"] }
}
output {
    redis { host => "10.140.45.190" port => 6379 db => 8 data_type => "channel" key => "logstash_list_0" }
}

This Logstash reads the Spring Boot log file and pushes raw lines to a Redis channel.

Configure Indexer Logstash

input {
    redis { host => "192.168.142.131" port => 6379 db => 8 data_type => "channel" key => "sb-logback" }
}
filter {
    grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NOTSPACE:threadName}\] %{LOGLEVEL:level}  %{DATA:logger} %{NOTSPACE:applicationName} -(?:.*=%{NUMBER:timetaken}ms|)" }
    }
}
output {
    stdout {}
    elasticsearch { hosts => "localhost:9200" index => "logback" }
}

The filter parses timestamps, thread names, log level, logger, application name and execution time using Grok patterns.

View Results

Start Elasticsearch, Kibana, the Indexer Logstash and the Shipper Logstash (using the commands shown earlier). Open Kibana, add the logback index in the Discover view, and you will see structured log entries.

Using ELK with Nginx

Nginx access logs (default at /var/log/nginx/access.log) can be parsed with a Grok pattern:

%{IPV4:ip} - - \[%{HTTPDATE:time}\] "%{NOTSPACE:method} %{DATA:requestUrl} HTTP/%{NUMBER:httpVersion}" %{NUMBER:httpStatus} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:agent}"

To handle both Spring Boot and Nginx logs, the Indexer Logstash input is defined with a type field, and conditional filters/output blocks route each type to the appropriate parsing and Elasticsearch index.

input {
    redis { type => "logback" ... }
    redis { type => "nginx" ... }
}
filter {
    if [type] == "logback" { ... }
    if [type] == "nginx" { ... }
}
output {
    if [type] == "logback" { elasticsearch { index => "logback" } }
    if [type] == "nginx" { elasticsearch { index => "nginx" } }
}

After updating the Shipper configuration to send both log files to Redis, restart the Logstash instances and add the nginx index in Kibana to view Nginx logs alongside Spring Boot logs.

Running ELK as Background Services

Running each component in the foreground is impractical. Using supervisor on Ubuntu allows ELK to start automatically and stay alive after logout.

[program:elasticsearch]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
directory=/home/elk/elk/elasticsearch
user=elk
command=/home/elk/elk/elasticsearch/bin/elasticsearch

[program:logstash]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
directory=/home/elk/elk/logstash
user=elk
command=/home/elk/elk/logstash/bin/logstash -f /home/elk/elk/logstash/indexer-logstash.conf

[program:kibana]
environment=LS_HEAP_SIZE=5000m
directory=/home/elk/elk/kibana
user=elk
command=/home/elk/elk/kibana/bin/kibana

After adding these sections to /etc/supervisor/supervisord.conf, run sudo supervisorctl reload to start all three services. Individual programs can be managed with sudo supervisorctl start|stop|restart program_name.

Conclusion

This tutorial introduced the ELK stack, demonstrated how to build a real‑time log analysis platform on Ubuntu, and showed integration with both Spring Boot (Logback) and Nginx logs.