Build a Real-Time ELK Log Analysis Platform on Ubuntu: Step-by-Step Guide

In troubleshooting online anomalies, logs are essential. In microservice architectures logs are scattered, making collection difficult. A unified real‑time log analysis platform such as ELK can greatly improve efficiency.

ELK Overview

ELK consists of three open‑source components: Elasticsearch, Logstash and Kibana.

Logstash

Logstash is a data‑collection engine that gathers logs from various sources, normalises them and forwards them to a destination.

Its pipeline has three stages:

Input – collects data from files, syslog, databases, message queues, etc.

Filter – parses and transforms data into a structured format.

Output – writes data to Elasticsearch or other sinks.

Elasticsearch

Elasticsearch is a distributed RESTful search and analytics engine with features such as full‑text search, aggregations, high speed, scalability, resilience and support for many data types.

Kibana

Kibana provides a browser‑based UI for visualising Elasticsearch data, creating dashboards and exploring logs without writing code.

ELK Implementation Diagram

Platform Setup

All components can be installed on a single Ubuntu machine for a simple tutorial.

Install Logstash

tar -xzvf logstash-7.3.0.tar.gz
cd logstash-7.3.0
bin/logstash -e 'input { stdin {} } output { stdout {} }'

Successful start is indicated by log output.

Install Elasticsearch

tar -xzvf elasticsearch-7.3.0-linux-x86_64.tar.gz
cd elasticsearch-7.3.0
bin/elasticsearch

Common startup issues: insufficient JVM memory and running as root. Adjust config/jvm.options or create a non‑root user.

curl http://localhost:9200

Install Kibana

tar -xzvf kibana-7.3.0-linux-x86_64.tar.gz
cd kibana-7.3.0
# edit config/kibana.yml to set elasticsearch.hosts, server.host, credentials
bin/kibana

Integrating Spring Boot Logs

Create spring‑logback.xml with a rolling‑file appender that writes lines such as:

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
  ...
  <appender name="ROLLING_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
      <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{25} %appName -%msg%n</pattern>
    </encoder>
  </appender>
</configuration>

Package and run the Spring Boot jar on the Ubuntu host.

Shipper Logstash (Log collection)

input {
  file { path => ["/log/sb-log.log"] }
}
output {
  redis { host => "10.140.45.190" port => 6379 db => 8 data_type => "channel" key => "logstash_list_0" }
}

Indexer Logstash (Log processing)

input {
  redis { host => "192.168.142.131" port => 6379 db => 8 data_type => "channel" key => "sb-logback" }
}
filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NOTSPACE:threadName}\] %{LOGLEVEL:level}  %{DATA:logger} %{NOTSPACE:applicationName} -(?:.*=%{NUMBER:timetaken}ms|)" }
  }
}
output {
  elasticsearch { hosts => "localhost:9200" index => "logback" }
  stdout {}
}

The Grok filter extracts timestamp, thread, level, logger, application name and request time.

Viewing Logs in Kibana

After starting Elasticsearch, Kibana and both Logstash instances, add the logback index in Kibana Discover to see structured logs.

Adding Nginx Logs

Define a Grok pattern for Nginx access logs:

%{IPV4:ip} - - [%{HTTPDATE:time}] "%{NOTSPACE:method} %{DATA:requestUrl} HTTP/%{NUMBER:httpVersion}" %{NUMBER:httpStatus} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:agent}"

Extend the Indexer configuration to handle two input types (logback and nginx) with conditional filters and outputs:

input {
  redis { type => "logback" ... }
  redis { type => "nginx" ... }
}
filter {
  if [type] == "logback" { ... }
  if [type] == "nginx" { ... }
}
output {
  if [type] == "logback" { ... }
  if [type] == "nginx" { ... }
}

Running ELK as Daemons

Use Supervisor to manage Elasticsearch, Logstash and Kibana as background services.

[program:elasticsearch]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
directory=/home/elk/elk/elasticsearch
user=elk
command=/home/elk/elk/elasticsearch/bin/elasticsearch

[program:logstash]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
directory=/home/elk/elk/logstash
user=elk
command=/home/elk/elk/logstash/bin/logstash -f /home/elk/elk/logstash/indexer-logstash.conf

[program:kibana]
environment=LS_HEAP_SIZE=5000m
directory=/home/elk/elk/kibana
user=elk
command=/home/elk/elk/kibana/bin/kibana

Reload Supervisor (e.g., sudo supervisorctl reload) to start all components automatically on boot.

Conclusion

The tutorial demonstrates how to build a real‑time ELK log analysis platform on Ubuntu, integrate Spring Boot and Nginx logs, and keep the services running continuously.