How to Build a Real‑Time ELK Log Analysis Platform on Ubuntu

ELK Overview

ELK is an open‑source real‑time log analysis platform composed of three components: Elasticsearch, Logstash, and Kibana.

Logstash

Logstash collects server logs, normalizes data, and forwards it to a destination. Its pipeline consists of three stages:

Input – supports many sources such as files, syslog, MySQL, message queues, etc.

Filter – parses and transforms data into a common format.

Output – can send data to Elasticsearch or other sinks.

Elasticsearch

Elasticsearch is a distributed RESTful search and analytics engine with full‑text search, aggregations, millisecond‑level response times, and horizontal scalability from a laptop to petabyte‑scale clusters.

Kibana

Kibana provides a browser‑based UI for creating dashboards and visualizing Elasticsearch data without writing code.

Implementation Architecture

The solution deploys Logstash on each service machine (shipper role) to collect logs and push them to a Redis queue. A separate Logstash instance (indexer role) reads from Redis, parses logs, and stores them in Elasticsearch. Kibana reads from Elasticsearch for visualization.

Installation Prerequisites

Ubuntu machine (or VM) with JDK 1.7+ installed and the Logstash, Elasticsearch, and Kibana tarballs.

Install Logstash

tar -xzvf logstash-7.3.0.tar.gz
cd logstash-7.3.0
bin/logstash -e 'input { stdin {} } output { stdout {} }'

Verify the start by checking the console output.

Install Elasticsearch

tar -xzvf elasticsearch-7.3.0-linux-x86_64.tar.gz
cd elasticsearch-7.3.0
bin/elasticsearch
# Adjust config/jvm.options if memory is insufficient
curl http://localhost:9200

The curl command should return node information if Elasticsearch is running.

Install Kibana

tar -xzvf kibana-7.3.0-linux-x86_64.tar.gz
cd kibana-7.3.0
# Edit config/kibana.yml to point to Elasticsearch
bin/kibana

Access http://<em>ip</em>:5601 to confirm Kibana is running.

Integrating Spring Boot

Create a Spring Boot project and add a spring-logback.xml configuration that defines a ROLLING_FILE appender with a custom pattern.

<configuration debug="false">
  <appender name="ROLLING_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>/log/sb-log.log</file>
    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
      <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{25} ${appName} -%msg%n</pattern>
    </encoder>
  </appender>
  <root level="INFO">
    <appender-ref ref="ROLLING_FILE"/>
  </root>
</configuration>

Package and run the JAR on the Ubuntu server; the log file will be generated at /log/sb-log.log.

Shipper Logstash Configuration

input {
  file { path => ["/log/sb-log.log"] }
}
output {
  redis {
    host => "10.140.45.190"
    port => 6379
    db => 8
    data_type => "channel"
    key => "logstash_list_0"
  }
}

This configuration reads the Spring Boot log file and forwards each line to a Redis channel.

Indexer Logstash Configuration

input {
  redis {
    host => "192.168.142.131"
    port => 6379
    db => 8
    data_type => "channel"
    key => "sb-logback"
  }
}
filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NOTSPACE:threadName}\] %{LOGLEVEL:level}  %{DATA:logger} %{NOTSPACE:applicationName} -(?:.*=%{NUMBER:timetaken}ms|)" }
  }
}
output {
  elasticsearch { hosts => ["localhost:9200"] index => "logback" }
  stdout {}
}

The Grok filter extracts timestamp, thread name, log level, logger, application name, and optional request duration, then stores the structured event in Elasticsearch.

Integrating Nginx

Nginx access logs are typically located at /var/log/nginx/access.log. Use the following Grok pattern to parse them:

%{IPV4:ip} - - \[%{HTTPDATE:time}\] "%{NOTSPACE:method} %{DATA:requestUrl} HTTP/%{NUMBER:httpVersion}" %{NUMBER:httpStatus} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:agent}"

Extend the Indexer Logstash configuration to handle both logback and nginx inputs by adding a type field and conditional filter/output blocks:

input {
  redis {
    type => "logback"
    ...
  }
  redis {
    type => "nginx"
    ...
  }
}
filter {
  if [type] == "logback" {
    # existing logback grok
  }
  if [type] == "nginx" {
    grok { match => { "message" => "%{IPV4:ip} - - \[%{HTTPDATE:time}\] \"%{NOTSPACE:method} %{DATA:requestUrl} HTTP/%{NUMBER:httpVersion}\" %{NUMBER:httpStatus} %{NUMBER:bytes} \"%{DATA:referer}\" \"%{DATA:agent}\"" } }
  }
}
output {
  if [type] == "logback" {
    elasticsearch { hosts => ["localhost:9200"] index => "logback" }
  }
  if [type] == "nginx" {
    elasticsearch { hosts => ["localhost:9200"] index => "nginx" }
  }
}

Running ELK as Background Services

Install Supervisor ( apt-get install supervisor) and add program sections for the three components:

[program:elasticsearch]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
command=/home/elk/elk/elasticsearch/bin/elasticsearch

[program:logstash]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
command=/home/elk/elk/logstash/bin/logstash -f /home/elk/elk/logstash/indexer-logstash.conf

[program:kibana]
environment=LS_HEAP_SIZE=5000m
command=/home/elk/elk/kibana/bin/kibana

Reload Supervisor with sudo supervisorctl reload to start all components automatically on boot.

Verification

Start Elasticsearch, Kibana, the shipper Logstash, and the indexer Logstash using the commands above. Invoke Spring Boot endpoints (or generate Nginx traffic). Then open Kibana at http://<em>ip</em>:5601, add the appropriate indices (e.g., logback and nginx) in the Discover view, and verify that structured log entries appear.