Build a Real-Time ELK Log Analysis Platform on Ubuntu: Step-by-Step Guide

ELK Overview

ELK is an open‑source real‑time log analysis platform composed of Elasticsearch, Logstash and Kibana.

Logstash

Logstash is a data‑collection engine that gathers logs from various sources, normalizes them and forwards them to a chosen destination. Its pipeline consists of input, filter and output stages.

Input: supports many sources such as files, syslog, MySQL, message queues, etc.

Filter: parses and transforms data into a structured format.

Output: can send data to Elasticsearch or other destinations.

Elasticsearch

Elasticsearch is a distributed RESTful search and analytics engine with features like multi‑type queries, aggregations, high speed, scalability from a laptop to petabyte‑scale clusters, resilience and flexible data handling.

Kibana

Kibana provides a browser‑based UI for visualizing Elasticsearch data, allowing quick creation and sharing of dashboards without writing code.

ELK Implementation Scheme

In a typical micro‑service environment logs are scattered across machines. The solution uses Logstash on each service (Shipper) to collect logs and push them to a Redis queue, an Indexer Logstash reads from Redis, filters the data and stores it in Elasticsearch, and Kibana runs on a separate server to display the logs.

Deploy a Logstash Shipper on each service to collect logs and push to Redis.

Deploy an Indexer Logstash to read from Redis, filter, and output to Elasticsearch.

Elasticsearch replicates data between master and replica nodes.

Kibana runs on a separate server to visualize the logs.

ELK Platform Setup

Prerequisites: an Ubuntu machine (or VM); JDK 1.7+; download packages for Logstash, Elasticsearch and Kibana.

Install Logstash

tar -xzvf logstash-7.3.0.tar.gz
cd logstash-7.3.0
bin/logstash -e 'input { stdin {} } output { stdout {} }'

Successful start is indicated by log output.

Install Elasticsearch

tar -xzvf elasticsearch-7.3.0-linux-x86_64.tar.gz
cd elasticsearch-7.3.0
bin/elasticsearch

If memory is insufficient, adjust jvm.options. Do not run as root; use a dedicated user. Verify with curl http://localhost:9200.

Install Kibana

tar -xzvf kibana-7.3.0-linux-x86_64.tar.gz
cd kibana-7.3.0-linux-x86_64
# edit config/kibana.yml to set elasticsearch.hosts, server.host, etc.
bin/kibana

Access http://<ip>:5601 to confirm the UI.

Using ELK with Spring Boot

Create a Spring Boot project and add a spring‑logback.xml that defines a ROLLING_FILE appender. Deploy the jar on Ubuntu.

Configure Shipper Logstash

input {
  file { path => ["/log/sb-log.log"] }
}
output {
  redis {
    host => "10.140.45.190"
    port => 6379
    db => 8
    data_type => "channel"
    key => "logstash_list_0"
  }
}

Configure Indexer Logstash

input {
  redis {
    host => "192.168.142.131"
    port => 6379
    db => 8
    data_type => "channel"
    key => "sb-logback"
  }
}
filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NOTSPACE:threadName}\] %{LOGLEVEL:level}  %{DATA:logger} %{NOTSPACE:applicationName} -(?:.*=%{NUMBER:timetaken}ms|)" }
  }
}
output {
  stdout {}
  elasticsearch { hosts => "localhost:9200" index => "logback" }
}

Grok parses timestamp, thread, level, logger, application name and response time.

Using ELK with Nginx

Collect Nginx access logs (default at /var/log/nginx/access.log) and add a Grok pattern:

%{IPV4:ip} - - \[%{HTTPDATE:time}\] "%{NOTSPACE:method} %{DATA:requestUrl} HTTP/%{NUMBER:httpVersion}" %{NUMBER:httpStatus} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:agent}"

Extend the Indexer configuration to handle two input types (logback and nginx) with conditional filters and outputs.

Running ELK as Daemons

Install supervisor and add programs for Elasticsearch, Logstash and Kibana in /etc/supervisor/supervisord.conf:

[program:elasticsearch]
command=/home/elk/elk/elasticsearch/bin/elasticsearch
directory=/home/elk/elk/elasticsearch
user=elk
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"

[program:logstash]
command=/home/elk/elk/logstash/bin/logstash -f /home/elk/elk/logstash/indexer-logstash.conf
directory=/home/elk/elk/logstash
user=elk
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"

[program:kibana]
command=/home/elk/elk/kibana/bin/kibana
directory=/home/elk/elk/kibana
user=elk
environment=LS_HEAP_SIZE=5000m

Reload supervisor with sudo supervisorctl reload to start all components automatically on boot.

Conclusion

This tutorial introduced the ELK stack, demonstrated how to set up a real‑time log analysis platform on Ubuntu, and showed integrations for Spring Boot (Logback) and Nginx logs.