How to Set Up ELK Logging for Spring Boot and Nginx with Docker

ELK Overview

ELK (Elasticsearch, Logstash, Kibana) collects logs via Filebeat, parses them with Logstash, stores them in Elasticsearch, and visualizes the data with Kibana.

Elasticsearch provides NoSQL storage and indexing. Logstash gathers logs, transforms them, and outputs to Elasticsearch. Kibana displays the data in a web UI.

Spring Boot ELK Configuration

pom.xml Dependency

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>6.0</version>
</dependency>

logback-spring.xml

<configuration>
    <springProperty scope="context" name="application" source="spring.application.name"/>
    <springProperty scope="context" name="host" source="logstash.host"/>
    <springProperty scope="context" name="port" source="logstash.port"/>
    <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>${host}:${port}</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <pattern>
{
    "indexname":"${application}",
    "appname":"${application}",
    "timestamp": "%d{yyyy-MM-dd HH:mm:ss.SSS}",
    "thread": "%thread",
    "level": "%level",
    "logger_name": "%logger",
    "traceId": "%X{traceId}",
    "message": "%msg",
    "stack_trace": "%exception"
}
                </pattern>
            </providers>
        </encoder>
    </appender>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
        <encoder>
            <pattern>%date [%thread]-[%X{traceId}] %-5level [%logger{50}] %file:%line - %msg%n</pattern>
            <charset>UTF-8</charset>
        </encoder>
    </appender>
    <root level="info">
        <appender-ref ref="LOGSTASH"/>
        <appender-ref ref="CONSOLE"/>
    </root>
</configuration>

application.yml Logstash Settings

logstash:
  host: 10.136.0.51
  port: 5000

Vue Frontend Log Collection with Nginx

Dockerfile (Test Environment)

# Filebeat collects logs
VOLUME ["/var/log/nginx"]
WORKDIR /app
ADD https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.15.1-linux-x86_64.tar.gz /app
RUN tar -zxvf filebeat-7.15.1-linux-x86_64.tar.gz
RUN rm -rf filebeat-7.15.1-linux-x86_64.tar.gz
COPY filebeat_test.yml /app/filebeat-7.15.1-linux-x86_64
COPY run.sh /app/filebeat-7.15.1-linux-x86_64
WORKDIR /app/filebeat-7.15.1-linux-x86_64
RUN mv filebeat_test.yml filebeat.yml
RUN chmod -R 777 /app/filebeat-7.15.1-linux-x86_64/run.sh
ENTRYPOINT ["/app/filebeat-7.15.1-linux-x86_64/run.sh"]

Dockerfile (Production Environment)

# Filebeat collects logs
VOLUME ["/var/log/nginx"]
WORKDIR /app
ADD https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.15.1-linux-x86_64.tar.gz /app
RUN tar -zxvf filebeat-7.15.1-linux-x86_64.tar.gz
RUN rm -rf filebeat-7.15.1-linux-x86_64.tar.gz
COPY filebeat_prod.yml /app/filebeat-7.15.1-linux-x86_64
COPY run.sh /app/filebeat-7.15.1-linux-x86_64
WORKDIR /app/filebeat-7.15.1-linux-x86_64
RUN mv filebeat_prod.yml filebeat.yml
RUN chmod -R 777 /app/filebeat-7.15.1-linux-x86_64/run.sh
ENTRYPOINT ["/app/filebeat-7.15.1-linux-x86_64/run.sh"]

filebeat.yml Example

filebeat.inputs:
  - type: log
    paths:
      - /var/log/nginx/access.log
setup.ilm.enabled: false
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
output.elasticsearch:
  hosts: ["10.136.0.51:9200"]
  username: "elastic"
  password: "123456"
  index: "iam-portal-%{+yyyy.MM.dd}"

run.sh Startup Script

#!/bin/bash
set -m
nginx -g "daemon off;" &
./filebeat -e -c /app/filebeat-7.15.1-linux-x86_64/filebeat.yml
fg %1

Creating Kibana Index Patterns

After logs are stored in Elasticsearch, create an index pattern in Kibana to view them. Use the Discover tab to query logs with KQL or Lucene syntax.