Operations 21 min read

Master ELK Log Collection: Docker‑Compose, Logback & Logstash for SpringBoot

This guide walks you through building a production‑ready ELK log collection system using Docker‑Compose, configuring Logback to route debug, error, business, and record logs, setting up Logstash pipelines, and leveraging Kibana for powerful log visualization in SpringBoot applications.

macrozheng

Jun 18, 2020

Master ELK Log Collection: Docker‑Compose, Logback & Logstash for SpringBoot

ELK Environment Installation

ELK refers to Elasticsearch, Kibana, and Logstash. This guide provides the latest Docker‑Compose script and key installation points.

docker-compose script

version: '3'
services:
  elasticsearch:
    image: elasticsearch:6.4.0
    container_name: elasticsearch
    environment:
      - "cluster.name=elasticsearch"
      - "discovery.type=single-node"
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - TZ=Asia/Shanghai
    volumes:
      - /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins
      - /mydata/elasticsearch/data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300
  kibana:
    image: kibana:6.4.0
    container_name: kibana
    links:
      - elasticsearch:es
    depends_on:
      - elasticsearch
    environment:
      - "elasticsearch.hosts=http://es:9200"
      - TZ=Asia/Shanghai
    ports:
      - 5601:5601
  logstash:
    image: logstash:6.4.0
    container_name: logstash
    environment:
      - TZ=Asia/Shanghai
    volumes:
      - /mydata/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    depends_on:
      - elasticsearch
    links:
      - elasticsearch:es
    ports:
      - 4560:4560
      - 4561:4561
      - 4562:4562
      - 4563:4563

Installation key points

Run all services with docker-compose up -d.

If Elasticsearch fails to start, grant permission to the data directory: chmod 777 /mydata/elasticsearch/data.

Install Logstash JSON lines plugin: logstash-plugin install logstash-codec-json_lines.

Scenario Log Collection

Four log scenarios are defined:

Debug logs : all logs at DEBUG level and above, collected in development/testing environments.

Error logs : only logs at ERROR level, collected in all environments.

Business logs : logs printed in the application’s specific package.

Record logs : interface access records, useful for performance analysis.

Logback Configuration Details

To implement the scenarios, customize logback-spring.xml. The full configuration is shown below.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration>
<configuration>
    <!-- include default configuration -->
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
    <springProperty scope="context" name="APP_NAME" source="spring.application.name" defaultValue="springBoot"/>
    <property name="LOG_FILE_PATH" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/logs}"/>
    <springProperty name="LOG_STASH_HOST" scope="context" source="logstash.host" defaultValue="localhost"/>

    <!-- DEBUG file appender -->
    <appender name="FILE_DEBUG" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>DEBUG</level>
        </filter>
        <encoder>
            <pattern>${FILE_LOG_PATTERN}</pattern>
            <charset>UTF-8</charset>
        </encoder>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>${LOG_FILE_PATH}/debug/${APP_NAME}-%d{yyyy-MM-dd}-%i.log</fileNamePattern>
            <maxFileSize>${LOG_FILE_MAX_SIZE:-10MB}</maxFileSize>
            <maxHistory>${LOG_FILE_MAX_HISTORY:-30}</maxHistory>
        </rollingPolicy>
    </appender>

    <!-- ERROR file appender -->
    <appender name="FILE_ERROR" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <encoder>
            <pattern>${FILE_LOG_PATTERN}</pattern>
            <charset>UTF-8</charset>
        </encoder>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>${LOG_FILE_PATH}/error/${APP_NAME}-%d{yyyy-MM-dd}-%i.log</fileNamePattern>
            <maxFileSize>${LOG_FILE_MAX_SIZE:-10MB}</maxFileSize>
            <maxHistory>${LOG_FILE_MAX_HISTORY:-30}</maxHistory>
        </rollingPolicy>
    </appender>

    <!-- DEBUG to Logstash -->
    <appender name="LOG_STASH_DEBUG" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>DEBUG</level>
        </filter>
        <destination>${LOG_STASH_HOST}:4560</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>Asia/Shanghai</timeZone>
                </timestamp>
                <pattern>
                    {"project":"mall-tiny","level":"%level","service":"${APP_NAME:-}","pid":"${PID:-}","thread":"%thread","class":"%logger","message":"%message","stack_trace":"%exception{20}"}
                </pattern>
            </providers>
        </encoder>
        <connectionStrategy>
            <roundRobin>
                <connectionTTL>5 minutes</connectionTTL>
            </roundRobin>
        </connectionStrategy>
    </appender>

    <!-- ERROR to Logstash -->
    <appender name="LOG_STASH_ERROR" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <destination>${LOG_STASH_HOST}:4561</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>Asia/Shanghai</timeZone>
                </timestamp>
                <pattern>
                    {"project":"mall-tiny","level":"%level","service":"${APP_NAME:-}","pid":"${PID:-}","thread":"%thread","class":"%logger","message":"%message","stack_trace":"%exception{20}"}
                </pattern>
            </providers>
        </encoder>
        <connectionStrategy>
            <roundRobin>
                <connectionTTL>5 minutes</connectionTTL>
            </roundRobin>
        </connectionStrategy>
    </appender>

    <!-- BUSINESS to Logstash -->
    <appender name="LOG_STASH_BUSINESS" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>${LOG_STASH_HOST}:4562</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>Asia/Shanghai</timeZone>
                </timestamp>
                <pattern>
                    {"project":"mall-tiny","level":"%level","service":"${APP_NAME:-}","pid":"${PID:-}","thread":"%thread","class":"%logger","message":"%message","stack_trace":"%exception{20}"}
                </pattern>
            </providers>
        </encoder>
        <connectionStrategy>
            <roundRobin>
                <connectionTTL>5 minutes</connectionTTL>
            </roundRobin>
        </connectionStrategy>
    </appender>

    <!-- RECORD to Logstash -->
    <appender name="LOG_STASH_RECORD" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>${LOG_STASH_HOST}:4563</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>Asia/Shanghai</timeZone>
                </timestamp>
                <pattern>
                    {"project":"mall-tiny","level":"%level","service":"${APP_NAME:-}","class":"%logger","message":"%message"}
                </pattern>
            </providers>
        </encoder>
        <connectionStrategy>
            <roundRobin>
                <connectionTTL>5 minutes</connectionTTL>
            </roundRobin>
        </connectionStrategy>
    </appender>

    <!-- framework log level control -->
    <logger name="org.slf4j" level="INFO"/>
    <logger name="springfox" level="INFO"/>
    <logger name="io.swagger" level="INFO"/>
    <logger name="org.springframework" level="INFO"/>
    <logger name="org.hibernate.validator" level="INFO"/>

    <root level="DEBUG">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="LOG_STASH_DEBUG"/>
        <appender-ref ref="LOG_STASH_ERROR"/>
    </root>

    <logger name="com.macro.mall.tiny.component" level="DEBUG">
        <appender-ref ref="LOG_STASH_RECORD"/>
    </logger>

    <logger name="com.macro.mall" level="DEBUG">
        <appender-ref ref="LOG_STASH_BUSINESS"/>
    </logger>
</configuration>

Key point explanations

Using default log configuration

The default console appender is defined in console-appender.xml inside the Spring Boot JAR.

springProperty

Allows values from application.yml (e.g., Logstash host) to be injected into the XML.

filter

ThresholdFilter

discards logs below a level; LevelFilter accepts only a specific level.

appender

Three main appenders: ConsoleAppender, RollingFileAppender, and LogstashTcpSocketAppender for sending logs to Logstash.

logger

Only loggers with attached appenders are active. The configuration maps package names to the appropriate Logstash appender.

Logstash Configuration Details

Input listens on ports 4560‑4563 for the four log types, a filter processes record logs, and output indexes into Elasticsearch with type‑based indices.

input {
  tcp { mode => "server" host => "0.0.0.0" port => 4560 codec => json_lines type => "debug" }
  tcp { mode => "server" host => "0.0.0.0" port => 4561 codec => json_lines type => "error" }
  tcp { mode => "server" host => "0.0.0.0" port => 4562 codec => json_lines type => "business" }
  tcp { mode => "server" host => "0.0.0.0" port => 4563 codec => json_lines type => "record" }
}
filter {
  if [type] == "record" {
    mutate { remove_field => ["port", "host", "@version"] }
    json { source => "message" remove_field => ["message"] }
  }
}
output {
  elasticsearch {
    hosts => ["es:9200"]
    action => "index"
    codec => json
    index => "mall-tiny-%{type}-%{+YYYY.MM.dd}"
    template_name => "mall-tiny"
  }
}

SpringBoot Configuration

Environment‑specific application-*.yml files can override Logstash host and root logging level.

# application-dev.yml
logstash:
  host: localhost
logging:
  level:
    root: debug

# application-test.yml
logstash:
  host: 192.168.3.101
logging:
  level:
    root: debug

# application-prod.yml
logstash:
  host: logstash-prod
logging:
  level:
    root: info

Kibana Advanced Usage

After the ELK stack is running, use Kibana to create index patterns, discover logs per scenario, and search by fields. Screenshots illustrate the steps.