Build a Custom Logging Service with Logback: From Appender to Queue

Logging Service is a system for collecting, processing, storing, and querying log information, acting as a record of system behavior to help developers quickly locate issues.

Two common solutions exist: (1) the ELK stack (Elasticsearch + Logstash + Kibana), where Logstash collects data, Elasticsearch stores it, and Kibana visualizes and monitors; (2) a custom log appender that transmits logs via UDP, RPC, or Kafka by overriding the Logback or Log4j appender.

Using the second approach, Logback outputs logs through a defined Appender; the Appender determines the output path, and implementing a custom Appender requires extending AppenderBase and overriding its append() method.

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
  <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
    <charset>${CHARSET}</charset>
    <pattern>${APP_PATTERN}</pattern>
  </encoder>
</appender>
<root level="INFO">
  <appender-ref ref="STDOUT"/>
</root>

Referencing the custom Appender in the root logger routes logs to the specified output.

Two design schemes are presented: a generic flow (client → Kafka → logging service → Elasticsearch) and a storage‑saving flow (client → UDP → service → compressed disk), the latter reducing two rounds of serialization and deserialization.

Simple demonstration

A sample BlockingQueueAppender stores log events in a BlockingQueue, uses a thread pool to consume them, and prints each message with a custom prefix.

package com.su4j.service;
import ch.qos.logback.core.AppenderBase;
import ch.qos.logback.classic.spi.ILoggingEvent;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class BlockingQueueAppender extends AppenderBase<ILoggingEvent> {
    private final BlockingQueue<String> logQueue = new LinkedBlockingQueue<>();
    private final ExecutorService executorService = Executors.newFixedThreadPool(4);

    @Override
    public void start() {
        super.start();
        executorService.submit(() -> {
            while (true) {
                try {
                    String logMessage = logQueue.take();
                    System.out.println("Custom outputter:" + logMessage);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }
        });
    }

    @Override
    protected void append(ILoggingEvent eventObject) {
        try {
            String logMessage = eventObject.getFormattedMessage();
            logQueue.put(logMessage);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    public void stopSender() {
        executorService.shutdown();
    }
}

Configuration example shows how to declare the BLOCKING_QUEUE appender and attach it to the root logger:

<appender name="BLOCKING_QUEUE" class="com.su4j.service.BlockingQueueAppender">
  <!-- optional custom parameters -->
</appender>
<root level="INFO">
  <appender-ref ref="BLOCKING_QUEUE"/>
</root>

Running the application prints logs prefixed with “Custom outputter:” confirming that the custom appender works as intended.

In summary, the article walks through the core mechanisms of a logging framework and demonstrates a complete custom Appender implementation, providing a practical reference for building a tailored logging service.