How to Use Spring Cloud Sleuth for Distributed Log Tracing

Overview

In a microservice architecture a single client request traverses multiple service nodes, forming a complex call chain where latency or errors in any segment can cause the whole request to fail. Spring Cloud Sleuth provides a complete solution for recording these inter‑service calls and attaching trace and span IDs to logs, while Zipkin visualizes the collected data.

Key Concepts

Span : the basic unit of work, e.g., sending an RPC or receiving a response. A span records timestamps, annotations, and parent/child relationships.

Trace : a tree of spans representing the entire request flow.

Annotation/Event : markers such as cs (client sent), sr (server received), ss (server sent), and cr (client received) that allow calculation of network latency and processing time.

Demo Project Setup

1. Create a Spring Boot project using version 2.6.8 and Spring Cloud 2021.0.3.

<properties>
    <maven.compiler.source>8</maven.compiler.source>
    <maven.compiler.target>8</maven.compiler.target>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <springboot.version>2.6.8</springboot.version>
    <springcloud.version>2021.0.3</springcloud.version>
</properties>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-dependencies</artifactId>
            <version>${springboot.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>${springcloud.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-sleuth</artifactId>
    </dependency>
</dependencies>

2. Add application.yml to set the server port and enable DEBUG logging for the DispatcherServlet:

server:
  port: 8989

spring:
  application:
    name: sleuth-demo
  zipkin:
    base-url: http://localhost:9411/
  sleuth:
    sampler:
      probability: 1
      rate: 10
logging:
  level:
    org.springframework.web.servlet.DispatcherServlet: DEBUG

3. Define a simple controller that logs a message:

@RestController
public class ExampleController {
    private static final Logger log = LoggerFactory.getLogger(ExampleController.class);

    @RequestMapping("/")
    public String home() {
        log.info("Hello world!");
        return "Hello World!";
    }
}

Run the application and access http://localhost:8989/. The console shows a log line that includes the trace and span IDs, e.g., [sleuth-demo,a79df4de56426209,a79df4de56426209].

Zipkin Setup

Download the latest Zipkin server JAR (2.23.18) from Maven Central and start it: java -jar zipkin-server-2.23.18-exec.jar Open http://localhost:9411/zipkin/ to view the UI. When the Sleuth‑enabled services send data, Zipkin displays the trace tree with colored spans (A‑G).

Building a Two‑Service Call Chain

Create a provider service (port 8990) and a consumer service (port 8989). The provider exposes /user/getUserList. The consumer uses OpenFeign to call the provider:

@FeignClient(value = "userService", url = "http://localhost:8990")
public interface UserService {
    @GetMapping("/user/getUserList")
    String getUserList();
}

@RestController
public class ExampleController {
    @Resource
    private UserService userService;

    @RequestMapping("/")
    public String home() {
        log.info("Hello world!");
        return userService.getUserList();
    }
}

After starting both services, invoke http://localhost:8989/. The trace ID shown in the logs of both services is identical, proving that Sleuth propagates the same trace across service boundaries. Zipkin UI visualizes the combined trace.

Exception Demonstration

Introduce a division‑by‑zero error in the provider’s getUserList method, restart the service, and call the consumer again. The logs now contain the exception stack trace together with the trace ID, allowing rapid pinpointing of the failing service by searching the trace ID in log files.

Sleuth Configuration Details

By default Sleuth sends data to localhost:9411. For distributed deployments set spring.zipkin.baseUrl to the Zipkin endpoint. Sampling can be tuned with spring.sleuth.sampler.probability (0‑1) or spring.sleuth.sampler.rate (max reports per second).

Sending Traces via Message Queues

Instead of HTTP, Sleuth can publish spans to RabbitMQ, Kafka, or ActiveMQ. Example RabbitMQ configuration:

spring:
  zipkin:
    base-url: http://localhost:9411/
    sender:
      type: rabbit
  rabbitmq:
    host: 127.0.0.1
    port: 5672
    username: guest
    password: guest
    listener:
      direct:
        retry:
          enabled: true
  sleuth:
    sampler:
      probability: 1
      rate: 10

Start Zipkin with RabbitMQ support:

java -jar zipkin-server-2.23.18-exec.jar \
  --RABBIT_ADDRESSES=127.0.0.1:5672 \
  --RABBIT_USER=guest \
  --RABBIT_PASSWORD=guest

Traces appear in Zipkin UI exactly as with HTTP transport.

Persisting Zipkin Data

Without persistence, stopping the Zipkin JAR clears all collected spans. Zipkin supports MySQL and Elasticsearch storage. For MySQL, create the schema from mysql.sql and start Zipkin with:

java -jar zipkin-server-2.23.18-exec.jar \
  --STORAGE_TYPE=mysql \
  --MYSQL_HOST=127.0.0.1 \
  --MYSQL_TCP_PORT=3306 \
  --MYSQL_DB=zipkin \
  --MYSQL_USER=root \
  --MYSQL_PASS=root

Injecting Trace IDs into Logback

Spring Boot’s default logback pattern includes ${LOG_DATEFORMAT_PATTERN}. When Sleuth is on the classpath, TraceEnvironmentPostProcessor modifies this pattern to prepend the trace ID, so every log line contains the ID without custom configuration.

For projects that use a custom logback.xml, add the Sleuth‑aware pattern:

<property name="CONSOLE_LOG_PATTERN" value="%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>

Other Distributed Tracing Solutions

CAT – comprehensive monitoring platform used by several large Chinese companies.

SkyWalking – open‑source, Apache‑incubated, focuses on tracing and performance monitoring with non‑intrusive instrumentation.

Pinpoint – Korean open‑source tracing system with strong UI, also non‑intrusive.

All of them can store traces in MySQL, Elasticsearch, or other backends.