Integrating SkyWalking Trace ID with ELK for Distributed Tracing in Microservices

This article discusses the integration of SkyWalking and ELK for building a comprehensive logging and tracing platform in microservices architecture.

Background: When building a logging platform using SkyWalking + ELK, the author discovered that ELK logs lacked Trace ID, making it impossible to trace the complete error chain in code execution.

SkyWalking vs ELK: SkyWalking is an Application Performance Monitoring (APM) system focused on distributed tracing, service performance analysis, and multi-dimensional monitoring. It supports automated code instrumentation and can track inter-service call relationships. ELK (Elasticsearch + Logstash + Filebeat + Kibana) provides centralized log management and analysis, with Kibana as the visualization platform.

Why Use Both: While SkyWalking excels at service performance analysis and distributed tracing, it has limitations: (1) It requires SDK or Agent for data collection and uses gRPC for backend communication; (2) For Nginx and MySQL logs, additional Lua scripts are needed; (3) Log visualization capabilities are limited compared to Kibana's rich options like line charts and pie charts; (4) Log search and display capabilities are weaker than Kibana's comprehensive search with highlighting support.

Solutions for Adding Trace ID to ELK:

1. SkyWalking Embeds Trace ID: Using SkyWalking's custom log layout class TraceIdPatternLogbackLayout , the Trace ID can be embedded into logs. Configure in logback-spring.xml to use this layout with [%tid] in the log pattern:

<configuration> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <encoder> <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n</pattern> </encoder> </appender> <!-- Define a log layout with TraceId --> <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout"> <pattern>${CONSOLE_LOG_PATTERN:-%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) [%tid] %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}</pattern> </layout> <root level="debug"> <appender-ref ref="STDOUT" /> </root> </configuration>

2. MDC Solution: Generate a random ID as traceId and put it into MDC:

MDC.put("traceId", UUID.randomUUID().toString());

MDC (Mapped Diagnostic Context) uses Thread-Local Storage to maintain context data specific to each thread.

3. Kibana's Recent Logs: Using "View surrounding documents" feature is inaccurate because similar logs in the same time period can cause confusion.

Conclusion: SkyWalking and ELK each play important roles in APM and log management. Although native ELK doesn't support distributed tracing directly, integrating with SkyWalking can complement their strengths and improve observability in microservices architecture.