How to Combine SkyWalking and ELK for End-to-End Trace ID Logging

Background

When building a logging platform, the author chose a combination of SkyWalking and ELK. The ELK stack (Elasticsearch, Logstash, Filebeat, Kibana) did not include the distributed Trace ID generated by SkyWalking, making it impossible to trace error chains across services.

Trace ID uniquely identifies a request or transaction in a micro‑service architecture, allowing all related logs and performance data to be correlated.

What Are SkyWalking and ELK?

SkyWalking : an Application Performance Monitoring (APM) system that provides distributed tracing, service performance analysis, and multi‑dimensional monitoring. It collects tracing data via agents or SDKs and stores it in a backend.

ELK : a centralized log management suite consisting of Filebeat (log collector), Logstash (pipeline), Elasticsearch (storage and indexing), and Kibana (visualization).

SkyWalking Architecture

The platform consists of four main components:

Tracing : agents collect trace data from applications and send it to the SkyWalking OAP server.

SkyWalking OAP Server : receives trace data, performs analysis, stores results, and provides query capabilities.

Storage : supports ES, MySQL, ShardingSphere, TiDB, H2, etc.

SkyWalking UI : a web UI for visualizing trace and metric data.

ELK Architecture

Log flow in ELK:

Beats (Filebeat) : runs on the application side, collects logs and forwards them to Logstash.

Logstash : filters, transforms, and routes logs to Elasticsearch.

Elasticsearch : stores logs and builds indexes for fast search.

Kibana : visualizes and queries logs.

Can We Use Only SkyWalking?

SkyWalking excels at service performance analysis and distributed tracing, but it has limitations:

Collection method : relies on agents or SDKs (gRPC) and may require custom scripts for Nginx or MySQL.

Visualization : its log exploration UI is less flexible than Kibana’s rich charts and search features.

Can We Use Only ELK for Tracing?

ELK does not provide native trace IDs, but you can achieve tracing through three common approaches:

Embed Trace ID via SkyWalking Agent.

Inject a Trace ID into MDC (Mapped Diagnostic Context) manually.

Rely on Kibana’s “recent logs” feature (less accurate).

1. Embedding Trace ID with SkyWalking

Configure TraceIdPatternLogbackLayout in logback‑spring.xml to add [%tid] to each log line.

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- Define a layout that includes TraceId -->
    <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
        <pattern>${CONSOLE_LOG_PATTERN:-%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) [%tid] %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}</pattern>
    </layout>

    <root level="debug">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

Running the application prints the Trace ID on the console, which can then be collected by Filebeat → Logstash → Elasticsearch. Kibana can query logs by this Trace ID.

2. MDC (Mapped Diagnostic Context) Approach

Generate a random ID and store it in MDC, then include it in the log pattern.

MDC.put("traceId", UUID.randomUUID().toString());

MDC is thread‑local storage, allowing any code in the same thread to retrieve the trace ID.

3. Kibana’s Recent Logs Feature

Kibana can display logs that are temporally close to a selected entry via the “View surrounding documents” button, but this method often mixes unrelated logs, making it hard to isolate the correct context.

Conclusion

SkyWalking and ELK each play a vital role in APM and log management. Although ELK does not natively support distributed tracing, integrating it with SkyWalking—either by embedding Trace IDs via the Logback layout or by using MDC—provides a complementary solution that enhances observability in micro‑service architectures.