How to Combine SkyWalking and ELK for End‑to‑End Trace ID Logging

Background

When building a logging platform with SkyWalking + ELK, the ELK logs did not contain the Trace ID, making it impossible to follow a request across services.

Trace ID uniquely identifies a request in distributed tracing and links logs and performance data across multiple service nodes.

SkyWalking vs ELK

SkyWalking

SkyWalking is an APM system that provides distributed tracing, service performance analysis, and multi‑dimensional monitoring. It uses language‑specific agents or SDKs (Java, Go, Node, .NET) to automatically instrument code and send tracing data to the OAP server.

ELK

ELK (Elasticsearch, Logstash, Kibana) is a centralized log collection and analysis stack. Filebeat collects logs, Logstash processes and forwards them to Elasticsearch, and Kibana visualizes the data.

Can you use only SkyWalking?

Limitations

Data collection relies on agents/SDKs for supported languages and requires custom scripts for Nginx and MySQL.

Log visualization is weaker than Kibana; Kibana offers richer charts, search, and highlighting.

Can you use only ELK for traceability?

ELK does not natively attach a Trace ID, but you can add it by:

Embedding the Trace ID via the SkyWalking agent.

Injecting a Trace ID into MDC (Mapped Diagnostic Context) manually.

Using Kibana’s “View surrounding documents” to approximate related logs.

Embedding Trace ID with SkyWalking

SkyWalking provides a custom Logback layout class TraceIdPatternLogbackLayout that inserts the Trace ID into log messages.

Configuration example

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- Define a layout that includes the Trace ID -->
    <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
        <pattern>${CONSOLE_LOG_PATTERN:-%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) [%tid] %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}</pattern>
    </layout>

    <root level="debug">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

When the application runs, the console output includes the Trace ID (e.g., [%tid]). The logs can then be shipped to Elasticsearch via Filebeat and Logstash, allowing Kibana to query by Trace ID.

MDC approach

Generate a random identifier and store it in MDC, then reference it in the log pattern.

MDC.put("traceId", UUID.randomUUID().toString());

MDC is thread‑local, so all logs emitted by the same thread carry the same Trace ID, enabling correlation across services.

Kibana recent‑logs feature

Kibana’s “View surrounding documents” button shows logs that are temporally close to a selected entry. This can help locate related logs but may also include unrelated entries, making precise context extraction difficult.

Conclusion

SkyWalking excels at distributed tracing and APM, while ELK provides powerful log storage and visualization. By embedding SkyWalking’s Trace ID into ELK logs—either through the TraceIdPatternLogbackLayout or an MDC‑based solution—you can achieve end‑to‑end observability in microservice architectures.