How to Combine SkyWalking and ELK for End-to-End Trace ID Logging

Background

When building a logging platform we chose SkyWalking + ELK, but discovered that ELK logs lack a Trace ID, making it impossible to trace the full error chain across services.

Trace ID is the unique identifier used in distributed tracing to correlate logs and performance data across multiple service nodes.

What Are SkyWalking and ELK?

SkyWalking : an Application Performance Monitoring (APM) system that provides distributed tracing, service performance analysis, and multi‑dimensional monitoring. It supports automatic instrumentation for Java, Go, Node, .NET, etc.

ELK : the Elasticsearch‑Logstash‑Kibana stack (often with Beats/Filebeat) for centralized log collection, storage, and visualization.

SkyWalking Components

Tracing : collects trace information from applications and sends it to the SkyWalking OAP server.

SkyWalking OAP : receives trace data, performs analysis, stores it in external storage (ES, MySQL, TiDB, etc.), and provides query capabilities.

Storage : persists trace data.

SkyWalking UI : a web interface for viewing trace data.

ELK Pipeline

Beats (Filebeat) : runs on the application side, collects logs and forwards them to Logstash.

Logstash : filters, parses, and forwards logs to Elasticsearch.

Elasticsearch : stores logs and builds indices for fast search.

Kibana : visualizes and queries logs.

Can We Use Only SkyWalking?

SkyWalking excels at service performance analysis and tracing, but has some limitations:

Data collection relies on agents or SDKs; integrating non‑Java services (e.g., Nginx, MySQL) requires custom scripts.

Log visualization is weaker than Kibana; Kibana offers richer charts, dashboards, and search features.

Can We Achieve Tracing with Only ELK?

ELK does not provide native trace IDs, but you can add them through three common approaches:

Embed Trace ID via SkyWalking agent.

Inject a generated ID into MDC (Mapped Diagnostic Context) and log it.

Manually add Trace ID in log messages (less accurate).

Method 1 – SkyWalking TraceIdPatternLogbackLayout

Configure Logback to use SkyWalking's custom layout and include [%tid] in the log pattern.

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- Define a layout that prints TraceId -->
    <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
        <pattern>${CONSOLE_LOG_PATTERN:-%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) [%tid] %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}</pattern>
    </layout>

    <root level="debug">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>

Running the application prints the Trace ID to the console, which can then be shipped by Filebeat → Logstash → Elasticsearch. Kibana can query logs by the Trace ID.

Method 2 – MDC Injection

Generate a UUID, put it into MDC, and reference it in the log pattern.

MDC.put("traceId", UUID.randomUUID().toString());

MDC stores data per thread, making the trace ID available to any code executing in that thread.

Method 3 – Kibana "View Surrounding Documents"

Kibana can display logs that are temporally close to a selected log, but this method cannot reliably associate logs that belong to the same request because many unrelated logs may appear in the same time window.

Conclusion

SkyWalking and ELK each play a distinct role: SkyWalking provides native distributed tracing, while ELK offers powerful log storage and visualization. By embedding SkyWalking’s Trace ID into ELK logs—either via the SkyWalking layout or MDC—you can achieve end‑to‑end observability in a micro‑service architecture.