How Major Companies Implement Distributed Tracing with TraceIdFilter, MDC, and SkyWalking

Pain Points

When viewing logs of a single Pod, multiple threads interleave their output, making it difficult to associate log lines with the originating request.

Log collection tools aggregate logs from many Pods into a single database, amplifying the interleaving problem.

Solution

TraceId + MDC

Frontend adds an X-App-Trace-Id request header. The value can be generated as timestamp+UUID to guarantee uniqueness.

Backend TraceIdFilter extracts the header value:

String traceId = httpServletRequest.getHeader(TRACE_ID_HEADER_KEY);
if (StrUtil.isBlank(traceId)) {
    traceId = UUID.randomUUID().toString();
}
MDC.put(MDC_TRACE_ID_KEY, traceId);

If the header is absent, the service generates a UUID (or Snowflake ID).

The traceId is stored in SLF4J MDC, and the Logback pattern uses %X{traceId} to print it.

Feign Integration

When making inter‑service calls, a Feign RequestInterceptor copies the MDC traceId into the outgoing request header:

@Override
public void apply(RequestTemplate template) {
    template.header(TRACE_ID_HEADER_KEY, MDC.get(MDC_TRACE_ID_KEY));
}

Multithread Adaptation

Please note that MDC as implemented by logback‑classic assumes moderate frequency of value insertion and that child threads do not automatically inherit the parent’s MDC.

Before a child thread runs, the parent’s MDC map is copied to the child; after execution, the child’s MDC is cleared.

Adapter for ThreadPoolExecutor:

public class MdcAwareThreadPoolExecutor extends ThreadPoolExecutor {
    @Override
    public void execute(Runnable command) {
        Map<String, String> parentThreadContextMap = MDC.getCopyOfContextMap();
        super.execute(MdcTaskUtils.adaptMdcRunnable(command, parentThreadContextMap));
    }
}

Adapter for Spring TaskDecorator:

@Component
public class MdcAwareTaskDecorator implements TaskDecorator {
    @Override
    public Runnable decorate(Runnable runnable) {
        Map<String, String> parentThreadContextMap = MDC.getCopyOfContextMap();
        return MdcTaskUtils.adaptMdcRunnable(runnable, parentThreadContextMap);
    }
}

Utility method MdcTaskUtils#adaptMdcRunnable() decorates the original Runnable, sets the parent MDC in the child before execution, and removes it afterwards:

public static Runnable adaptMdcRunnable(Runnable runnable, Map<String, String> parentThreadContextMap) {
    return () -> {
        if (MapUtils.isEmpty(parentThreadContextMap) || !parentThreadContextMap.containsKey(MDC_TRACE_ID_KEY)) {
            MDC.put(MDC_TRACE_ID_KEY, UUID.randomUUID().toString());
        } else {
            MDC.put(MDC_TRACE_ID_KEY, parentThreadContextMap.get(MDC_TRACE_ID_KEY));
        }
        try {
            runnable.run();
        } finally {
            MDC.remove(MDC_TRACE_ID_KEY);
        }
    };
}

SkyWalking Integration

SkyWalking provides apm-toolkit-logback-1.x to print the SkyWalking traceId in Logback. Using the TraceIdPatternLogbackLayout class, the %tid placeholder is mapped to the SkyWalking traceId.

Select the implementation class TraceIdPatternLogbackLayout.

In the Logback pattern, use %tid to output the SkyWalking traceId.

<configuration debug="false">
    <property name="pattern" value="[%d{yyyy-MM-dd HH:mm:ss.SSS}] %-5level [%thread] %logger %line [%X{traceId}] [%tid] - %msg%n"/>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
                <pattern>${pattern}</pattern>
            </layout>
        </encoder>
    </appender>
</configuration>

The layout registers two converters:

tid : LogbackPatternConverter converts %tid to the SkyWalking traceId.

sw_ctx : LogbackSkyWalkingContextPatternConverter converts %sw_ctx to the SkyWalking context.

Start the Java application with the SkyWalking agent:

-javaagent:/opt/tools/skywalking-agent.jar

MDC Principle

MDC is defined in the slf4j-api jar. All MDC operations delegate to the MDCAdapter interface.

public class MDC {
    static MDCAdapter mdcAdapter;
    public static void put(String key, String val) throws IllegalArgumentException {
        if (key == null) {
            throw new IllegalArgumentException("key parameter cannot be null");
        }
        if (mdcAdapter == null) {
            throw new IllegalStateException("MDCAdapter cannot be null.");
        }
        mdcAdapter.put(key, val);
    }
}

Logback provides LogbackMDCAdapter, which implements MDCAdapter using a ThreadLocal<Map<String, String>> to store the context.

public class LogbackMDCAdapter implements MDCAdapter {
    final ThreadLocal<Map<String, String>> copyOnThreadLocal = new ThreadLocal<>();
    // put/get implementation omitted for brevity
}

Logback Placeholders

During Logback initialization, PatternLayout registers converters for common placeholders, including MDC support via %X{key} or %mdc{key}, which resolve to MDC.get(key).

public class PatternLayout extends PatternLayoutBase<ILoggingEvent> {
    public static final Map<String, String> DEFAULT_CONVERTER_MAP = new HashMap<>();
    static {
        DEFAULT_CONVERTER_MAP.put("X", MDCConverter.class.getName());
        // other converters omitted for brevity
    }
}

The MDCConverter stores the key from the pattern (e.g., traceId) and returns the corresponding MDC value during conversion.

public class MDCConverter extends ClassicConverter {
    private String key;
    @Override
    public void start() {
        String[] keyInfo = extractDefaultReplacement(getFirstOption());
        key = keyInfo[0];
        super.start();
    }
    @Override
    public String convert(ILoggingEvent event) {
        Map<String, String> mdcPropertyMap = event.getMDCPropertyMap();
        if (key == null) {
            return outputMDCForAllKeys(mdcPropertyMap);
        }
        String value = mdcPropertyMap.get(key);
        return value != null ? value : "";
    }
}

The LoggingEvent implementation retrieves the MDC map from the underlying MDCAdapter when needed.

public class LoggingEvent implements ILoggingEvent {
    public Map<String, String> getMDCPropertyMap() {
        if (mdcPropertyMap == null) {
            MDCAdapter mdc = MDC.getMDCAdapter();
            if (mdc instanceof LogbackMDCAdapter) {
                mdcPropertyMap = ((LogbackMDCAdapter) mdc).getPropertyMap();
            } else {
                mdcPropertyMap = mdc.getCopyOfContextMap();
            }
        }
        return mdcPropertyMap != null ? mdcPropertyMap : Collections.emptyMap();
    }
}