How to Trace Requests Across Threads and Services with MDC, Feign, and SkyWalking

Pain Points

When viewing logs online, logs from multiple threads within the same Pod interleave, making it hard to trace which request a log belongs to. Collecting logs from many Pods into a single database makes the problem even worse.

Solution

TraceId + MDC

MDC (Mapped Diagnostic Context) allows storing a traceId per request.

Frontend adds an X-App-Trace-Id header to each request; the value can be generated as timestamp+UUID to guarantee uniqueness.

Backend extracts the header in a TraceIdFilter. If missing, generates a UUID or Snowflake ID, then puts it into MDC with MDC.put(MDC_TRACE_ID_KEY, traceId) and uses %X{traceId} in the Logback pattern to print it.

Feign interceptor copies the traceId from MDC to the outgoing request header.

For asynchronous tasks, the parent thread’s MDC map is copied to the child thread before execution and cleared afterwards. Example implementations for ThreadPoolExecutor and Spring TaskDecorator are provided.

public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
    throws IOException, ServletException {
    HttpServletRequest httpServletRequest = (HttpServletRequest) request;
    String traceId = httpServletRequest.getHeader(TRACE_ID_HEADER_KEY);
    if (StrUtil.isBlank(traceId)) {
        traceId = UUID.randomUUID().toString();
    }
    MDC.put(MDC_TRACE_ID_KEY, traceId);
    try {
        chain.doFilter(request, response);
    } finally {
        MDC.remove(MDC_TRACE_ID_KEY);
    }
}

@Override
public void apply(RequestTemplate template) {
    template.header(TRACE_ID_HEADER_KEY, MDC.get(MDC_TRACE_ID_KEY));
}

public class MdcAwareThreadPoolExecutor extends ThreadPoolExecutor {
    @Override
    public void execute(Runnable command) {
        Map<String, String> parentThreadContextMap = MDC.getCopyOfContextMap();
        super.execute(MdcTaskUtils.adaptMdcRunnable(command, parentThreadContextMap));
    }
}

@Component
public class MdcAwareTaskDecorator implements TaskDecorator {
    @Override
    public Runnable decorate(Runnable runnable) {
        Map<String, String> parentThreadContextMap = MDC.getCopyOfContextMap();
        return MdcTaskUtils.adaptMdcRunnable(runnable, parentThreadContextMap);
    }
}

SkyWalking Integration

SkyWalking provides apm-toolkit-logback-1.x which adds a TraceIdPatternLogbackLayout. The layout prints SkyWalking traceId using %tid and context using %sw_ctx. Configuration example:

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
    <property name="pattern" value="[%d{yyyy-MM-dd HH:mm:ss.SSS}] %-5level [%thread] %logger %line [%X{traceId}] [%tid] - %msg%n"/>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
                <pattern>${pattern}</pattern>
            </layout>
        </encoder>
    </appender>
</configuration>

The custom LogbackPatternConverter originally returns "TID: N/A". When the SkyWalking Java agent is attached ( -javaagent:/opt/tools/skywalking-agent.jar), it intercepts the convert() method to output the real traceId.

MDC Mechanism

MDC is defined in the slf4j-api jar. All MDC operations delegate to an MDCAdapter. Logback provides LogbackMDCAdapter which stores the map in a ThreadLocal. The adapter copies the map only when a write occurs to avoid unnecessary copying.

Logback’s PatternLayout registers converters for common placeholders ( %d, %thread, %X, etc.). The MDCConverter reads the key from the pattern, retrieves the value from the event’s MDC map, and returns a default value if the key is missing.

public class MDCConverter extends ClassicConverter {
    private String key;
    private String defaultValue = "";
    @Override
    public void start() {
        String[] keyInfo = extractDefaultReplacement(getFirstOption());
        key = keyInfo[0];
        if (keyInfo[1] != null) {
            defaultValue = keyInfo[1];
        }
        super.start();
    }
    @Override
    public String convert(ILoggingEvent event) {
        Map<String, String> mdcPropertyMap = event.getMDCPropertyMap();
        if (mdcPropertyMap == null) {
            return defaultValue;
        }
        if (key == null) {
            return outputMDCForAllKeys(mdcPropertyMap);
        } else {
            String value = mdcPropertyMap.get(key);
            return (value != null) ? value : defaultValue;
        }
    }
}