Trace Requests Across Threads with TraceId, MDC, and SkyWalking in Java

Pain Points

When viewing online logs, logs from multiple threads within the same Pod interleave, making it hard to trace each request's log information.

Log collection tools aggregate logs from many Pods into a single database, worsening the chaos.

Solution

TraceId + MDC

Frontend adds an X-App-Trace-Id request header for every request; the value can be generated as timestamp + UUID to guarantee uniqueness.

Backend extracts the header in a TraceIdFilter. If the header is missing, a UUID or Snowflake algorithm generates a traceId. The traceId is then stored in SLF4J MDC and printed via the logback pattern %X{traceId}:

public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
    HttpServletRequest httpServletRequest = (HttpServletRequest) request;
    String traceId = httpServletRequest.getHeader(TRACE_ID_HEADER_KEY);
    if (StrUtil.isBlank(traceId)) {
        traceId = UUID.randomUUID().toString();
    }
    MDC.put(MDC_TRACE_ID_KEY, traceId);
    try {
        chain.doFilter(request, response);
    } finally {
        MDC.remove(MDC_TRACE_ID_KEY);
    }
}

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
    <property name="pattern" value="[%d{yyyy-MM-dd HH:mm:ss.SSS}] %-5level [%thread] %logger %line [%X{traceId}] [%tid] - %msg%n"/>
</configuration>

Integrating Feign

When services call each other via Feign, a Feign RequestInterceptor copies the traceId from MDC into the outgoing request header:

@Override
public void apply(RequestTemplate template) {
    template.header(TRACE_ID_HEADER_KEY, MDC.get(MDC_TRACE_ID_KEY));
}

Multi‑thread Adaptation

Please note that MDC as implemented by logback-classic assumes that values are placed into the MDC with moderate frequency. Also note that a child thread does not automatically inherit a copy of the mapped diagnostic context of its parent.

Before a child thread executes, copy the parent MDC map; after execution, clear the child MDC. Example for ThreadPoolExecutor:

public class MdcAwareThreadPoolExecutor extends ThreadPoolExecutor {
    @Override
    public void execute(Runnable command) {
        Map<String, String> parentThreadContextMap = MDC.getCopyOfContextMap();
        super.execute(MdcTaskUtils.adaptMdcRunnable(command, parentThreadContextMap));
    }
}

Spring TaskDecorator adaptation:

@Component
public class MdcAwareTaskDecorator implements TaskDecorator {
    @Override
    public Runnable decorate(Runnable runnable) {
        Map<String, String> parentThreadContextMap = MDC.getCopyOfContextMap();
        return MdcTaskUtils.adaptMdcRunnable(runnable, parentThreadContextMap);
    }
}

The utility method decorates a Runnable to set the parent MDC before execution and remove it afterwards:

@Slf4j
public abstract class MdcTaskUtils {
    public static Runnable adaptMdcRunnable(Runnable runnable, Map<String, String> parentThreadContextMap) {
        return () -> {
            if (MapUtils.isEmpty(parentThreadContextMap) || !parentThreadContextMap.containsKey(MDC_TRACE_ID_KEY)) {
                MDC.put(MDC_TRACE_ID_KEY, UUID.randomUUID().toString());
            } else {
                MDC.put(MDC_TRACE_ID_KEY, parentThreadContextMap.get(MDC_TRACE_ID_KEY));
            }
            try {
                runnable.run();
            } finally {
                MDC.remove(MDC_TRACE_ID_KEY);
            }
        };
    }
}

Integrating SkyWalking

SkyWalking provides apm-toolkit-logback-1.x to print SkyWalking traceId. By using the TraceIdPatternLogbackLayout class, the log pattern can include both the custom X-App-Trace-Id and SkyWalking traceId via %tid:

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
    <property name="pattern" value="[%d{yyyy-MM-dd HH:mm:ss.SSS}] %-5level [%thread] %logger %line [%X{traceId}] [%tid] - %msg%n"/>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
                <pattern>${pattern}</pattern>
            </layout>
        </encoder>
    </appender>
</configuration>

The custom layout registers converters tid (maps to SkyWalking traceId) and sw_ctx (SkyWalking context).

MDC Principle

MDC is defined by the SLF4J MDCAdapter interface; Logback implements it with LogbackMDCAdapter using a ThreadLocal map.

public class MDC {
    public static void put(String key, String val) { mdcAdapter.put(key, val); }
}

The adapter stores a copy‑on‑write map to avoid contention between threads.

Logback Placeholders

Logback’s PatternLayout registers many converters, including X for MDC ( %X{key}), thread, level, etc.

DEFAULT_CONVERTER_MAP.put("X", MDCConverter.class.getName());
DEFAULT_CONVERTER_MAP.put("thread", ThreadConverter.class.getName());
// ... other converters

The MDCConverter reads the configured key and returns the corresponding value from the event’s MDC map:

public class MDCConverter extends ClassicConverter {
    private String key;
    @Override
    public void start() {
        String[] keyInfo = extractDefaultReplacement(getFirstOption());
        key = keyInfo[0];
        super.start();
    }
    @Override
    public String convert(ILoggingEvent event) {
        Map<String, String> map = event.getMDCPropertyMap();
        if (key == null) return outputMDCForAllKeys(map);
        String value = map.get(key);
        return value != null ? value : "";
    }
}