How Misusing ThreadLocal Triggered a Production Outage and How to Fix It

Remember the ThreadLocal‑induced production incident that cost a bonus and threatened a job.

The Spark

While looking for optimization opportunities, the author added a local cache for user information using ThreadLocal, believing it was thread‑safe and would improve the order‑list query speed.

Implementation

A utility class was created to store and retrieve the user object:

/**
 * @author 一灯
 * @apiNote Local cache for user information
 */
public class ThreadLocalUtil {
    // Store user info in ThreadLocal
    private static final ThreadLocal<User> threadLocal = new ThreadLocal<>();

    /**
     * Get user information
     */
    public static User getUser() {
        // If ThreadLocal has no user, parse from request and set it
        if (threadLocal.get() == null) {
            threadLocal.set(UserUtil.parseUserFromRequest());
        }
        return threadLocal.get();
    }
}

The order‑list service then fetched the user from this utility:

/**
 * Get order list
 */
public List<Order> getOrderList() {
    // 1. Retrieve user from ThreadLocal cache
    User user = ThreadLocalUtil.getUser();
    // 2. Call user service to obtain orders
    return orderService.getOrderList(user);
}

Initial Success

After deployment, the interface response time dramatically improved, and the author imagined a promotion.

Disaster Strikes

Within an hour, users reported missing orders or seeing orders that belonged to others. Debugging revealed that ThreadLocal data disappears when a thread ends, but in containers like Tomcat, Jetty, SpringBoot, or Dubbo, requests are handled by a thread pool, so threads are reused across different users.

Consequently, cached user data leaked between requests, causing data over‑reach.

Solution

After using ThreadLocal, explicitly call remove() to clear the stored data, preferably in a finally block:

/**
 * Get order list
 */
public List<Order> getOrderList() {
    User user = ThreadLocalUtil.getUser();
    try {
        return orderService.getOrderList(user);
    } catch (Exception e) {
        throw new RuntimeException(e.getMessage());
    } finally {
        ThreadLocalUtil.removeUser(); // Clean up ThreadLocal
    }
}

The updated ThreadLocalUtil now includes a removal method:

/**
 * Delete user information
 */
public static void removeUser() {
    threadLocal.remove();
}

Incident Classification

If affected users exceed 100 k, erroneous data exceeds 100 k, or financial loss exceeds 1 M, the incident is classified as P1, impacting annual performance.

Key Takeaways

Don’t over‑engineer without understanding the underlying framework.

Know the limits of the tools you use; ThreadLocal is not a universal cache.

Prioritize safety over cleverness – aim for no mistakes rather than heroic feats.

Deep refactoring can be risky; ensure you fully grasp the implications.