How Virtual Threads Cut Latency from 50s to 1s in a Real Java 21 Service

A Real-World Slow Endpoint Scenario

The team inherited an endpoint that serially calls three external services, each averaging ~1 second response time. Running on a fixed thread pool of 200 platform threads, daytime traffic caused response times to climb to 50 seconds with constant timeout alerts. Code review revealed no algorithmic errors; the bottleneck was that each 1-second wait consumed a costly OS thread. Adding machines, tuning timeouts, and rate limiting failed to fix the late‑peak instability until the perspective shifted from how code is written to how threads wait .

After Java 21 standardized virtual threads (JEP 444), the blocking code was rewritten with virtual threads — downstream calls unchanged — and latency dropped from 50 seconds to just over 1 second.

Reproducing the Problem

To reproduce, the author replaces real downstream calls with Thread.sleep(1000). The following runs on a fixed 200‑thread pool; each request serially waits for three 1‑second calls:

ExecutorService pool = Executors.newFixedThreadPool(200);

List<CompletableFuture<String>> calls = IntStream.range(0, 3)
    .mapToObj(i -> CompletableFuture.supplyAsync(() -> {
        try {
            Thread.sleep(1000); // simulate 1‑second external call
        } catch (InterruptedException ignored) {}
        return "resp-" + i;
    }, pool))
    .toList();

calls.forEach(f -> f.join()); // serial wait, ~3 seconds per request

With 200 threads, three serial calls take ~3 seconds per request. Under concurrency, requests queue in batches; 100 concurrent requests require multiple batches, pushing tail latency to 50 seconds. The root cause is not slow code but the fact that 1‑second waits are spread across a limited pool of expensive platform threads — increasing the pool size cannot overcome massive blocking.

Why Thread Pools Hit a Ceiling

Platform threads are 1:1 mapped to OS threads; scheduling, stack, and context switches live in the kernel. Each platform thread defaults to a 1 MB stack, so tens of thousands exhaust memory, and more threads mean more CPU spent on switching rather than work. Thread pools were designed to avoid this: reuse a small set of threads instead of creating one per request. This works for CPU‑bound or fast‑downstream scenarios, but when downstream is slow and waiting is frequent, the pool itself becomes a hard ceiling — you cannot simply create 10,000 platform threads. The production 200‑thread pool saturated throughput because threads were locked in blocking. Worse, with many threads the kernel scheduler spends significant CPU on fair switching, further diluting useful compute capacity. It’s like a single‑lane road: widening helps only until every vehicle occupies the lane for a long time; the scheduling overhead itself consumes the road.

What Virtual Threads Are

Virtual threads are not faster threads; they are cheap enough to create freely. Originating from Project Loom (started ~2018, incubated in Java 19/20, standardized in Java 21 via JEP 444), they are now production‑ready without preview‑API drift risk. Treating them as a silver bullet to replace all thread pools causes trouble — they solve scaling for wait‑heavy workloads, not CPU‑bound ones.

Key mechanisms:

Stack frames live in the Java heap, not OS memory; initial size is tiny and grows on demand, so millions of virtual threads don’t blow memory.

They run on a small set of carrier threads (regular platform threads) whose count tracks CPU cores, completely decoupled from task count.

On blocking (sleep, network I/O, lock wait), the JVM unmounts the virtual thread from its carrier; the carrier immediately runs another virtual thread. When the block ends, the virtual thread remounts and continues. Business code still uses synchronous blocking style, but the bottleneck shifts from thread count to wait count.

The author prefers viewing virtual threads as cheap blocking threads rather than faster threads. For pure compute they perform similarly to platform threads because the bottleneck is CPU, not waiting — virtual threads don’t help there.

Same Blocking Logic, Two Implementations Compared

Rewriting the slow endpoint with virtual threads changes only the executor; downstream calls remain untouched. Each task gets its own virtual thread, created and discarded per task:

try (ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor()) {
    List<Future<String>> calls = IntStream.range(0, 3)
        .mapToObj(i -> executor.submit(() -> {
            Thread.sleep(1000); // same 1‑second external call
            return "resp-" + i;
        }))
        .toList();

    for (Future<String> f : calls) {
        f.get();
    }
}

Test results: the 200‑thread pool gave 3 seconds per request and 50‑second tail latency at 100 concurrency. The virtual‑thread version handled 100 concurrent requests in ~1 second overall; thread count grew from 200 to tens of thousands with no memory pressure. The numbers are straightforward — waiting no longer consumes a thread — but in production the difference is stark: the old style queues requests into multi‑second tails during traffic spikes, while virtual threads let waiting proceed concurrently, preventing the pile‑up.

Caution: virtual threads are unbounded by default; 10,000 tasks create 10,000 virtual threads. For most I/O this is fine, but if downstream has a capacity limit you must add a semaphore (shown in the migration section). The change is not about writing less code; it’s about repricing the cost of waiting from occupying an expensive kernel thread to parking a lightweight heap object — a completely different order of magnitude.

Three Creation Patterns and When to Use Each

1. Executors.newVirtualThreadPerTaskExecutor() — simplest; one virtual thread per task, no pool sizing needed.

2. Thread.startVirtualThread(Runnable) — fire a single virtual thread directly:

Thread vt = Thread.startVirtualThread(() -> {
    System.out.println("Running on " + Thread.currentThread());
});
vt.join();

3. Thread.Builder — for naming, daemon status, or custom carrier pool. Useful for production diagnostics because named virtual threads are easier to locate in logs:

ThreadFactory factory = Thread.ofVirtual()
    .name("order-vt-", 0)
    .factory();

try (ExecutorService executor = Executors.newThreadPerTaskExecutor(factory)) {
    executor.submit(() -> handleOrder(123L));
}

Structured concurrency ( StructuredTaskScope) is still preview in Java 21 (standardized in Java 24). For ordinary business code, newVirtualThreadPerTaskExecutor suffices; adopt structured concurrency after upgrading to Java 24 to avoid preview‑feature maintenance burden.

What Structured Concurrency Solves

When concurrently calling multiple downstreams and needing all to succeed or taking the fastest, manual Future handling risks orphaned tasks that keep running after the result is ignored. StructuredTaskScope confines a group of subtasks to a scope: they either all succeed or all fail, and cancellation propagates automatically.

ShutdownOnFailure — all must succeed; any failure cancels the rest:

try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
    Future<User> user = scope.fork(() -> fetchUser(userId));
    Future<Order> order = scope.fork(() -> fetchOrder(orderId));
    scope.join();
    scope.throwIfFailed(); // any failure fails the whole group
    render(user.get(), order.get());
}

ShutdownOnSuccess — first success wins; others cancelled immediately. Ideal for multi‑source fallback (e.g., cache → DB → RPC):

try (var scope = new StructuredTaskScope.ShutdownOnSuccess<Price>()) {
    scope.fork(() -> priceFromCache());
    scope.fork(() -> priceFromDb());
    scope.fork(() -> priceFromRpc());
    return scope.join().get(); // first successful result
}

These patterns shine in aggregation and fan‑out queries. Reminder: in Java 21 this is preview; evaluate JDK constraints before adopting in production.

Why synchronized Causes Virtual Threads to Degrade

Pinning is subtle. A virtual thread can unmount only when not in code that prevents unmounting. Two constructs pin the carrier thread: synchronized blocks and native methods. While pinned, blocking does not unmount; the carrier platform thread stays blocked, and scalability collapses back to a traditional thread pool. The symptom is confusing — latency looks identical to the thread‑pool era, making you think virtual threads aren’t working, when in fact pinning silently dragged you back.

Dangerous: blocking inside synchronized

// Danger: blocking in synchronized block pins carrier thread
synchronized (lock) {
    Thread.sleep(1000); // carrier thread stuck for this second
}

Switching to ReentrantLock allows normal unmounting; the carrier runs other virtual threads during the wait. The trade‑off is more boilerplate, but you gain true elastic scaling even under saturation.

lock.lock();
try {
    Thread.sleep(1000); // carrier thread runs other virtual threads now
} finally {
    lock.unlock();
}

Services full of synchronized should audit hot paths and replace with ReentrantLock before migrating. Skipping this step can make performance worse than not migrating — the author’s team learned this the hard way.

How to Detect Pinning

During development, enable -Djdk.tracePinnedThreads=full. Pinned stacks print directly, revealing exactly which synchronized or native call pinned the carrier.

# Runtime flag
java -Djdk.tracePinnedThreads=full -jar app.jar

In production, use jcmd to capture a full thread dump and analyze pinned distribution — safer than restarting with flags. This shows which call paths frequently pin carriers.

jcmd <pid> Thread.dump -format=json > dump.json

The team discovered a third‑party SDK using synchronized around a blocking call; bypassing it finally unlocked throughput. Diagnostics cannot be skipped; otherwise you assume migration worked while still stuck in the pool. Guessing from metrics is far slower than reading the stacks — pinning points and offending libraries become immediately clear.

ThreadLocal Pitfalls in Virtual Threads

When thread count jumps from hundreds to tens of thousands, ThreadLocal memory usage amplifies by orders of magnitude — each virtual thread gets its own copy. Storing large objects in ThreadLocal and forgetting remove() creates sudden memory pressure.

More insidious: InheritableThreadLocal behaves inconsistently in virtual threads; parent values may not propagate as expected. Java 21 introduces preview ScopedValue as a replacement — scoped, automatically reclaimed, suited for passing a value through a call chain. Legacy code often stuffs user context into InheritableThreadLocal for async access; under virtual threads the value may not transmit (yielding null) or leak across requests — a correctness hazard harder to debug than performance regression.

Practical advice: avoid ThreadLocal when possible; if required, always remove() in finally. For cross‑virtual‑thread values, prefer ScopedValue; do not rely on InheritableThreadLocal.

Migrating from Thread Pools: What Must Change

Migration is not a one‑line executor swap. The most critical addition is rate limiting because virtual threads are unbounded; you must cap concurrency to avoid overwhelming downstream. The real friction lies in legacy monitoring, rate limiters, and thread‑pool tuning — all built on the assumption of stable platform thread counts. Virtual threads break that assumption.

Semaphore concurrency = new Semaphore(200); // downstream capacity limit

try (ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor()) {
    var tasks = IntStream.range(0, 10_000).mapToObj(i -> executor.submit(() -> {
        concurrency.acquire();
        try {
            Thread.sleep(1000); // one downstream blocking call
            return i;
        } finally {
            concurrency.release();
        }
    }));
    tasks.forEach(f -> f.join());
}

Replace the fixed thread pool with newVirtualThreadPerTaskExecutor and guard downstream calls with a semaphore. APM dashboards must also change: many metrics key off platform thread count, which will spike with virtual threads. Dashboards should track task throughput, not thread count. The team nearly wasted a day chasing a false “thread leak” alert because dashboards weren’t updated.

Scenarios Where Virtual Threads Don’t Fit

Defining boundaries matters more than hype. Pure CPU‑bound work gains nothing — stick with parallel streams or platform thread pools. Downstream native libraries that block internally still pin carriers, so virtual threads provide no benefit. Strict concurrency limits require manual semaphores anyway. Projects stuck on Java 17 or older must upgrade the JDK first; weigh migration benefits against upgrade cost.

Comparison Summary

Key differences at a glance:

Stack location: Platform thread — OS memory; Virtual thread — Java heap

Default cost: Platform thread — ~1 MB each; Virtual thread — tiny initial, grows on demand

Scheduler: Platform thread — OS kernel; Virtual thread — JVM ForkJoinPool

On blocking: Platform thread — occupies kernel thread; Virtual thread — unmounts, carrier reused

Count limit: Platform thread — thousands become tight; Virtual thread — millions feasible

Best for: Platform thread — CPU‑intensive; Virtual thread — I/O‑intensive, high waiting

synchronized blocking: Platform thread — normal thread occupancy; Virtual thread — pins carrier thread

Three takeaways for adoption: only migrate endpoints with heavy blocking; replace synchronized with ReentrantLock before migrating; don’t treat virtual threads as a pool substitute and don’t let unbounded concurrency hammer downstream. Do these three, and the formerly 50‑second endpoint stabilizes around 1 second with higher throughput.