Cross-Thread Log Sampling for High-Throughput Java Services

Background

System logs are essential for tracing user actions and diagnosing issues, but excessive logging in high‑throughput JD services can degrade performance and disk I/O, especially during traffic spikes.

Problem Statement

When log volume grows, teams often wrap Log4j/Logback to add dynamic degradation, which reduces log detail and creates new problems. The goal is to sample request logs while preserving traceability across request, child, and thread‑pool threads.

Solution Exploration

Approach 1 – Transmittable Thread‑Local‑like Wrapper

Wrap thread‑pool or Runnable/Callable classes to propagate a sampling flag. JD uses components such as pfinder and jade . This works but introduces nested wrappers, increasing complexity and risk.

Approach 2 – Explicit Propagation

Manually pass the sampling flag through code wherever asynchronous tasks are submitted. It achieves the goal but tightly couples business logic with logging and requires widespread code changes.

Approach 3 – Decoupled Component with Extensible API

Encapsulate request‑thread sampling logic into a reusable component that exposes an API. Business systems can adopt the component and, if needed, extend the API to handle async threads. This balances minimal code changes with consistent sampling.

If a system uses few async threads, simply adopt the component for request‑thread sampling.

If a system heavily uses async threads, adopt the component and use the extended API to coordinate sampling across threads.

Other possibilities such as AOP are mentioned but involve larger refactoring for existing systems.

Practical Implementation

In a JD promotion‑transaction service, the existing JD JSF filter‑based log component was extended with the sampling component and an API. The diagram below shows the high‑level flow.

For async threads, the API wrapper is applied as follows:

// Before refactor
threadPoolExecutor.execute(() -> "your business logic");

// After refactor using the sampling wrapper
threadPoolExecutor.execute(XxxUtils.wrap(() -> "your business logic"));

Open Questions for Further Design

Random vs. traceId‑modulo sampling; possible scenario‑based sampling linked to request parameters.

What functions should the extended API expose to minimise business code changes?

How to guarantee global traceId consistency across async threads (pfinder offers a reference).

What is the minimal granularity of sampling (0.01%, 0.1%, 1%?).

Should a single global sampling probability be used or per‑level settings?

Policy for logging levels when sampling (e.g., info + error vs. error only).

How to throttle massive error logs, possibly linking disk I/O control with logging rate.