Taming Chaotic Logs: Practical Governance and Stack Trace Optimization

1. Log Chaos

Logging is the fastest source of entropy in a system, accumulating all side‑effects of business growth. The article begins by describing typical chaos: ignored logs, over‑use of simple info calls without knowing output locations, and a proliferation of ad‑hoc LogUtil classes—sometimes dozens in a single project—leading to a classic broken‑window effect.

Common problems include:

Unclear classification, with many unrelated log files created for isolated monitoring needs.

Inconsistent formats, where a single file may contain both business logs and stack traces.

Multiple logging tools producing different structures, making parsing and analysis difficult.

These issues cause developers to constantly add new log files for new requirements, turning the logging subsystem into a tangled elephant.

2. Log Layering

To restore order, the author proposes a three‑layer architecture: Entry (traffic‑in/out parameters such as HSF or controller), Core (business processing logs), and Exit (third‑party service calls). Stack traces are treated separately because they have special characteristics.

3. Format Design

The format should be human‑readable and monitorable. Key guidelines:

Abstract dimensions (industry, service, merchant, etc.) and store them in ThreadLocal so the logger only needs a few parameters.

Keep the line simple: fixed‑position fields separated by commas, avoiding large JSON blobs.

Prefer self‑describing codes over raw bit‑mask values (e.g., print a descriptive refund status instead of 1,2,4).

Example format line:

timeStamp|threadName logLevel loggerName|sourceAppName,flowId,traceId,sceneCode,identityCode,loginUserId,scpCode,rpcId,isYace,ip||businessCode,isSuccess||parameters||returnResult||

Concrete log entry example:

2023-08-14 14:37:12.919|http-nio-7001-exec-10 INFO c.a.u.m.s.a.LogAspect|default,c04e4b7ccc2a421995308b3b33503dda,0bb6d59616183822328322237e84cc,queryOrderStatus,XIAODIAN,5000000000014,123456,0.1.1.8,null,255.255.255.255||queryOrderStatus,success||{...json...}||{...json...}||

4. Stack Reversal

Standard stack traces are folded in log viewers, hiding the root cause at the bottom. The author introduces “stack reversal” so the deepest exception appears first, making the real problem instantly visible.

Two pain points addressed:

Folded stacks require manual expansion to find the root cause.

Full stack traces consume excessive storage.

Implementation ideas:

Recursively print the cause chain from the deepest exception upward.

Compress each stack line by shortening fully‑qualified class names to c.a.u.m.s.LogAspect#log:88 (first letter of each package segment, then class and method).

Key recursive method:

public static void recursiveReversePrintStackCause(Throwable t, int causeDepth, ForwardCounter counter, int stackDepth, StringBuilder sb) {
    if (t == null) {
        return;
    }
    if (t.getCause() != null) {
        recursiveReversePrintStackCause(t.getCause(), causeDepth, counter, stackDepth, sb);
    }
    if (counter.i++ < causeDepth) {
        doPrintStack(t, stackDepth, sb);
    }
}

Helper to print a compressed stack line:

public static void doPrintStack(Throwable t, int stackDepth, StringBuilder sb) {
    StackTraceElement[] stackTraceElements = t.getStackTrace();
    if (sb.lastIndexOf("\t") > -1) {
        sb.deleteCharAt(sb.length() - 1);
        sb.append("Caused: ");
    }
    sb.append(t.getClass().getName()).append(": ").append(t.getMessage()).append("
\t");
    for (int i = 0; i < stackDepth; ++i) {
        if (i >= stackTraceElements.length) {
            break;
        }
        StackTraceElement element = stackTraceElements[i];
        sb.append(reduceClassName(element.getClassName()))
          .append("#").append(element.getMethodName())
          .append(":").append(element.getLineNumber())
          .append("
\t");
    }
}

Tests show an 88% compression ratio for typical stack traces while preserving essential information.

5. Thought Expansion

The author cautions against treating any practice as a universal silver bullet. While the three‑layer log design works well for a typical mid‑platform application, client‑heavy systems may need different dimensions because most logs are internal execution traces.

Furthermore, the cost reduction discussed only targets stack‑trace storage; overall log storage savings will be less dramatic when business logs dominate.

In summary, the article provides a pragmatic, cost‑effective logging strategy that balances simplicity, observability, and storage efficiency, while emphasizing the need to adapt solutions to specific contexts.