Why 1.5 ms RPC Calls Sometimes Exceed 100 ms? Deep Dive & Elastic Timeout Fix

Background

Customer support engineer P reported that the lookupWarehouseIdRandom RPC interface, which normally completes in 1.5 ms, was failing to meet the company’s five‑nine availability standard, with more than 500 daily timeouts despite a 100 ms client‑side timeout.

Validation & Analysis

We first illustrated the SCF RPC call flow (serialization, network transmission, deserialization, execution, response serialization, and return) with a diagram. Monitoring confirmed the average execution time of about 1.5 ms.

However, the caller’s timeout setting of 100 ms still produced many timeout events, as shown by latency distribution charts where a noticeable tail extended beyond 100 ms.

Problem Analysis

By separating the call chain into framework (network, SCF processing) and business (service logic) components, we determined that the long tail originates from both layers. Business‑side latency distribution is low, indicating most outliers stem from the framework.

Further breakdown of the framework showed that I/O operations and occasional GC pauses cause spikes, while simple CPU work also exhibits sudden jumps from 1 ms to 20 ms.

Investigation

Detailed instrumentation of the RPC path revealed that I/O jitter, GC activity, and CPU time‑slice allocation are the primary contributors to the >100 ms tail.

Root Causes

I/O operations are prone to jitter, frequently producing >100 ms delays.

CPU‑bound tasks, though generally fast, sometimes experience 20 ms pauses, likely due to GC or scheduler effects.

Solution: Elastic Timeout

We propose an “elastic timeout” mechanism that, without changing the 100 ms hard limit, allows a configurable small number of requests to exceed the timeout up to a higher threshold (e.g., 200 ms) for a short window. This tolerates occasional spikes while preserving overall service quality.

Implementation steps:

Configure per‑service and per‑function elastic timeout parameters (e.g., allow 15 requests every 40 seconds to extend to 1300 ms).

Deploy the configuration via the service management platform.

Effectiveness was demonstrated by a dramatic reduction of sporadic timeouts after enabling elastic timeout.

Applicable Scenarios

Elastic timeout is suitable for intermittent latency spikes caused by network jitter, GC pauses, CPU jitter, or cold starts. It is not a substitute for systematic analysis when large‑scale timeouts occur.

Conclusion

The analysis shows that even ultra‑fast RPC calls can suffer long‑tail latency due to framework‑level factors such as GC and I/O jitter. Applying an elastic timeout strategy mitigates these occasional outliers and helps achieve the stringent five‑nine reliability goal.