How to Prevent System Failures: Suspect Third‑Party Services, Guard Consumers, and Strengthen Your Own Service

For every programmer, failures feel like a sword hanging over the head, and avoiding them is a constant pursuit. The author summarizes the approach in one sentence: suspect third‑party services, guard the consumers, and do your own part well.

1. Suspect Third‑Party Services

Assume all external services are unreliable and take actions such as providing fallback mechanisms, defining degradation plans, and setting strict timeouts. Examples include caching hot items when a user‑center service fails, using both push (message) and pull (HTTP) synchronization to guarantee data freshness, and keeping snapshots of index files to roll back when data becomes polluted.

Setting short timeout values (e.g., 200 ms) prevents a slow third‑party from exhausting thread pools and causing load spikes.

Retry mechanisms must be used judiciously; blind retries can amplify pressure on downstream services.

2. Guard Consumers

Design APIs (RPC/REST) that are hard to misuse. Follow principles such as exposing the minimal necessary endpoints, avoiding overly granular calls, limiting batch sizes, providing clear parameter contracts, and returning meaningful exceptions instead of swallowing errors.

Implement traffic‑control measures like per‑service rate limiting or circuit breakers to protect against sudden traffic spikes, whether caused by external attacks or internal misuse.

3. Do Your Own Part

Apply fundamental engineering principles across the whole lifecycle—requirements, architecture, coding, testing, code review, deployment, and operations. Highlights include:

Single‑responsibility principle to keep services and classes focused.

Resource‑control tactics for CPU (algorithm optimization, lock reduction, thread‑pool sizing, JVM tuning), memory (JVM limits, collection pre‑allocation, object pools, caching strategies), network (batch calls, payload reduction), and disk (log volume control, remote log storage).

Avoid single points of failure by horizontal scaling, multi‑zone deployment, and sharding.

Code example illustrating a simple method with proper try‑catch handling:

public List<Integer> test(){
    try {
        // ...
    } catch (Exception e) {
        return Collections.emptyList();
    }
}

Conclusion

The distilled advice is: “suspect third parties, guard consumers, and do your own part well.” Readers are encouraged to reflect on their own experiences and share best practices.