How to Keep Your Distributed System Running Even When Upstream Services Fail

When building distributed systems, a key requirement for developers is that the system must remain operational even if upstream or downstream services fail.

This principle reflects the fundamental approach of protecting system health by controlling fault propagation at the service level to prevent systemic crashes.

To maintain robustness, two essential practices are recommended:

Implement rate limiting so that the interfaces you expose cannot be overwhelmed by excessive calls, which could otherwise bring your service down.

Implement circuit breaking so that when calling external interfaces, slow or failing downstream services do not drag your service down.

In theory, if every interface call adheres to these practices, only large-scale disasters (e.g., an entire data center outage) would cause systemic failures; isolated service issues are mitigated by mutual protection.

In practice, many incidents arise from neglecting these safeguards. The author invites readers to reflect on whether they fully protect their services when providing or consuming APIs.