Building Resilient Microservices: Patterns and Practices for High Availability

Microservice architecture isolates failures by defining clear service boundaries, but network, hardware, and application errors are common, and any component may become temporarily unavailable; therefore, fault‑tolerant services are needed to handle interruptions gracefully.

This article, based on RisingStack's Node.js consulting experience, introduces the most common techniques and architectural patterns for building and operating highly available microservice systems.

If you are unfamiliar with these patterns, it does not mean you are doing something wrong—building reliable systems always incurs additional cost.

Risks of Microservice Architecture

Moving application logic into separate services and communicating over the network adds latency and increases system complexity, leading to a higher rate of network failures.

One major advantage of microservices is that teams can independently design, develop, and deploy their services, owning the full lifecycle; however, they cannot control the services they depend on, which may be managed by other teams and become temporarily unavailable due to bugs, configuration changes, or other issues.

Graceful Service Degradation

Microservices allow fault isolation and graceful degradation; for example, during an outage a photo‑sharing app might prevent new uploads while still allowing users to browse, edit, and share existing photos.

Microservice fault isolation

Because services depend on each other, achieving graceful degradation often requires implementing several failover logics, which are described later in this article.

If services lack failover logic, a single failure can cause the entire chain to collapse.

Change Management

Google's Site Reliability Engineering team found that roughly 70% of incidents are caused by changes to existing systems; deploying new code or configuration can introduce bugs or failures.

To mitigate this, adopt change‑management strategies such as canary deployments—rolling out a new version to a small subset of instances, monitoring key metrics, and rolling back immediately if negative impact is observed.

Another approach is blue‑green (or red‑black) deployment: run two production environments, switch the load balancer to the new version only after verification.

Rolling back code is not a failure; the earlier you roll back, the better.

Health Checks and Load Balancing

Instances may become temporarily or permanently unavailable due to failures, deployments, or autoscaling. Load balancers should skip unhealthy instances that cannot serve traffic.

Health can be verified by repeatedly calling the GET /health endpoint or via self‑reporting. Modern service‑discovery solutions continuously collect health information and configure load balancers to route traffic only to healthy components.

Self‑Healing

Self‑healing helps applications recover from errors by having an external system monitor instance health and restart instances that remain unhealthy for a prolonged period. While useful, frequent restarts due to overload or database‑connection timeouts can cause additional problems.

For special cases such as lost database connections, add extra logic so the external system knows the instance should not be immediately restarted.

Failover Cache

A failover cache provides data when a service is down, using two expiration times: a short TTL for normal operation and a longer TTL that keeps data usable during service outages.

Standard HTTP response headers can configure this behavior. For example, the max-age attribute defines the normal cache lifetime, while stale-if-error allows stale data to be served when an error occurs.

Modern CDNs and load balancers support various caching and failover features, and companies can also build shared libraries for reliable caching.

Retry Logic

When operations are expected to succeed after a short delay, retry them with exponential back‑off and a maximum limit. Ensure retries are idempotent—use a unique idempotency key for each transaction to avoid duplicate charges or actions.

Rate Limiter and Load Shedding

Rate limiting defines how many requests a client or application may send within a time window, protecting resources from overload and allowing priority traffic to receive sufficient resources.

Concurrent request limiters can protect critical endpoints from exceeding a defined call count while still providing service.

Load‑shedding mechanisms reserve resources for high‑priority requests and prevent low‑priority traffic from exhausting capacity, helping the system stay responsive during spikes.

Fast‑Fail Principle and Independence

Microservices should fail fast and remain independent. Instead of static timeouts (an anti‑pattern), use circuit breakers that open when many errors occur in a short period, stop further requests, and close after a cooldown period.

Bulkhead Pattern

Inspired by ship compartments, bulkheads isolate resources (e.g., separate connection pools for different database operations) so that a failure in one pool does not affect others.

Circuit Breaker

A circuit breaker opens when a threshold of failures is reached, preventing further calls to the failing service; after a timeout, a trial request checks if the service has recovered before closing the circuit.

Testing Failures

Continuously test common failure scenarios to ensure teams can handle incidents. Chaos engineering tools like Netflix's Chaos Monkey can terminate random instances or entire zones to simulate cloud‑provider failures.

Conclusion

Implementing and operating reliable services requires significant effort and budget; reliability spans many layers, and teams must allocate sufficient resources and make reliability a core business decision.

Main Takeaways

Dynamic environments and distributed systems increase failure rates.

Services should isolate faults and degrade gracefully to improve user experience.

About 70% of incidents stem from changes; rolling back code is acceptable.

Fast‑fail and independence are essential because teams cannot control dependent services.

Patterns such as caching, bulkheads, circuit breakers, and rate limiters help build reliable microservice architectures.