Stability Patterns for Maintaining Network Reliability in Distributed Systems

Stability Patterns in Distributed Systems

When a distributed system’s reliability depends on a weak component, even a tiny internal failure can destabilise the whole system. Understanding and applying stability patterns helps predict and mitigate network hotspots in RESTful services.

Five Core Patterns

Retry Mode

Timeout Mode

Circuit Breaker Mode

Handshaking Mode

Bulkhead Mode

Demo: Online Payment Service

A demo shop decides the payment method (credit card, PayPal, pre‑payment, invoice) based on a credit‑score service. The service shows how failures in credit‑score lookup are handled by the stability patterns.

Key JAX‑RS endpoint:

@Singleton
@Path("/")
public class PaymentService {
    private final PaymentDao paymentDao;
    private final URI creditScoreURI;
    // ...
    @GET
    @Path("paymentmethods")
    @Produces(MediaType.APPLICATION_JSON)
    public ImmutableSet<PaymentMethod> getPaymentMethods(@QueryParam("addr") String address) {
        // implementation uses the patterns
    }
}

Retry Mode

Clients such as Apache HttpClient automatically retry transient network errors, but retries waste resources on permanent failures.

Timeout Mode

Setting connection and socket timeouts prevents a slow credit‑score service from exhausting thread‑pool threads.

restClient.property(ClientProperties.CONNECT_TIMEOUT, 2000);
restClient.property(ClientProperties.READ_TIMEOUT, 2000);

Circuit Breaker

Implements three states (closed, open, half‑open) to stop calls to an unhealthy service.

public class CircuitBreaker {
    private final AtomicReference<CircuitBreakerState> state =
        new AtomicReference<>(new ClosedState());
    public boolean isRequestAllowed() { return state.get().isRequestAllowed(); }
    // state classes omitted for brevity
}

Client filter registers the breaker:

client.register(new ClientCircutBreakerFilter());

Server‑Side Circuit Breaker

Similar logic can be applied on the server using a @Provider filter that checks a health policy before invoking the resource.

Bulkhead (Isolation Wall)

Limits resources per client, per application, or per operation to prevent cascading failures. Example sets separate connection pools for credit‑score and exchange‑rate services:

CloseableHttpClient httpClient = HttpClientBuilder.create()
    .setMaxConnTotal(30)
    .setMaxConnPerRoute(30)
    .setDefaultRequestConfig(reqConfig)
    .build();
Client addrScoreClient = new ResteasyClientBuilder()
    .httpEngine(new ApacheHttpClient4Engine(httpClient, true))
    .build();

Java 8 Asynchronous Calls

Using CompletableFuture and JAX‑RS async response removes thread blocking:

@GET
@Produces(MediaType.APPLICATION_JSON)
public void getPaymentMethodsAsync(@QueryParam("addr") String address,
                                 @Suspended AsyncResponse resp) {
    paymentDao.getPaymentsAsync(address, 50)
        .thenCompose(pmts -> pmts.isEmpty()
            ? restClient.target(addrScoreURI).queryParam("addr", address)
                .request().async().get(Score.class)
            : CompletableFuture.completedFuture(pmts.stream()
                .anyMatch(Payment::isDelayed) ? Score.NEGATIVE : Score.POSITIVE))
        .exceptionally(err -> Score.NEUTRAL)
        .thenApply(SCORE_TO_PAYMENTMETHOD)
        .whenComplete(ResultConsumer.write(resp));
}

Conclusion

The presented stability patterns enable a distributed system to continue operating despite component failures, and the same concepts apply to any communication endpoint such as databases or message queues.