How to Implement Circuit Breaking for Thrift Services with Aeraki Mesh and Istio

Prerequisites and Installation

Install Aeraki Mesh, Istio, and the sample applications in a Kubernetes cluster. After installation, two namespaces meta-dubbo and meta-thrift appear, each containing a sample program that implements the Dubbo or Thrift protocol via Aeraki's MetaProtocol support.

kubectl get ns | grep meta
meta-dubbo   Active   16m
meta-thrift  Active   16m

Simulating a Thrift Service Failure

Create a deployment that runs an nginx container but is registered as a Thrift service endpoint. This deployment, named thrift-sample-server-fake, adds an endpoint that cannot handle Thrift requests.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: thrift-sample-server-fake
  namespace: meta-thrift
  labels:
    app: thrift-sample-server
spec:
  selector:
    matchLabels:
      app: thrift-sample-server
  replicas: 1
  template:
    metadata:
      annotations:
        sidecar.istio.io/bootstrapOverride: aeraki-bootstrap-config
        sidecar.istio.io/proxyImage: aeraki/meta-protocol-proxy:1.0.1
        sidecar.istio.io/rewriteAppHTTPProbers: "false"
    spec:
      containers:
        - name: thrift-sample-server
          image: nginx
          ports:
            - containerPort: 9090

The thrift-sample-server service now has three endpoints, one of which points to the fake deployment (IP 172.19.0.102). Client logs show a Thrift application exception for every third request because the fake endpoint cannot process the request.

org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'

Creating the Circuit‑Breaking Rule

Apply an Istio DestinationRule that triggers outlier detection: after five consecutive 5xx errors, the offending host is ejected for 15 minutes.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: thrift-sample-server
  namespace: meta-thrift
spec:
  host: thrift-sample-server
  trafficPolicy:
    outlierDetection:
      baseEjectionTime: 15m
      consecutive5xxErrors: 5
      interval: 5m

Once the rule is active, the client stops sending requests to the failing endpoint after the threshold is reached, confirming that the circuit‑breaker works as intended.

Verifying the Ejection

Query Aeraki sidecar statistics to see that the host has been ejected:

aerakictl_sidecar_stats client meta-thrift | grep -i outlier
cluster.outbound|9090|thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_active: 1
cluster.outbound|9090|thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_consecutive_5xx: 1
cluster.outbound|9090|thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_detected_consecutive_5xx: 1
cluster.outbound|9090|thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_enforced_consecutive_5xx: 1
cluster.outbound|9090|thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_total: 1

How It Works

Envoy (the data plane used by Istio) monitors request outcomes. When the configured consecutive error threshold is met, the outlier detection module marks the host as unhealthy and removes it from the load‑balancing pool for the duration specified by baseEjectionTime. This prevents a single faulty service from causing a cascade failure across the mesh.