Building a Go Circuit Breaker: Concepts, States, and Code Walkthrough

What is a circuit breaker?

A circuit breaker is a mechanism that proactively closes or limits upstream requests when downstream errors occur.

Principle

The breaker has three periods: CLOSED, OPEN, HALF‑OPEN.

When RPC works normally, it stays CLOSED.

When RPC errors increase, the breaker trips to OPEN.

After a cooling period, it moves to HALF‑OPEN.

In HALF‑OPEN, strategic requests are made to decide whether to return to CLOSED or stay OPEN.

The overall state transition is illustrated below:

Go Implementation

Reference implementation: github.com/rubyist/circuitbreaker

IsAllowed – determines if a request is allowed based on the current state

CLOSED

Requests are allowed.

OPEN

During the cooling timeout, requests are blocked.

After the cooling period, the state becomes HALF‑OPEN and requests are allowed.

HALF‑OPEN

During the detection timeout, requests are allowed.

Otherwise, requests are blocked.

atomic.StoreInt32((*int32)(&b.state), int32(HALFOPEN))

trip – decides whether the breaker should open (customizable)

type TripFunc func(Metricser) bool

ThresholdTripFunc – error threshold.

ConsecutiveTripFunc – consecutive errors exceed threshold.

RateTripFunc – decides based on minimum request count and error rate.

Metricser – statistics interface

type Metricser interface {<br/>    Fail()    // records a failure<br/>    Succeed() // records a success<br/>    Timeout() // records a timeout<br/><br/>    Failures() int64   // number of failures<br/>    Successes() int64  // number of successes<br/>    Timeouts() int64   // number of timeouts<br/>    ConseErrors() int64 // recent consecutive errors<br/>    ErrorRate() float64 // (timeouts+failures)/(timeouts+failures+successes)<br/>    Samples() int64    // total samples (timeouts+failures+successes)<br/>    Counts() (int64, int64, int64) // (successes, failures, timeouts)<br/><br/>    Reset()<br/>}

window – implementation class

type window struct {<br/>    sync.RWMutex<br/>    oldest  int32 // oldest bucket index<br/>    latest  int32 // latest bucket index<br/>    buckets []bucket<br/><br/>    bucketTime time.Duration // duration each bucket represents<br/>    bucketNums int32          // number of buckets<br/>    inWindow  int32          // current number of buckets in the window<br/><br/>    allSuccess int64<br/>    allFailure int64<br/>    allTimeout int64<br/><br/>    conseErr int64<br/>}<br/><br/>type bucket struct {<br/>    failure int64<br/>    success int64<br/>    timeout int64<br/>}

The implementation uses a ring buffer to perform dynamic statistics. A continuous time interval is divided into multiple small buckets; each bucket stores statistics for BucketTime. When BucketTime expires, the corresponding bucket is recycled.

if w.inWindow == w.bucketNums {<br/>    // the latest covered the oldest (latest == oldest)<br/>    oldBucket := &w.buckets[w.oldest]<br/>    atomic.AddInt64(&w.allSuccess, -oldBucket.Successes())<br/>    atomic.AddInt64(&w.allFailure, -oldBucket.Failures())<br/>    atomic.AddInt64(&w.allTimeout, -oldBucket.Timeouts())<br/>    w.oldest++<br/>    if w.oldest >= w.bucketNums {<br/>        w.oldest = 0<br/>    }<br/>} else {<br/>    w.inWindow++<br/>}<br/><br/>w.latest++<br/>if w.latest >= w.bucketNums {<br/>    w.latest = 0<br/>}<br/>(&w.buckets[w.latest]).Reset()

Panel Metricser container

PanelStateChangeHandler – circuit breaker events

type PanelStateChangeHandler func(key string, oldState, newState State, m Metricser)

Limitations

All breakers share the same BucketTime; the statistics period cannot be updated dynamically.

The cooling timeout cannot be changed at runtime.