Mastering Go Circuit Breakers: Boost System Resilience with gobreaker

In Go projects, circuit‑breaker techniques improve system fault tolerance. This article introduces Go circuit breakers and demonstrates their usage.

A circuit breaker acts like a fuse: when a dependent service fails, it quickly isolates the fault, preventing cascade failures and reducing request rates to give the upstream service time to recover.

Circuit breakers are widely used not only in applications but also in web gateways and micro‑services. This article studies Sony’s open‑source implementation github.com/sony/gobreaker (source comments also available at github.com/lpflpf/gobreaker).

Circuit‑Breaker Pattern

gobreaker is a Go implementation of the circuit‑breaker pattern described in *Microsoft Cloud Design Patterns*. It is open‑source by Sony and has over 1.2K stars.

The pattern is defined by a state machine:

The breaker has three states and four possible transitions:

Three States:

Closed – service operates normally.

Open – service is unhealthy.

Half‑Open – limited requests are allowed.

Four Transitions:

From Closed to Open when failures meet the configured threshold.

From Open to Half‑Open after a timeout.

From Half‑Open back to Open if a request fails.

From Half‑Open to Closed when enough successful requests occur.

Implementation of gobreaker

gobreaker builds on the state machine to provide a functional circuit breaker.

Definition

type CircuitBreaker struct {
  name          string
  maxRequests   uint32 // max requests in half‑open state
  interval      time.Duration // statistics interval
  timeout       time.Duration // timeout after opening
  readyToTrip   func(counts Counts) bool // custom trip condition
  onStateChange func(name string, from State, to State) // hook on state change

  mutex      sync.Mutex // protects following fields
  state      State
  generation uint64
  counts     Counts // success/failure statistics
  expiry     time.Time // next period time
}

Customizable parameters include:

MaxRequests – maximum allowed requests in half‑open state.

Interval – length of a normal statistics period (0 clears counts each call).

Timeout – duration before a new request is allowed after opening.

readyToTrip – function that decides when to open the breaker.

onStateChange – hook invoked on state transitions.

Executing Requests

The execution flow has three phases: pre‑check, request execution, and post‑request state update.

// Execute the circuit breaker
func (cb *CircuitBreaker) Execute(req func() (interface{}, error)) (interface{}, error) {
    generation, err := cb.beforeRequest()
    if err != nil {
        return nil, err
    }
    defer func() {
        if e := recover(); e != nil {
            // handle panic
            cb.afterRequest(generation, false)
            panic(e)
        }
    }()
    result, err := req()
    cb.afterRequest(generation, err == nil)
    return result, err
}

Pre‑request Check

Before a request, the breaker checks its current state. If it is Open, the request is rejected; if Half‑Open and the request count exceeds MaxRequests, it is also rejected.

func (cb *CircuitBreaker) beforeRequest() (uint64, error) {
    cb.mutex.Lock()
    defer cb.mutex.Unlock()
    now := time.Now()
    state, generation := cb.currentState(now)
    if state == StateOpen {
        return generation, ErrOpenState
    } else if state == StateHalfOpen && cb.counts.Requests >= cb.maxRequests {
        return generation, ErrTooManyRequests
    }
    cb.counts.onRequest()
    return generation, nil
}

The current state is computed based on timers: an Open breaker moves to Half‑Open after its timeout, and a Closed breaker may start a new generation after the interval expires.

func (cb *CircuitBreaker) currentState(now time.Time) (State, uint64) {
    switch cb.state {
    case StateClosed:
        if !cb.expiry.IsZero() && cb.expiry.Before(now) {
            cb.toNewGeneration(now)
        }
    case StateOpen:
        if cb.expiry.Before(now) {
            cb.setState(StateHalfOpen, now)
        }
    }
    return cb.state, cb.generation
}

Post‑request Handling

After each request, the breaker updates its counters and may transition states based on success or failure.

func (cb *CircuitBreaker) afterRequest(before uint64, success bool) {
    cb.mutex.Lock()
    defer cb.mutex.Unlock()
    now := time.Now()
    state, generation := cb.currentState(now)
    if generation != before {
        return
    }
    if success {
        cb.onSuccess(state, now)
    } else {
        cb.onFailure(state, now)
    }
}

In Half‑Open state, a series of successful requests equal to MaxRequests closes the breaker; a failure re‑opens it. In Closed state, successes simply update counters, while failures trigger the readyToTrip function to decide if the breaker should open.

Summary

For services with unreliable remote dependencies, circuit breakers prevent cascading failures and protect your own service.

The implementation maintains extensive statistics and uses mutexes, incurring some overhead.

In Half‑Open state, excessive requests can cause “too many requests” errors if the success threshold is not met.

Source: segmentfault.com/a/1190000023033343