Go Microservice Stability: Rate Limiting, Circuit Breaking, Degradation and K8s Production Architecture

Real traffic overload case

During a promotion the order service normally handled 3,000 QPS, but traffic spiked to 48,000 QPS. The failure was caused by a combination of issues: inventory-service slowed down because a SQL query became slow, raising P99 from 40 ms to 1.8 s. risk-service started rate‑limiting and returned many 429 responses. order-api performed three synchronous retries, amplifying downstream pressure.

Goroutine count exploded, connection pools were exhausted and memory grew quickly.

Kubernetes HPA scaled pods based on CPU, but the database and third‑party services did not scale, so the failure was replicated across more pods.

Why simple rate limiting, circuit breaking and degradation are not enough

Stability governance must form a closed loop that includes rate limiting, timeout, retry, circuit breaking, isolation, asynchronous shaping, configuration distribution, scaling, monitoring and recovery. Treating each technique in isolation leads to “pseudo‑stability”.

Six fundamental questions

How much traffic can be admitted?

When should a caller abandon a slow downstream?

Which capabilities are core and which can be sacrificed?

How to protect databases, connection pools and third‑party services before scaling?

How to detect, stop and recover from failures quickly?

How to implement the above in a Go + Kubernetes environment?

Rate limiting fundamentals

Fixed window

Counts requests per fixed interval (e.g., 1 000 per second). Simple but suffers from burst spikes:

00:00:00.999 1000 requests
00:00:01.001 another 1000 requests

Sliding window

Divides a large window into smaller buckets and sums recent buckets for higher accuracy, at the cost of higher computation.

Leaky bucket

Emits tokens at a constant rate, suitable for traffic shaping but not friendly to bursts.

Token bucket

Generates tokens at a fixed rate; a request proceeds only when a token is available. The bucket capacity determines how many bursts can be absorbed. It is the most common algorithm for online services because it satisfies:

Long‑term average traffic is controllable.

Short‑term bursts are tolerated.

Implementation is simple.

Go’s standard library provides golang.org/x/time/rate:

limiter := rate.NewLimiter(rate.Limit(800), 1200)

This means:

Steady release of 800 requests per second.

Burst capacity of up to 1 200 requests.

Requests arriving within the burst are smoothed.

Local vs distributed limiting

Local limiting protects a single instance and is fast, but it cannot see the global traffic. For a cluster of 10 pods each limited to 500 QPS, the global capacity could be 5 000 QPS, which may exceed downstream capacity (e.g., 3 000 QPS). In that case a global limiter at the ingress layer or a Redis‑based distributed limiter is required.

Conclusion: Use local token‑bucket limiting for fast first‑line protection and a distributed limiter for global quota.

Circuit breaking

A breaker has three states: Closed → Open → Half‑Open → Closed.

Closed: Normal traffic, counting total requests, failures, slow calls, and consecutive failures.

Open: When thresholds are reached, calls are short‑circuited.

Half‑Open: After a cool‑down period a limited number of probe requests are allowed; success closes the breaker, failure re‑opens it.

Threshold design must avoid two extremes:

Too sensitive – e.g., 2 failures out of 2 requests open the breaker, causing flapping.

Too sluggish – e.g., downstream latency of 3 s is tolerated, dragging the caller down.

Typical production thresholds combine:

Minimum request count (e.g., ≥20).

Error‑rate threshold (e.g., ≥50%).

Slow‑call ratio (e.g., >60%).

Cool‑down time (e.g., 30 s).

Half‑open probe count (e.g., 3).

Conclusion: Circuit breaking should consider both error rate and latency.

Degradation strategies

Degradation is a hierarchy of business‑level fallbacks, not just a static page:

Feature trimming: Hide “You may also like” on the order page.

Data degradation: Read from a cache snapshot instead of real‑time data.

Path switch: Replace synchronous risk check with asynchronous compensation.

Capacity degradation: Allow only whitelisted users.

Manual switch: Turn off non‑core capabilities before a big promotion.

The degradation logic itself must not depend on unstable components. For example, coupon degradation should return a local template instead of calling another slow service.

Timeout, retry and back‑off

Unified timeout budget

All services should receive a deadline from the entry point and propagate it downstream. In Go this is done with context.Context:

func WithBudget(parent context.Context, max time.Duration) (context.Context, context.CancelFunc) {
    if deadline, ok := parent.Deadline(); ok {
        remain := time.Until(deadline)
        if remain < max {
            return context.WithTimeout(parent, remain)
        }
    }
    return context.WithTimeout(parent, max)
}

Usage in a handler:

ctx, cancel := stability.WithBudget(req.Context(), 900*time.Millisecond)
defer cancel()

Retry conditions

Retry is safe only for idempotent operations, clear transient failures, with back‑off and a maximum attempt count, and must not bypass circuit breakers or capacity limits.

Exponential back‑off

Typical back‑off sequence: 100 ms → 200 ms → 400 ms, optionally with jitter to avoid retry storms.

Isolation (resource partitioning)

Because Go has no thread‑pool exhaustion problem, the real bottleneck is resource contention:

Blocking goroutines hold sockets, memory, timers, and contexts.

Slow downstream calls can pile up handlers.

Core and non‑core requests share the same resources.

Isolation measures:

Separate http.Transport per downstream.

Dedicated concurrency semaphores for high‑risk calls.

Separate worker pools for async tasks.

Separate quotas for core tenants, internal traffic and callbacks.

Conclusion: Limiting entry rate controls “how many”; isolation controls “who can use which resources”.

Production‑grade Go implementation

Project layout

stability-demo/
├── cmd/order-api/main.go
├── internal/config/config.go
├── internal/transport/httpserver/server.go
├── internal/middleware/ratelimit.go
├── internal/middleware/trace.go
├── internal/stability/breaker.go
├── internal/stability/degrade.go
├── internal/stability/budget.go
├── internal/client/inventory/client.go
├── internal/client/risk/client.go
├── internal/client/coupon/client.go
├── internal/service/order_service.go
├── internal/repository/idempotency_repo.go
├── internal/metrics/metrics.go
└── deployments/

Configuration model (JSON)

{
  "http": {
    "addr": ":8080",
    "readTimeout": "1s",
    "readHeaderTimeout": "500ms",
    "writeTimeout": "2s",
    "idleTimeout": "60s"
  },
  "stability": {
    "qpsLimit": 800,
    "burst": 1200,
    "maxInFlight": 300,
    "requestTimeout": "900ms",
    "degradeCoupon": false,
    "degradeRecommend": true
  },
  "downstream": {
    "inventory": {
      "baseURL": "http://inventory-service",
      "timeout": "150ms",
      "maxIdleConns": 200,
      "maxConnsPerHost": 150,
      "idleConnTimeout": "30s",
      "breakerWindow": "10s",
      "breakerTimeout": "30s",
      "breakerMinRequests": 20,
      "breakerErrorPercent": 0.5
    }
  }
}

Local token‑bucket + in‑flight limit middleware

type RateLimit struct {
    limiter    *rate.Limiter
    inFlight   atomic.Int64
    maxInFlight int64
}

func NewRateLimit(qps, burst int, maxInFlight int64) *RateLimit {
    return &RateLimit{limiter: rate.NewLimiter(rate.Limit(qps), burst), maxInFlight: maxInFlight}
}

func (r *RateLimit) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
        if !r.limiter.Allow() {
            writeJSON(w, http.StatusTooManyRequests, map[string]any{"code": "RATE_LIMITED", "message": "system is busy"})
            return
        }
        cur := r.inFlight.Add(1)
        if cur > r.maxInFlight {
            r.inFlight.Add(-1)
            writeJSON(w, http.StatusServiceUnavailable, map[string]any{"code": "TOO_MANY_INFLIGHT", "message": "server overloaded"})
            return
        }
        defer r.inFlight.Add(-1)
        next.ServeHTTP(w, req)
    })
}

Unified timeout budget helper

func WithBudget(parent context.Context, max time.Duration) (context.Context, context.CancelFunc) {
    if deadline, ok := parent.Deadline(); ok {
        remain := time.Until(deadline)
        if remain < max {
            return context.WithTimeout(parent, remain)
        }
    }
    return context.WithTimeout(parent, max)
}

Circuit‑breaker wrapper (github.com/sony/gobreaker)

var ErrCircuitOpen = errors.New("circuit breaker open")

type Breaker struct { cb *gobreaker.CircuitBreaker }

func NewBreaker(name string, minRequests uint32, window, timeout time.Duration, errorPercent float64) *Breaker {
    settings := gobreaker.Settings{
        Name:        name,
        Interval:    window,
        Timeout:     timeout,
        MaxRequests: 3,
        ReadyToTrip: func(c gobreaker.Counts) bool {
            if c.Requests < minRequests { return false }
            return float64(c.TotalFailures)/float64(c.Requests) >= errorPercent
        },
    }
    return &Breaker{cb: gobreaker.NewCircuitBreaker(settings)}
}

func (b *Breaker) Execute(fn func() error) error {
    _, err := b.cb.Execute(func() (any, error) { return nil, fn() })
    if errors.Is(err, gobreaker.ErrOpenState) || errors.Is(err, gobreaker.ErrTooManyRequests) {
        return ErrCircuitOpen
    }
    return err
}

Per‑downstream HTTP client (example: inventory)

type Client struct {
    baseURL string
    http    *http.Client
    breaker *stability.Breaker
}

func NewClient(rule config.ClientRule) *Client {
    transport := &http.Transport{
        Proxy:               http.ProxyFromEnvironment,
        MaxIdleConns:        rule.MaxIdleConns,
        MaxConnsPerHost:     rule.MaxConnsPerHost,
        IdleConnTimeout:     rule.IdleConnTimeout,
        TLSHandshakeTimeout: 3 * time.Second,
        ResponseHeaderTimeout: rule.Timeout,
        DialContext: (&net.Dialer{Timeout: 200 * time.Millisecond, KeepAlive: 30 * time.Second}).DialContext,
    }
    return &Client{
        baseURL: rule.BaseURL,
        http: &http.Client{Timeout: rule.Timeout, Transport: transport},
        breaker: stability.NewBreaker("inventory-client", rule.BreakerMinRequests, rule.BreakerWindow, rule.BreakerTimeout, rule.BreakerErrorPercent),
    }
}

func (c *Client) Reserve(ctx context.Context, reqBody ReserveRequest) error {
    return c.breaker.Execute(func() error {
        ctx, cancel := stability.WithBudget(ctx, 120*time.Millisecond)
        defer cancel()
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/reserve", nil)
        if err != nil { return err }
        resp, err := c.http.Do(req)
        if err != nil { return err }
        defer resp.Body.Close()
        if resp.StatusCode >= http.StatusInternalServerError {
            return fmt.Errorf("inventory status=%d", resp.StatusCode)
        }
        if resp.StatusCode != http.StatusOK {
            var payload map[string]any
            _ = json.NewDecoder(resp.Body).Decode(&payload)
            return fmt.Errorf("inventory rejected: %v", payload)
        }
        return nil
    })
}

Degradation policy component

type DegradePolicy struct { CouponDisabled bool; RecommendDisabled bool }

func (d DegradePolicy) AllowCoupon() bool { return !d.CouponDisabled }
func (d DegradePolicy) AllowRecommend() bool { return !d.RecommendDisabled }

Order service main flow (idempotency, budget, risk, inventory, optional coupon, async event)

func (s *OrderService) Submit(ctx context.Context, req SubmitOrderRequest) (*SubmitOrderResponse, error) {
    if req.RequestID == "" || req.UserID <= 0 || req.SKUId <= 0 || req.Quantity <= 0 {
        return nil, fmt.Errorf("invalid request")
    }
    // Idempotency check
    if orderID, ok, err := s.idempotency.GetResult(ctx, req.RequestID); err == nil && ok {
        return &SubmitOrderResponse{OrderID: orderID, CouponApplied: req.CouponID != ""}, nil
    }
    // Acquire lock for this request
    locked, err := s.idempotency.TryLock(ctx, req.RequestID, 2*time.Minute)
    if err != nil { return nil, err }
    if !locked { return nil, ErrDuplicateRequest }
    // Unified timeout budget for the whole flow
    ctx, cancel := stability.WithBudget(ctx, 900*time.Millisecond)
    defer cancel()
    // Risk check (core)
    if err := s.risk.Check(ctx, req.UserID); err != nil { return nil, err }
    // Inventory reservation (core)
    if err := s.inventory.Reserve(ctx, inventory.ReserveRequest{SKUId: req.SKUId, Quantity: req.Quantity}); err != nil { return nil, err }
    // Optional coupon (degradable)
    couponApplied := false
    if req.CouponID != "" && s.degrade.AllowCoupon() {
        if err := s.coupon.Consume(ctx, req.UserID, req.CouponID); err == nil { couponApplied = true }
    }
    // Create order ID
    orderID := generateOrderID()
    // Record idempotency result
    if err := s.idempotency.MarkDone(ctx, req.RequestID, orderID, 24*time.Hour); err != nil { return nil, err }
    // Publish async events (marketing, points, etc.)
    _ = s.publisher.PublishOrderCreated(ctx, orderID)
    return &SubmitOrderResponse{OrderID: orderID, CouponApplied: couponApplied}, nil
}

Distributed rate limiting with Redis + Lua

-- KEYS[1]: rate limit key, e.g. rl:order:tenant:1001
-- ARGV[1]: limit
-- ARGV[2]: window seconds
local current = redis.call("INCR", KEYS[1])
if current == 1 then redis.call("EXPIRE", KEYS[1], ARGV[2]) end
if current > tonumber(ARGV[1]) then return 0 else return 1 end

Use this script for tenant‑level or hotspot‑level quotas. Remember that Redis itself can become a bottleneck, so the recommended pattern is “global ingress quota + local token bucket”.

Engineering details that decide the system ceiling

Avoid making every downstream a synchronous strong dependency

Only core services (risk, inventory, order persistence, payment routing) should be verified synchronously. Non‑core capabilities (recommendations, profiling, analytics, marketing) should be async.

Back‑pressure is more important than an infinite queue

Queueing consumes memory, connections and timeout budget. Prefer short online paths, Kafka‑based peak‑shaving, bounded consumer concurrency, and proactive alerts when backlog grows.

Connection‑pool size is a hard capacity limit, not “the bigger the better”

Set MaxConnsPerHost based on downstream capacity; over‑provisioning can kill the downstream service.

Goroutine is cheap but blocking resources are not

Limit concurrent goroutines per high‑risk call, use worker pools for async jobs, reuse buffers, and sample logs to avoid OOM during incidents.

Data consistency, idempotency and compensation

Idempotent key for retry safety

Client sends requestId.

Server builds a key from requestId+userId+method and stores it in Redis with SETNX.

Successful execution writes the result; subsequent retries read and return the stored result.

Inventory‑order consistency pattern

Pre‑reserve inventory.

Create order.

After payment succeeds, deduct inventory.

If payment times out, release the reservation.

Compensation after degradation

If the coupon service is degraded, mark coupon_pending=true, push a compensation task to Kafka, let an async consumer retry, and fall back to manual compensation if retries exceed a threshold.

Kubernetes production rollout

ConfigMap for dynamic parameters

apiVersion: v1
kind: ConfigMap
metadata:
  name: order-api-config
data:
  config.json: |
    {
      "http": {"addr": ":8080", "readTimeout": "1s", "readHeaderTimeout": "500ms", "writeTimeout": "2s", "idleTimeout": "60s"},
      "stability": {"qpsLimit": 800, "burst": 1200, "maxInFlight": 300, "requestTimeout": "900ms", "degradeCoupon": false, "degradeRecommend": true},
      "downstream": {"inventory": {"baseURL": "http://inventory-service", "timeout": "150ms", "maxIdleConns": 200, "maxConnsPerHost": 150, "idleConnTimeout": "30s", "breakerWindow": "10s", "breakerTimeout": "30s", "breakerMinRequests": 20, "breakerErrorPercent": 0.5}}
    }

Deployment with readiness, preStop and resource limits

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: order-api
  template:
    metadata:
      labels:
        app: order-api
    spec:
      terminationGracePeriodSeconds: 40
      containers:
      - name: order-api
        image: registry.example.com/order-api:1.0.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "2"
            memory: "1Gi"
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]

HPA with multiple metrics (CPU + in‑flight requests)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-api
  minReplicas: 6
  maxReplicas: 30
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 65
  - type: Pods
    pods:
      metric:
        name: http_inflight_requests
      target:
        type: AverageValue
        averageValue: "180"

PodDisruptionBudget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: order-api-pdb
spec:
  minAvailable: 4
  selector:
    matchLabels:
      app: order-api

Observability

Essential metrics (grouped)

Traffic: QPS, entry‑rate‑limit count, tenant distribution.

Latency: P50/P95/P99, downstream RT, timeout count.

Stability: breaker open count, half‑open failures, degradation hits.

Resources: goroutine count, heap usage, GC pause, connection‑pool utilization.

Async: Kafka lag, consumer failures, compensation backlog.

Prometheus metric name examples:

http_requests_total
http_request_duration_seconds_bucket
stability_rate_limited_total
stability_breaker_open_total
stability_degrade_hit_total
http_inflight_requests
downstream_request_timeout_total

Actionable alerts

RATE_LIMITED – traffic burst; first action: raise entry‑level limit or enable activity degradation.

Breaker opened frequently – downstream slowdown or high error rate; first action: switch degradation, investigate downstream.

inflight_requests high – slow calls piling up; first action: reduce timeout, check hot dependencies.

Kafka lag rising – consumer capacity shortage; first action: scale consumer group or temporarily lower sampling rate.

Trace usage

Identify which component consumes the timeout budget.

Distinguish entry queueing from downstream latency.

Spot tenant‑ or product‑specific anomalies.

Verify that degradation policies are actually applied.

Load testing and capacity validation

Test scenarios

Steady‑state load – verify RT, error rate and resource usage remain stable.

Spike test – increase traffic 3‑5× for a short period, check that rate limiting activates and no cascade failures occur.

Fault injection – deliberately add latency or 5xx errors to inventory-service, observe breaker and degradation behavior.

Recovery test – after fault removal, ensure half‑open probes, scaling and cache warm‑up happen smoothly.

Metrics to watch

RT: P99 linear growth vs sudden cliff.

Error rate: Controlled rate‑limit rejections, absence of massive 5xx.

Breaker: Early opening on upstream failure.

Degradation: Core path recovers after non‑core capabilities are turned off.

Resources: No unbounded growth of goroutine, memory or connections.

Fault‑injection examples

Add 300 ms delay to inventory-service.

Make coupon-service return 500 for 50 % of calls.

Reduce Redis QPS quota.

Shrink DB connection pool.

Take 30 % of pods offline and observe PDB and HPA behavior.

Common pitfalls

Using retries to mask downstream overload – amplifies pressure.

Applying the same thresholds to all interfaces – core and non‑core traffic have different SLAs.

Breaker returning plain 500 without business fallback – turns a recoverable issue into a hard failure.

Only application‑level limiting, no ingress quota – hot tenants or bots can starve real users.

Relying solely on CPU‑based HPA – ignores I/O bottlenecks and downstream capacity.

Readiness probe without graceful pre‑stop – pods are killed while still handling traffic.

Evolution roadmap

Stage 1: Local rate limiting, unified timeout, idempotency, core‑service circuit breakers, basic Prometheus metrics.

Stage 2: Global ingress quota, dynamic config center, Kafka async shaping, dependency isolation, degradation switches.

Stage 3: Distributed hot‑parameter limiting, per‑tenant quotas, multi‑region active‑active, automated chaos‑engineering platform.

Practical rollout checklist

Add a unified timeout budget and standard error codes to all entry points.

Protect core interfaces with a local token bucket and maxInFlight limit.

Give each critical downstream its own HTTP client, connection pool and circuit breaker.

Convert non‑core synchronous calls to degradable or asynchronous paths.

Introduce idempotency keys for order creation, payment and inventory reservation.

Externalize all limit values, breaker thresholds and degradation flags to configuration.

Expose Prometheus metrics for rate limiting, breaker state, timeouts, RT and resource usage.

Complete Kubernetes readiness, preStop, PodDisruptionBudget and HPA configurations.

Run a fault‑injection drill to verify degradation and recovery procedures.

Derive final thresholds from load‑test data instead of intuition.

Final takeaway

Stability governance is not about eliminating failures; it is about making failures predictable, isolated and quickly recoverable. By answering the six fundamental questions and implementing the layered mechanisms described above, a Go microservice can survive traffic spikes, downstream degradations and resource exhaustion while keeping the core business flow alive.