How We Built a Scalable Configuration Center for Weibo Advertising with Consul

1. What Is a Configuration Center?

A configuration center centralizes dynamic runtime settings for services, allowing changes without restarting or recompiling applications. It evolved from single‑machine config files to distributed KV stores with micro‑service architectures.

2. What Can It Do?

The center provides unified configuration management, gray‑release, full‑update and rollback, KV‑style storage, and can incorporate service registration, effectively acting as a combined middleware layer.

3. Implementation Overview

The configuration center is integrated into the Kunkka automation‑ops platform and offers configuration publishing, service registration, service‑tree management, environment‑variable handling, and operation‑history tracking.

It relies on a Consul cluster (≈20 service units, >500 servers) for service discovery and KV storage.

Service Registration Example – Redis

curl -XPUT -H "X-Consul-Token: xxx-xxx-xx-xxxx-xx" http://localhost:8500/v1/agent/service/register

{
  "id": "redis",
  "name": "redis",
  "tags": ["primary"],
  "address": "10.13.x.x",
  "meta": "for my service",
  "weight": "50",
  "port": 8000,
  "client_addr": "0.0.0.0",
  "enable_tag_override": false,
  "checks": [{
    "args": ["/usr/local/bin/check_redis.py"],
    "interval": "10s"
  }]
}

This registers a Redis instance with Consul, including metadata, health checks, and tagging for easy discovery.

Docker Service Registration with Registrator

docker run -d --name=registrator \
  -v /var/run/docker.sock:/tmp/docker.sock \
  -e CONSUL_HTTP_TOKEN="xxxx-xxxx-xxxx-xxxx" \
  --net=host gliderlabs/registrator \
  -tags="`hostname`" \
  --deregister=always \
  -ip="10.xx.xx.xx" consul://10.xx.xx.xx:8500

Registrator automatically discovers Docker containers and registers them to Consul (or Etcd, Zookeeper, etc.).

4. Configuration Publishing

For load balancing, the team adopted Consul‑Template, orchestrated by the Kunkka platform, to generate configuration files on each host dynamically, with instant rollback capability.

Beyond load balancing, other service‑to‑service configurations are managed via Consul + Consul‑Template + SaltStack. Two KV directories— app and env —link services with environment variables, which are then pushed to servers through the SaltStack API.

5. Access Control

Initially, ACLs were disabled, leading to chaotic service naming. After upgrading to Consul 1.4, ACLs were enabled, a naming convention ( product‑service) was enforced, and tokens were generated per service with read/write policies.

Index Service Permission Example

# Service policy (default read, write for specific service)
Service "aladdin-index" {
  Policy = "write"
}
# Tokens are created by linking the policy in the "Apply an existing policy" UI.

6. Cluster Monitoring

Consul Telemetry streams metrics to a local node exporter, which Prometheus scrapes and visualizes in Grafana.