Nacos Service Discovery & Config Center: AP Model, Heartbeats & Hot Reload Explained

Two Opening Incidents

Incident 1: During a scale-down, two instances disappeared from the Nacos console but the gateway kept sending 30% traffic to the terminated pods for five minutes, resulting in 502 errors. Root cause: instance removal has a latency window, and the gateway cache did not invalidate in time.

Incident 2: A rate-limit threshold was changed from 100 to 50 in the config center, yet the service QPS remained at 100. Investigation revealed the config was written to the public namespace while the service read from dev; moreover, the reading bean lacked a refreshable scope, so the new value was fetched but never applied.

Key Insight: These two incidents map to Nacos' two core mechanisms: service discovery relies on heartbeat + push + local cache — any delay causes misrouted traffic; config management relies on the namespace/group/dataId triplet + long polling + refresh scope — any misalignment causes "changed but not effective".

Technical Analysis: Registry Capability Boundaries

Nacos: Supports AP/CP switching (ephemeral instances use Distro eventual consistency; persistent instances use Raft strong consistency). Combines registry and config, friendly console, mature Chinese ecosystem. Drawback: large-scale push pressure and memory tuning required.

Eureka: Pure AP, async replication, 30s client cache, high availability but weak consistency; no new development; not recommended for new projects.

ZooKeeper: CP, ephemeral nodes + Watch, session disconnect deletes immediately, strong consistency; low write performance, unavailable during split-brain; suited for leader election and metadata, not high-frequency registration.

Consul: Primarily CP (can be AP), built-in health checks and multi-datacenter, gRPC/DNS interfaces; smaller adoption in China than Nacos.

Selection Conclusion: Internet businesses choose Nacos (AP first, brief inconsistency acceptable); financial ledger scenarios requiring strong consistency can mark key instances as persistent to use CP. The essence of selection is not "which is stronger" but how long you can tolerate instance view inconsistency .

Ephemeral vs Persistent Instances

Ephemeral (ephemeral=true, default): Client sends heartbeats; server does not persist; missing heartbeats trigger removal. Suits stateless microservices.

Persistent (ephemeral=false): Server actively probes (TCP/HTTP), data persisted via Raft; instance survives client disconnect. Suits databases, middleware needing manual confirmation before removal.

Distro Protocol: Nacos' custom AP protocol — each node owns a data shard; writes route to the responsible node then async replicate; periodic full checksum repair. Pros: non-blocking writes, high availability. Cost: second-level inconsistency window .

Core Workflows: Registration, Heartbeat, Discovery, Config

3.1 Service Registration

App startup → NacosServiceRegistry.register assembles Instance (IP, port, weight, cluster, metadata) → registers with server → server writes in-memory registry and syncs via Distro → notifies subscribed clients (push).

3.2 Heartbeat & Health Status

1.x: Client sends HTTP heartbeat every 5s; 15s no heartbeat → mark unhealthy; 30s no heartbeat → remove instance.

2.x: Switches to gRPC long-lived connection; connection alive = healthy; disconnect = immediate removal. Heartbeat overhead drops from "HTTP every 5s" to "connection keep-alive", drastically reducing registry load.

Client side: Subscribed instance list cached locally; server pushes changes; if registry entirely unavailable, client continues calling via local snapshot — the key AP fallback.

3.3 Config Location & Push

Triplet Location: namespace (environment isolation) → group (business grouping) → dataId (filename, usually ${spring.application.name}.${file-extension}). Any difference yields a distinct config.

Long Polling (1.x): Client sends 30s hanging request with local MD5; server holds 29.5s, returns new MD5 on change, else empty — real-time without empty-loop pressure.

gRPC Bidirectional Stream (2.x): Server pushes changes immediately.

Refresh: On change → publish RefreshEvent → @RefreshScope beans destroyed and rebuilt on next access, new values take effect.

3.4 Minimal Client Configuration

spring:
  application:
    name: mall-order
  config:
    import:
      - optional:nacos:mall-order.yaml
      - optional:nacos:common.yaml
  cloud:
    nacos:
      server-addr: 10.0.0.1:8848,10.0.0.2:8848,10.0.0.3:8848
      username: nacos
      password: ${NACOS_PWD}
      discovery:
        namespace: ${NS_ID}
        group: ORDER_GROUP
        ephemeral: true
      config:
        namespace: ${NS_ID}
        group: ORDER_GROUP
        file-extension: yaml
        refresh-enabled: true

Group Trap: discovery and config groups are independent; changing only one causes "service discoverable but config not fetched". Namespace must be the ID (UUID); using the name falls back to public .

Core Source Code: Four Key Implementations

4.1 Registration Entry: NacosServiceRegistry

// spring-cloud-alibaba registration
public void register(
    Registration registration) {
    Instance instance = NacosService
    InstanceUtils
    .getInstance(registration);
    // with retry: registry jitter won't crash startup
    namingService.registerInstance(
        instance.getServiceName(),
        instance.getGroupName(),
        instance);
    // metadata carries preserved.register
    // .source=SPRING_CLOUD for tracing
}

4.2 Server Heartbeat Handling

// InstanceController.beat (1.x)
// heartbeat only refreshes timestamp, no DB write
// high-frequency writes would blow storage
beat.getServiceName();
// lastBeat = now
// 15s no update -> healthy=false
// 30s no update -> remove + push

// 2.x: connection disconnect event triggers removal directly
// ClientConnectionEventListener
// -> clientDisconnected ->
//      removeInstance(...)

4.3 Config Long Polling: ClientWorker

// client scheduled task every 30s
// LongPollingRunnable.run():
// 1. collect all listened dataIds
//      with local MD5
// 2. POST /v1/cs/configs/listener
// server holds 29.5s
// 3. return changed list -> pull new config
// write to local snapshot dir
// nacos/config/ (crash fallback)
// 4. publish RefreshEvent
// 5. immediately start next round -> near real-time

4.4 Refresh Chain: NacosContextRefresher

// NacosContextRefresher registers listener per dataId:
configService.addListener(dataId, group,
    new AbstractListener() {
    public void receiveConfigInfo(
        String info) {
        // publish refresh event
        publishEvent(new RefreshEvent(
            this, "nacos",
            dataId, group));
    }
});
// RefreshEvent -> ContextRefresher
// .refresh() -> rebuild @RefreshScope
// Bean + rebind @ConfigurationProperties

Config Hot Reload: Three Must-Understand Points

1. @RefreshScope is "lazy rebuild". It places the bean in a refresh scope; on RefreshEvent the bean is marked stale and rebuilt only on next injection/access. Hence "config changed but log still shows old value" is normal until the bean is invoked again.

2. @ConfigurationProperties natively supports refresh. If the class is a bean and refresh enabled, property binding re-runs on refresh, usually easier to maintain than scattered @Value.

3. Connection pools / thread pools do NOT auto-rebuild. Druid, HikariCP, Tomcat thread pools — initialized at startup — are not recreated by refresh events. Dynamic datasources must listen events and rebuild manually (see dynamic datasource article).

@RefreshScope
@RestController
public class RateLimitController {

    @Value("${limit.qps:100}")
    private int qps;

    @GetMapping("/limit")
    public int limit() {
        return qps; // new value read after refresh
    }
}
// Preferred: @ConfigurationProperties
// + @Validated for unified management & validation

Production Deployment: Cluster, Auth & Isolation

Minimum Cluster Size: 3 nodes (odd number for Raft election); client address list must include all three to avoid single point.

External Storage: Production must use MySQL for config and user data persistence; embedded Derby only for single-node trial; schema upgrade scripts must be run on version upgrades.

Authentication: Enable nacos.core.auth.enabled and set token.secret.key; otherwise anyone reaching port 8848 can modify configs — the most overlooked high-risk item.

# cluster.conf (identical on each node)
10.0.0.1:8848
10.0.0.2:8848
10.0.0.3:8848

# application.properties key items
spring.datasource.platform=mysql
db.num=1
nacos.core.auth.enabled=true
nacos.core.auth.plugin
.nacos.token.secret.key=${SECRET}
# JVM: fixed heap to avoid resize jitter
-Xms4g -Xmx4g -Xmn2g

Correct Environment Isolation: Use namespace for dev/test/prod isolation, group for business domain isolation, dataId for app vs shared config (common.yaml via shared-configs). Do NOT use "different dataId prefixes under same namespace" for environment isolation — that's only a naming convention, not physical isolation.

Eight High-Frequency Pitfalls

Pitfall 1: Namespace filled with name. Console shows name, API expects ID; mistake registers into public, causing cross-env service bleed.

Pitfall 2: Discovery and config group mismatch. Service discoverable but config not fetched, or vice versa. Define both, preferably as variables.

Pitfall 3: Missing spring.config.import . Boot 2.4+ no longer auto-fetches config center; without this line configs are inert, and startup logs no error.

Pitfall 4: @Value doesn't get new value. Missing @RefreshScope or bean is singleton that cached value at startup; refresh doesn't retroactively update.

Pitfall 5: Ephemeral instance for stateful service. Pod restart triggers removal and re-registration, causing blips in long-lived connections; stateful services should use persistent instances or custom registration logic.

Pitfall 6: Traffic still hits after scale-down. Removal latency + K8s preStop + gateway cache TTL (see graceful shutdown article).

Pitfall 7: Shared config override order confusion. shared-configs and extension-configs have priority; app's own dataId highest; avoid duplicate definitions.

Pitfall 8: No auth on public exposure. Exposing 8848/9848 equals arbitrary config tampering. Internal network + auth + network policies — all three required.

Troubleshooting Three-Step

Check service list for instance health, verify namespace/group match; config page confirm dataId and group hit.

Enable Nacos client DEBUG logs (naming/config packages) to see if push received, MD5 changed.

Inspect local snapshot dir nacos/config and naming cache to distinguish "not fetched" vs "fetched but not applied".

Learning Path & Interview Questions

8.1 Four-Stage Roadmap

Beginner (3 days): Stand up single-node Nacos, run registration + config pull, understand triplet and config.import.

Intermediate (1 week): 3-node cluster + MySQL persistence + auth; verify ephemeral/persistent differences and health-check timing.

Advanced (2 weeks): Read Distro sync, heartbeat handling, ClientWorker long polling, ContextRefresher refresh chain; simulate network partition to validate local snapshot fallback.

Practice (ongoing): Design config layering (common/app/env), canary release flow, config change audit and rollback plan.

8.2 Twelve Interview Questions (with Answers)

Q1 · Where are Nacos AP and CP used?

Answer: Ephemeral instances use AP (Distro, async replication, eventual consistency) for registry HA; persistent instances use CP (Raft) to guarantee instance data durability. Most stateless microservices use AP; strong-consistency registration and metadata use CP.

Q2 · What is the core idea of Distro protocol?

Answer: Data sharded by service; each node owns a shard; writes route to owner then async replicate; nodes periodically full-checksum and repair differences. Trades strong consistency for availability and throughput, hence second-level instance view inconsistency.

Q3 · Nacos 1.x vs 2.x communication differences?

Answer: 1.x uses HTTP short connections: registration/heartbeat/query are separate requests, 5s heartbeat; 2.x introduces gRPC long connection, registration/subscription/config listening share connection, disconnect triggers immediate removal, overhead and latency significantly reduced.

Q4 · Ephemeral vs persistent instance differences and use cases?

Answer: Ephemeral maintained by client heartbeat, deleted on disconnect, suits stateless microservices; persistent probed by server and persisted, survives client disconnect, suits databases/middleware needing manual confirmation before removal.

Q5 · Why does traffic still hit after service offline?

Answer: Instance removal latency (heartbeat timeout or disconnect detection), client local cache not expired, gateway/load-balancer cache TTL not elapsed — all three stack. Fix: shorten removal detection, combine preStop for pre-drain, control cache refresh frequency.

Q6 · Why can Nacos client still call when registry is down?

Answer: Subscribed instance list persisted in local cache (memory + snapshot file); when server unavailable, client reads cache and continues calling; config similarly has local snapshot fallback. This is the core AP value, but also means possible calls to already-removed instances.

Q7 · Meanings of config namespace, group, dataId?

Answer: Namespace for environment/tenant isolation (physical, must use ID), group for business grouping, dataId for config filename (usually app name + suffix). Together they uniquely locate a config; any difference yields a separate config.

Q8 · How is Nacos config real-time push implemented?

Answer: 1.x uses long polling: client sends 30s hanging request with MD5, server returns immediately on change, else after 29.5s returns empty, client pulls full config on change. 2.x uses gRPC bidirectional stream, server pushes directly, more real-time and resource-efficient.

Q9 · @RefreshScope principle? Why sometimes config change not effective?

Answer: Places bean in refresh scope; on RefreshEvent marks stale, rebuilds on next access with re-bound properties. Common failures: bean missing annotation, value cached in static variable/connection pool at startup, or config not located (namespace/group/dataId wrong).

Q10 · Why don't connection pools / thread pools refresh?

Answer: These resources initialize at startup; refresh only rebuilds beans and rebinds properties, does not recreate existing pool objects. Must listen EnvironmentChangeEvent and manually shutdown then rebuild.

Q11 · Production Nacos cluster deployment essentials?

Answer: At least 3 nodes (odd) for Raft election, client writes full address list; external MySQL persistence; enable auth and set token secret; uniform JVM heap; 8848/9848 internal only with network policies. Embedded Derby only for single-node trial.

Q12 · How to choose among Nacos, Eureka, ZooKeeper?

Answer: Need AP + integrated config center → Nacos; only extreme availability, can accept weak consistency → Eureka (deprecated, not recommended for new projects); need strong consistency, leader election & metadata → ZooKeeper. Choice depends on how long you can tolerate instance view inconsistency.

8.3 One-Page Cheatsheet

Two mechanisms: Registry (Distro AP)
Config (Long polling / Push)
Two instance types: Ephemeral (heartbeat, delete on disconnect)
Persistent (probe, persist Raft)
Health timing 1.x: 5s heartbeat / 15s unhealthy / 30s remove; 2.x watches connection
Config triplet: namespace / group / dataId (namespace must be ID)
Refresh three pitfalls: missing config.import
missing @RefreshScope
connection pools won't rebuild
One-liner: Registry tolerates second-level inconsistency,
Config must locate precisely

Summary

Nacos packs "service discovery" and "config management" into one process, but they are two completely different mechanisms: one uses AP for availability, the other uses long polling for real-time. Mastering heartbeat timing, triplet location, and refresh scope avoids the vast majority of registry and config incidents. Next up: the invocation layer — OpenFeign & Load Balancing: Proxy Internals & Timeout Governance .