Mastering Microservices: RPC, Service Discovery, Config, Scheduling & More

Benefits of Microservices

Microservices enable cross‑team decoupling, higher concurrency (beyond the single‑machine c10k limit), shared base services, better service governance, and smoother compatibility with cloud platforms.

Remote Procedure Call (RPC)

RPC lets a client invoke remote functions as if they were local methods. The client typically creates a dynamic proxy that serializes the interface name, parameters, and method call, decides between synchronous or asynchronous execution, and handles callbacks. The client also manages load balancing, timeout handling, and connection‑pool management, discarding connections to failed servers. The server deserializes the request, uses reflection to invoke the target method, and returns the result.

Serialization can be field‑only (space‑saving) or whole‑object binary (faster deserialization). The trade‑off mirrors Huffman coding or database row storage.

Service Discovery (Registration Center)

Three common registration patterns exist:

F5 as a centralized proxy.

Embedded client‑side proxies such as Dubbo.

Hybrid mode where multiple clients share a proxy process (Service Mesh).

Zookeeper sacrifices availability for strong consistency, which is unnecessary for a registration center; it also becomes unavailable when a majority of nodes fail, making it a poor choice for this role.

Configuration Center

A configuration center must provide high availability, real‑time notifications, gray releases, access control, one‑click rollback, and environment isolation. Popular open‑source solutions include Nacos, Disconf, and Apollo.

Disconf modules:

scan – scans annotations and listeners.

store – persists remote configs locally and notifies listeners on change.

fetch – retrieves configs via HTTP.

watch – watches Zookeeper nodes for changes and triggers fetch.

Apollo consists of four modules:

portal – management UI with its own database.

adminservice – core config modification and release service (shares DB with configservice).

configservice – polls the DB for new release messages and pushes them to clients (uses DeferredResult for async).

eruka – provides registration and discovery for adminservice and configservice.

Task Scheduling

The executor (application) registers task units (threads) with the scheduler, which can elect a master node and run in a cluster to avoid a single point of failure. Features include load‑balanced task assignment, failure retry, executor removal on disconnection, concurrency control, task dependencies, sharding, cancellation, and a “glue” mode that runs tasks without a new deployment.

Distributed Lock

Redis can implement a lock with SET resource_name my_random_value NX PX 30000, storing the lock owner in the value to avoid accidental release. Zookeeper uses temporary nodes; other threads watch the node for lock status.

Unified Monitoring

Log collection and analysis can be correlated with RPC traces, filtered or compressed, and exposed via APIs or Java‑agent interceptors. Implementations may use OpenTracing, a Disruptor ring‑buffer for producer‑consumer patterns, Elasticsearch for massive log storage, and generate dashboards and metrics for RPC, databases, CPU, HTTP status, and middleware.

Cache Strategies

When updating data, deleting the cache first then updating the DB avoids stale reads, but may cause cache‑penetration under high traffic. Updating the cache without deletion can lead to permanent inconsistency. Recommended practices include multi‑level caching, setting varied TTLs to prevent cache avalanche, and using different expiration times to mitigate cache penetration.

Message Queue

To guarantee ordering, a single producer sends to one broker and one queue; RocketMQ lets the producer choose the queue per order key. Deduplication relies on idempotent processing or a deduplication table. Sending flow:

Fetch topic routing info from NameServer (cached locally).

If the topic does not exist, auto‑create it via NameServer and broker.

Select a queue based on routing strategy and broker health.

Build the send request with body, queue, and topic; assign a message ID.

On repeated failures, move the message to a dead‑letter queue.

Message storage uses 1 GB commit‑log files; offsets are calculated modulo 1 GB. Files are pre‑warmed by writing a byte to each 4 KB page. Two write methods exist: FileChannel.map for a MappedByteBuffer or an off‑heap memory pool with explicit flush.

Distributed Transaction

Two‑phase commit (2PC) involves a transaction manager and resource managers, but suffers from poor performance. Compensation transactions and TCC (Try‑Confirm‑Cancel) provide alternatives. Seata intercepts business SQL, generates undo/redo logs, and creates row locks. In the second phase, it deletes logs on commit or replays undo logs on rollback, checking for dirty writes. The global transaction ID (XID) is propagated via RPC and stored in ThreadLocal.

CAP Theorem

CAP states that a distributed system can only simultaneously provide two of Consistency, Availability, and Partition tolerance. Zookeeper chooses Partition tolerance and eventual consistency, sacrificing strong consistency and full availability. Traditional databases illustrate CP (master‑slave sync) and AP (asynchronous replication) trade‑offs.

Seckill Handling

Techniques include static‑dynamic separation (AJAX, CDN), hotspot detection, business and database isolation, fallback mechanisms (rate limiting, degradation), traffic shaping (queues, captchas, MQ), and inventory reduction strategies (optimistic DB lock, order‑time reservation, rollback on non‑payment). Direct SQL checks prevent overselling but can cause row‑level lock contention and high CPU usage.

Docker Isolation

Docker can assign namespaces (PID, mount, network) so a container only sees resources within its namespace. Cgroups limit CPU, memory, and network usage for a group of processes. chroot changes the root filesystem, providing an isolated environment. Docker images are built as layered filesystems; each modification adds a new read‑write layer on top of immutable lower layers, merged via a union‑fs. Deleting a file from a lower layer is achieved by creating a whiteout file in the top layer.

Release & Deployment

Typical CI/CD pipelines:

Create a pipeline specifying project name, tag, and dependencies.

Pull the latest code from GitLab.

Package with Maven (optionally in a dedicated workspace).

Build a Docker image from the Maven artifact and push it to a registry.

Deploy to a test environment via the Kubernetes API.

Promote to production using canary or gray‑release strategies.