Deep Dive into gRPC: Load Balancing, Service Discovery, and Pitfall Guide

10. Error Handling

10.1 Status Codes

OK (0) – Success

CANCELLED (1) – Call cancelled by client

UNKNOWN (2) – Uncaught server exception

INVALID_ARGUMENT (3) – Request validation failure

DEADLINE_EXCEEDED (4) – Deadline reached

NOT_FOUND (5) – Resource not found

ALREADY_EXISTS (6) – Duplicate creation

PERMISSION_DENIED (7) – Authenticated but not authorized

UNAUTHENTICATED (16) – Missing or invalid token

RESOURCE_EXHAUSTED (8) – Quota or rate‑limit exhausted

FAILED_PRECONDITION (9) – Invalid state transition

ABORTED (10) – Concurrency conflict

OUT_OF_RANGE (11) – Pagination overflow

UNIMPLEMENTED (12) – Method not implemented

INTERNAL (13) – Internal server error (e.g., NPE)

UNAVAILABLE (14) – Service down or connection failure

DATA_LOSS (15) – Data loss (e.g., storage failure)

10.2 Throwing Errors (Server)

import io.grpc.Status;
import io.grpc.protobuf.StatusProto;

// Simple status
public void predict(PredictRequest req, StreamObserver<PredictResponse> rsp) {
    if (req.getUserId() <= 0) {
        rsp.onError(Status.INVALID_ARGUMENT
            .withDescription("user_id must be positive")
            .asRuntimeException());
        return;
    }
    // ...
}

// Rich error with google.rpc.Status
import com.google.rpc.ErrorInfo;
com.google.rpc.Status status = com.google.rpc.Status.newBuilder()
    .setCode(Code.INVALID_ARGUMENT.getNumber())
    .setMessage("user_id invalid")
    .addDetails(Any.pack(ErrorInfo.newBuilder()
        .setReason("USER_ID_NEGATIVE")
        .setDomain("algo-service")
        .build()))
    .build();
rsp.onError(StatusProto.toStatusRuntimeException(status));

10.3 Client Capture

try {
    PredictResponse resp = stub.predict(req);
} catch (StatusRuntimeException e) {
    Status s = e.getStatus();
    Metadata md = e.getTrailers(); // custom metadata
    switch (s.getCode()) {
        case DEADLINE_EXCEEDED -> retryOrFallback();
        case UNAVAILABLE -> retryWithBackoff();
        case INVALID_ARGUMENT -> throw new BizException(s.getDescription());
        case UNAUTHENTICATED -> refreshTokenAndRetry();
        default -> throw new RuntimeException(e);
    }
}

11. Load Balancing

11.1 Three LB Modes

Mode 1: Proxy (L7)      Mode 2: Client‑side LB      Mode 3: xDS (service mesh)
Client → LB(Nginx/Envoy)   Client ──┐                Client ──┐
          │                ├─ pick instance   ├─ xDS server pushes
          └→ Server        └─ Server          └─ Server
(Extra hop, client simple)   (No extra hop)      (Control plane unified)

11.2 Client‑Side Load Balancing (gRPC Java)

gRPC ships with NameResolver + LoadBalancer SPI. The default policy is pick_first (select the first ready address and stick to it).

ManagedChannel c = ManagedChannelBuilder
    .forAddress("dns:///algo-service:9090") // DNS resolves multiple A records
    .defaultLoadBalancingPolicy("round_robin") // round‑robin
    .build();

pick_first

– sequentially try addresses, use the first that connects; all subsequent calls go to that instance. round_robin – distribute calls across all ready instances; requires every instance to be in READY state. grpclb – delegate to an external gRPC load‑balancer service (being superseded by xDS).

Pitfall of pick_first: if the first instance crashes traffic switches to the next, but under normal conditions only one instance receives traffic, leaving others idle. Use round_robin for true load distribution.

11.3 Proxy LB

Deploy Envoy or Nginx as an L7 proxy; the gRPC client connects only to the proxy. Advantages: zero client awareness. Disadvantages: an extra hop and the proxy must maintain long‑lived connections. Nginx 1.13.10+ supports native gRPC via grpc_pass.

11.4 xDS

Cloud‑native approach: the client receives service‑discovery, load‑balancing, and circuit‑breaking policies from an xDS control plane (e.g., Istiod). Switching to xDS only requires changing the target:

ManagedChannel c = ManagedChannelBuilder
    .forTarget("xds:///algo-service")
    .build();

12. Service Discovery & Registration

12.1 NameResolver SPI

gRPC provides a DNS resolver that parses DNS A records. To integrate with Nacos, Consul, or Eureka, implement a custom NameResolverProvider and NameResolver:

public class NacosNameResolverProvider extends NameResolverProvider {
    @Override
    public NameResolver newNameResolver(URI targetUri, NameResolver.Args args) {
        return new NacosNameResolver(targetUri.getAuthority(), namingService, args);
    }
    @Override
    public String getDefaultScheme() { return "nacos"; }
}

class NacosNameResolver extends NameResolver {
    @Override
    public void start(Listener2 listener) {
        namingService.subscribe(serviceName, event -> {
            List<Instance> instances = namingService.getAllInstances(serviceName);
            List<EquivalentAddressGroup> addrs = instances.stream()
                .map(i -> new EquivalentAddressGroup(new InetSocketAddress(i.getIp(), i.getPort()), Attributes.newBuilder().build()))
                .toList();
            listener.onAddresses(addrs, Attributes.EMPTY);
        });
    }
    @Override public String getServiceAuthority() { return serviceName; }
    @Override public void shutdown() { /* cancel subscription */ }
}

After registering the provider on the classpath, the target nacos:///algo-service resolves via Nacos:

ManagedChannel c = ManagedChannelBuilder
    .forTarget("nacos:///algo-service")
    .defaultLoadBalancingPolicy("round_robin")
    .build();

12.2 Difference with OpenFeign / Dubbo

Resolution timing : OpenFeign + Nacos resolves before each Ribbon call; Dubbo resolves before each call; gRPC NameResolver pushes updates to the load‑balancer.

Connection model : OpenFeign creates a new HTTP connection (or pool) per call; Dubbo uses a shared long‑lived connection; gRPC reuses a single HTTP/2 connection for all RPCs.

Client implementation : OpenFeign relies on Ribbon/LoadBalancer; Dubbo has built‑in mechanisms; gRPC uses NameResolver + LoadBalancer SPI.

13. Circuit Breaking & Rate Limiting

13.1 Integration at Interceptor Layer

gRPC does not provide built‑in circuit breaking. Implement it with interceptors that delegate to Sentinel (server side) or Resilience4j (client side).

// Server‑side rate limiting (Sentinel)
public class SentinelServerInterceptor implements ServerInterceptor {
    @Override
    public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(
            ServerCall<ReqT, RespT> call, Metadata headers, ServerCallHandler<ReqT, RespT> next) {
        Entry entry = null;
        try {
            entry = SphU.entry(call.getMethodDescriptor().getFullMethodName());
            return new ForwardingServerCallListener.SimpleForwardingServerCallListener<ReqT, RespT>(next.startCall(call, headers)) {
                @Override public void onComplete() { if (entry != null) entry.exit(); super.onComplete(); }
                @Override public void onCancel() { if (entry != null) entry.exit(); super.onCancel(); }
            };
        } catch (BlockException e) {
            call.close(Status.RESOURCE_EXHAUSTED
                .withDescription("rate limited: " + e.getClass().getSimpleName())
                .asRuntimeException(), new Metadata());
            return new ServerCall.Listener<>() {};
        }
    }
}

// Client‑side circuit breaker (Resilience4j)
public class CircuitBreakerClientInterceptor implements ClientInterceptor {
    private final CircuitBreaker cb;
    @Override
    public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
            MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
        if (!cb.tryAcquirePermission()) {
            call.cancel("circuit open", null);
            return new ClientCall<>() {};
        }
        return new ForwardingClientCall.SimpleForwardingClientCall<ReqT, RespT>(next.newCall(method, callOptions)) {
            @Override public void start(Listener<RespT> responseListener, Metadata headers) {
                super.start(new ForwardingClientCallListener.SimpleForwardingClientCallListener<RespT>(responseListener) {
                    @Override public void onClose(Status status, Metadata trailers) {
                        if (status.getCode() == Status.Code.UNAVAILABLE ||
                            status.getCode() == Status.Code.DEADLINE_EXCEEDED) {
                            cb.onError(status.getCause());
                        } else {
                            cb.onSuccess();
                        }
                        super.onClose(status, trailers);
                    }
                }, headers);
            }
        };
    }
}

13.2 Rate‑Limit Strategy Highlights

Limit dimension by method name ( grpc.method) to avoid a single slow method exhausting the global quota. UNAVAILABLE and DEADLINE_EXCEEDED count toward circuit‑breaker failure rate; INVALID_ARGUMENT does not because it is a client‑side validation error.

When resources are exhausted, return RESOURCE_EXHAUSTED so the client can degrade gracefully.

14. Security

14.1 TLS Encryption

// Server side
Server server = NettyServerBuilder.forPort(9090)
    .sslContext(GrpcSslContexts.forServer(certFile, keyFile).build())
    .addService(new AlgoServiceImpl())
    .build();

// Client side
ManagedChannel channel = ManagedChannelBuilder.forAddress("algo-service", 9090)
    .sslContext(GrpcSslContexts.forClient().trustManager(caFile).build())
    .build();

Production should enable mutual TLS (mTLS) so the client also presents a certificate, typically managed by an internal PKI.

14.2 Token / JWT Interceptor

Tokens travel via metadata. HTTP/2 requires lower‑case keys:

static final Metadata.Key<String> AUTH_TOKEN_KEY =
    Metadata.Key.of("authorization", Metadata.ASCII_STRING_MARSHALLER);

14.3 Security Layers

Transport layer : TLS / mTLS

Authentication : JWT / OAuth2 injected by an interceptor

Authorization : Method‑level ACL enforced after authentication

Rate limiting : Sentinel or Resilience4j

Auditing : Logging interceptor records method, caller, and status

15. Java Practical Demo

15.1 Project Structure & Dependencies

grpc-demo/
├── proto/                # .proto contracts (independent module)
│   └── algo_v1.proto
├── grpc-server/          # server module
│   └── src/main/java/.../AlgoServiceImpl.java
└── grpc-client/          # client module
    └── src/main/java/.../AlgoClient.java

Spring Boot 3.2.4 + JDK 21 with net.devh:grpc-spring-boot-starter (community edition) for automatic wiring. Maven properties include:

<properties>
    <java.version>21</java.version>
    <grpc-spring-boot-starter.version>3.1.0.RELEASE</grpc-spring-boot-starter.version>
    <protobuf-java.version>3.25.2</protobuf-java.version>
    <os-maven-plugin.version>1.7.1</os-maven-plugin.version>
    <protobuf-maven-plugin.version>0.6.1</protobuf-maven-plugin.version>
</properties>

<dependencies>
    <dependency>
        <groupId>net.devh</groupId>
        <artifactId>grpc-spring-boot-starter</artifactId>
        <version>${grpc-spring-boot-starter.version}</version>
    </dependency>
    <dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <version>${protobuf-java.version}</version>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.xolstice.maven.plugins</groupId>
            <artifactId>protobuf-maven-plugin</artifactId>
            <version>${protobuf-maven-plugin.version}</version>
            <configuration>
                <protocArtifact>com.google.protobuf:protoc:3.25.2:exe:${os.detected.classifier}</protocArtifact>
                <pluginId>grpc-java</pluginId>
                <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.63.0:exe:${os.detected.classifier}</pluginArtifact>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>compile-custom</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

15.2 Full proto

syntax = "proto3";
package algo.v1;
option java_package = "com.example.algo.v1.grpc";
option java_multiple_files = true;

message PredictRequest { int64 user_id = 1; string scene = 2; repeated string tags = 3; }
message PredictResponse { double score = 1; repeated string reasons = 2; }
message WatchRequest { string path = 1; }
message LogEvent { string line = 1; }
message Metric { string name = 1; double value = 2; }
message UploadSummary { int64 count = 1; }
message ChatMessage { string user = 1; string text = 2; }

service AlgoService {
  rpc Predict (PredictRequest) returns (PredictResponse);
  rpc WatchLog (WatchRequest) returns (stream LogEvent);
  rpc UploadMetrics (stream Metric) returns (UploadSummary);
  rpc Chat (stream ChatMessage) returns (stream ChatMessage);
}

15.3 Server Implementation (Spring Boot 3)

@SpringBootApplication
public class AlgoServerApp {
    public static void main(String[] args) {
        SpringApplication.run(AlgoServerApp.class, args);
    }
}

@GrpcService // auto‑registered by the starter
public class AlgoServiceImpl extends AlgoServiceGrpc.AlgoServiceImplBase {
    @Override
    public void predict(PredictRequest req, StreamObserver<PredictResponse> rsp) {
        if (req.getUserId() <= 0) {
            rsp.onError(Status.INVALID_ARGUMENT
                .withDescription("user_id must be positive")
                .asRuntimeException());
            return;
        }
        double score = computeScore(req);
        rsp.onNext(PredictResponse.newBuilder()
            .setScore(score)
            .addReasons("model_v2")
            .build());
        rsp.onCompleted();
    }

    @Override
    public void watchLog(WatchRequest req, StreamObserver<LogEvent> rsp) {
        try (Stream<String> lines = Files.lines(Path.of(req.getPath()))) {
            lines.forEach(l -> rsp.onNext(LogEvent.newBuilder().setLine(l).build()));
        } catch (IOException e) {
            rsp.onError(Status.INTERNAL.withCause(e).asRuntimeException());
            return;
        }
        rsp.onCompleted();
    }

    @Override
    public StreamObserver<Metric> uploadMetrics(StreamObserver<UploadSummary> rsp) {
        AtomicLong count = new AtomicLong();
        return new StreamObserver<>() {
            @Override public void onNext(Metric m) { process(m); count.incrementAndGet(); }
            @Override public void onError(Throwable t) { log.warn("client stream error", t); }
            @Override public void onCompleted() {
                rsp.onNext(UploadSummary.newBuilder().setCount(count.get()).build());
                rsp.onCompleted();
            }
        };
    }

    @Override
    public StreamObserver<ChatMessage> chat(StreamObserver<ChatMessage> rsp) {
        return new StreamObserver<>() {
            @Override public void onNext(ChatMessage msg) {
                rsp.onNext(ChatMessage.newBuilder()
                    .setUser("bot")
                    .setText("echo: " + msg.getText())
                    .build());
            }
            @Override public void onError(Throwable t) {}
            @Override public void onCompleted() { rsp.onCompleted(); }
        };
    }
}

15.4 Client Implementation (Spring Boot 3)

@SpringBootApplication
public class AlgoClientApp {
    public static void main(String[] args) {
        SpringApplication.run(AlgoClientApp.class, args);
    }
}

@Component
public class AlgoRunner implements CommandLineRunner {
    @GrpcClient("algo-service")
    private AlgoServiceGrpc.AlgoServiceBlockingStub blockingStub;
    @GrpcClient("algo-service")
    private AlgoServiceGrpc.AlgoServiceStub asyncStub;

    @Override
    public void run(String... args) {
        // 1. Unary call
        PredictResponse resp = blockingStub.predict(
            PredictRequest.newBuilder().setUserId(123L).setScene("home").build());
        System.out.println("score = " + resp.getScore());

        // 2. Server‑streaming
        Iterator<LogEvent> it = blockingStub.watchLog(
            WatchRequest.newBuilder().setPath("D:/logs/app.log").build());
        while (it.hasNext()) {
            System.out.println(it.next().getLine());
        }

        // 3. Client‑streaming
        StreamObserver<Metric> upload = asyncStub.uploadMetrics(new StreamObserver<UploadSummary>() {
            @Override public void onNext(UploadSummary s) { System.out.println("count=" + s.getCount()); }
            @Override public void onError(Throwable t) { t.printStackTrace(); }
            @Override public void onCompleted() { System.out.println("done"); }
        });
        for (int i = 0; i < 10; i++) {
            upload.onNext(Metric.newBuilder().setName("qps").setValue(i).build());
        }
        upload.onCompleted();

        // 4. Bidirectional streaming
        StreamObserver<ChatMessage> chat = asyncStub.chat(new StreamObserver<ChatMessage>() {
            @Override public void onNext(ChatMessage m) { System.out.println(m.getUser() + ":" + m.getText()); }
            @Override public void onError(Throwable t) {}
            @Override public void onCompleted() {}
        });
        chat.onNext(ChatMessage.newBuilder().setUser("alice").setText("hi").build());
        chat.onCompleted();
    }
}

15.5 Deadline (Timeout)

PredictResponse resp = blockingStub
    .withDeadlineAfter(500, TimeUnit.MILLISECONDS)
    .predict(req);

If the deadline expires, the server receives a CANCELLED status and the client sees DEADLINE_EXCEEDED.

16. Pitfall Guide

16.1 Field Numbers Must Not Change

// WRONG – reusing number 2 after deletion leads to incompatibility
message User { string name = 1; int64 phone = 2; }

// CORRECT – reserve the old number and use a new one for the new field
message User { string name = 1; reserved 2; int64 phone = 3; }

16.2 Backward‑Compatibility Rules

Adding a field with a new number – compatible.

Deleting a field but keeping it reserved – compatible.

Changing a field number – incompatible.

Changing field type – mostly incompatible (only safe widening like int32→int64).

Changing label (optional/repeated) – incompatible.

Making a field required – incompatible (proto3 has no required).

16.3 Default Values & optional

In proto3 a scalar such as int32 a = 1; defaults to 0 and the generated code cannot tell whether the caller omitted the field or explicitly sent 0. Use optional int32 a = 1; to get a hasA() method for presence detection.

16.4 Flow Control

Each side has a 32 KB window by default. High‑throughput streams can stall if the window is exhausted. Increase the per‑message limit with NettyServerBuilder.maxInboundMessageSize(...) and use onReady / isReady callbacks to apply back‑pressure.

16.5 Connection Management

Reuse a single ManagedChannel per process – creating a new channel per call incurs a full TCP/TLS handshake and is a performance disaster.

Use idleTimeout (e.g., ManagedChannelBuilder.idleTimeout(5, TimeUnit.MINUTES)) to close idle connections.

Graceful shutdown: channel.shutdown().awaitTermination(5, SECONDS).

16.6 Deadline Must Be Set

Never rely on the default unlimited wait. Set a deadline on every stub call ( withDeadlineAfter) or configure a global deadline via CallOptions.DEFAULT.withDeadlineAfter(...).

16.7 Large Message Chunking

gRPC caps a single message at 4 MB (configurable via maxInboundMessageSize). For files or large logs, split the payload into a streaming RPC instead of forcing a single large message.

16.8 Metadata Keys Lower‑case

HTTP/2 header names are lower‑case. A key like Auth-Token will be normalized to auth-token, so the client must use the lower‑case form to retrieve the value.

16.9 round_robin Requires All Instances Ready

With round_robin, traffic is distributed only among instances that are in the READY state. If any instance is not ready, the load balancer skews traffic toward the ready ones. Combine with health‑checking ( health service) to filter out unhealthy instances.

17. Scenario Discussion

17.1 Cross‑language Calls (Java ↔ Go / Python)

One .proto file is compiled by each language’s protoc plugin, guaranteeing a strong contract. Align field types: int64 maps to long in Java, int in Python, and int64 in Go. Beware of JSON bridging – integers larger than 2^53 lose precision.

17.2 Mobile → Backend

Direct gRPC from browsers requires gRPC‑Web (via a proxy) or native gRPC libraries on Android/iOS. For public APIs, REST remains the primary entry point, while internal services communicate via gRPC (often with grpc‑gateway to expose REST).

17.3 Internal High‑Performance Links

For small messages and high‑frequency calls (e.g., recommendation, risk control, search), gRPC reduces CPU and bandwidth by 30‑70 % compared with REST + JSON, thanks to binary Protobuf and HTTP/2 connection reuse.

17.4 Streaming Scenarios

Model token streams, log push, large file upload, collaborative editing – all fit naturally into gRPC’s streaming modes. Proper back‑pressure (window size, onReady) is essential to avoid overwhelming the receiver.

17.5 When gRPC Is Not Suitable

Direct browser access without a gRPC‑Web proxy.

Need for human‑readable payloads (e.g., debugging with curl).

Very simple, low‑frequency public APIs where REST’s lightweight nature is preferable.

Legacy infrastructure lacking HTTP/2 support or load balancers that cannot handle HTTP/2 trailers.

18. Summary & Best Practices

18.1 When to Choose gRPC

Use gRPC for internal services, cross‑language contracts, high‑performance or streaming requirements, and when a strong schema is desired.

Avoid gRPC for direct browser calls, when human‑readable payloads are required, or for simple low‑traffic public APIs.

18.2 Best‑Practice Checklist

Proto as an independent module – package contracts separately and publish them for all languages.

Never change field numbers – delete with reserved, add new fields with fresh numbers.

Channel singleton – reuse the same ManagedChannel within a process.

Always set Deadline – per‑call or globally, to avoid indefinite hangs.

Layered interceptors – authentication → rate‑limiting → logging → tracing, in that order.

Semantic error codes – use INVALID_ARGUMENT for client errors; employ google.rpc.Status for rich error details.

Client‑side LB – prefer round_robin over the default pick_first to avoid traffic hot‑spots.

Chunk large messages – stream payloads larger than 1 MB to stay under the default 4 MB limit.

Enable TLS / mTLS in production – plaintext only for development.

Adopt xDS / service mesh – dynamic service discovery, circuit‑breaking, and zero‑downtime configuration.

18.3 Core Mental Model

proto → contract
stub → generated client/server code
channel → transport (long‑lived HTTP/2 connection)
HTTP/2 → multiplexed lane
Protobuf → binary payload

Remembering this chain helps reason about performance, scalability, and debugging.