Resolving gRPC‑gateway Limits and mTLS Certificate Issues in etcd 3.x for Apache APISIX

When etcd upgraded to the 3.x series, its external API changed from plain HTTP/1 to gRPC, and to support clients that cannot use gRPC it provides a gRPC‑gateway that proxies HTTP/1 requests to the new gRPC API.

Apache APISIX originally used etcd v2’s API; starting with APISIX 2.0 the dependency was upgraded to etcd 3.x. Because the Lua ecosystem lacks a native gRPC library, the HTTP compatibility offered by etcd’s gRPC‑gateway was essential.

After eight months of production use, several subtle differences between the HTTP API and the gRPC API were observed, revealing that the gRPC‑gateway does not perfectly emulate gRPC behavior.

Breaking the default gRPC limit

etcd’s gRPC server enforces a default maximum receive message size (MaxCallRecvMsgSize) of 4 MiB. During a full sync of many configurations, Apache APISIX can exceed this limit and receive the error “grpc: received message larger than max”. The etcdctl client can avoid this error because it sets a much larger limit when establishing the connection.

Two mitigation ideas were considered:

Simulate a full sync with incremental sync, which would require substantial code changes and increase synchronization time.

Modify etcd to lift the limit for the gRPC‑gateway as well. The second approach was chosen and a pull request was submitted ( https://github.com/etcd-io/etcd/pull/13077 ), which was merged in etcd v3.5.0. This change also back‑ported to the 3.4 branch, so upgrading to v3.5.0 (or the next 3.4 release) resolves the “grpc: received message larger than max” issue. Interesting use of server certificates After enabling mTLS support in etcd, some users reported that HTTP requests through the gRPC‑gateway failed certificate verification, while direct etcdctl access succeeded. The etcd logs showed errors such as: 2021-06-09 11:10:13.022735 I | embed: rejected connection from "127.0.0.1:50898" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "") WARNING: 2021/06/09 11:10:13 grpc: addrConn.createTransport failed to connect to {127.0.0.1:12379 0 }. Err :connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting... The root cause was that etcd, once client‑certificate verification was enabled, required the gRPC‑gateway to present a client certificate. etcd mistakenly reused the server certificate as the client certificate, which works only if the certificate includes both server and client authentication usages. Inspecting the problematic certificate with: openssl x509 -text -noout -in /tmp/bad.crt revealed extensions such as: X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication Adding “TLS Web Client Authentication” to the Extended Key Usage (or removing the server‑only extension) resolves the issue. The related etcd issue is tracked at https://github.com/etcd-io/etcd/issues/9785 . Conclusion Although the gRPC‑gateway has some limitations, etcd’s support for HTTP access remains a valuable feature, and the community contributions that fixed the message size limit and certificate usage improve the reliability of Apache APISIX’s integration with etcd.