Why CKafka Cross‑Region Sync Stalled at 64KB/s: TCP Window Scaling & Kernel Tuning

Background

To meet customers' cross‑region disaster‑recovery and cold‑standby requirements, Tencent Cloud CKafka provides a connector‑based cross‑region data‑sync capability that aims for near‑real‑time, second‑level synchronization. The architecture relies on a Kafka Connect cluster and VPCGW PrivateLink to bridge cloud environments.

Problem Symptoms

In a customer scenario, data synced from a Hong Kong CKafka instance to an East‑US instance exhibited extremely high latency. The Connect consumer on the source side accumulated a large backlog, and the observed consumption rate was only about 325 KB/s, far below expectations.

Initial Analysis – Common Causes of Message Backlog

Broker cluster overload : high CPU, memory, or disk I/O reduces consumption throughput.

Insufficient consumer processing capacity : not enough consumers or inefficient consumer logic.

Consumer crashes : abnormal exits leave messages unconsumed.

Offset commit failures : lead to duplicate consumption or loss.

Network or broker failures : disrupt transmission and storage.

Producer sending too fast : exceeds consumer capacity.

All cluster‑level metrics appeared healthy, and the Connect consumer showed no obvious bottleneck.

First Investigation – Network Throughput

We measured raw network bandwidth between the Hong Kong and East‑US regions using iperf3 and wget. The iperf3 test reported 225 Mbps, and wget achieved about 20 MB/s, indicating that the underlying network link was not the limiting factor.

Second Investigation – Kernel Parameter Tuning

We suspected kernel‑level network settings might be sub‑optimal. The default values were increased and the TCP congestion control algorithm was switched to BBR, which is known to improve long‑haul throughput and reduce latency.

sysctl -w net.core.rmem_max=51200000
sysctl -w net.core.wmem_max=51200000
sysctl -w net.core.rmem_default=2097152
sysctl -w net.core.wmem_default=2097152
sysctl -w net.ipv4.tcp_rmem="40960 873800 671088640"
sysctl -w net.ipv4.tcp_wmem="40960 655360 671088640"
sysctl -w net.ipv4.tcp_congestion_control=bbr

After applying these changes, latency improvement was minimal.

Third Investigation – Application‑Level Socket Buffers

We then adjusted Kafka’s socket buffers. The broker’s Socket.Send.Buffer.Bytes was set to use the system’s send buffer, and the Connect consumer’s Receive.Buffer.Bytes was also increased to the system default. Additionally, Max.Partition.Fetch.Bytes was raised to 5 MB.

These changes raised the average consumption speed from ~300 KB/s to over 2 MB/s.

Deeper Dive – TCP Window Scaling Failure

Despite the buffer tweaks, some partitions still exhibited low throughput. Packet captures revealed that for the slow connections the TCP send window was stuck at 64 KB, the historic maximum without window scaling. Normal connections showed a larger window, confirming that the Window Scale option was not being negotiated.

Further analysis showed that during massive concurrent Connect consumer startups, the server’s SYN‑cookie protection was triggered. Because the client did not include the TCP Timestamp option, the server cleared the Window Scale option, leaving the connection limited to a 64 KB window.

Root Cause

The root cause was a combination of high‑frequency TCP connection bursts (triggering SYN‑cookie checks) and the VPCGW stripping TCP Timestamp fields in NAT mode, which prevented the Window Scale option from being negotiated. Consequently, the effective TCP send window remained at 64 KB, severely throttling cross‑region data transfer.

Solution

Mitigation : Reduce the concurrency of Connect worker initialization to avoid triggering SYN‑cookie protection.

Final fix : Work with the VPCGW team to enable forwarding of TCP Timestamp fields, allowing Window Scale negotiation.

Conclusion

The apparent cross‑region sync slowness was ultimately traced to a low‑level network issue—missing TCP Timestamp causing Window Scale to be disabled under heavy connection bursts. Addressing the VPCGW behavior and adjusting Connect startup concurrency restored expected throughput.