Cross-Network Data Transmission System Based on Dubbo: Architecture Evolution and Technical Challenges

This article introduces the technical challenges, community collaboration, and deeper architectural thinking involved in implementing a cross-network data transmission system based on Dubbo at Zhengcaiyun, a government procurement cloud platform. For cloud-based businesses, the cloud platform is an internal, fully controllable local area network, while the island end is an independent internal network with its own security policies. When cloud-island communication is needed, ports are opened according to customer requirements, which involves certain costs and is not completely controllable. As cross-network demands increase, this becomes a pain point. Building a transparent cross-network transmission network with good top-level design can help balance business support, security control, and operational costs.

The article describes the evolution of Zhengcaiyun's cross-network data transmission system, from early single-direction transmission schemes to the current "highway" architecture. The highway architecture uses tunnel mechanisms where Dubbo private protocol is used at both ends, while HTTP protocol is used for cross-network transmission. This design provides low intrusion to business applications.

Key technical challenges addressed include client routing (supporting multiple routing methods like annotations, configuration center, and thread-based routing), Dubbo request address switching (using Dubbo's Route extension to dynamically switch addresses), exit gateway implementation (using Dubbo-Proxy to receive and forward Dubbo traffic), and center gateway (using Apisix as a unified gateway for all cross-network traffic).

The article also discusses future directions including stability improvements (resource pooling, QoS guarantees, anomaly detection), functional enhancements (protocol support for HTTP/GRPC, security enhancements, usability improvements), and new technology exploration (Go-based implementations, Higress protocol).

The authors, Wang Xiaobin and Xu Xiping, are senior engineers at Zhengcaiyun responsible for basic service work.