How UCloud’s Physical Network Orchestrator Cuts IDC Build Time from Days to Hours

Background and Motivation

To meet the demands of massive data‑center network configuration automation, operational efficiency, and real‑time hybrid‑cloud connectivity, UCloud developed a Physical Network Orchestrator. The system serves as the foundation for network‑operations automation, offering a simple tool for configuring switches with high accuracy, stability, and security.

Using the orchestrator, IDC network construction for tens of thousands of devices has been reduced from 2‑3 days to 2‑3 hours, with success rates rising from 80% to 99%. It now manages over 3000 switches, more than 100 carrier lines, supports 200 Gbps access throughput and ~2 Tbps DCI throughput, and enables real‑time hybrid‑cloud network bridging (excluding physical cabling).

Challenges

Traditional manual deployment becomes inefficient and error‑prone at scale. Differences among vendor command syntax, version‑specific command sets, and varying configuration models make automation difficult. These issues must be abstracted into concrete scenarios and categorized to enable reliable automation.

Architecture and Business Model

The system treats each business need as a specific scenario (e.g., deploying a static route to connect to a public cloud). Scenarios are vendor‑agnostic, requiring abstraction of both command syntax differences and configuration model differences. The resulting model includes atomic command classes, templates, and API keys.

Implementation Steps

Step 1: Build atomic command classes based on vendor, device model, version, and patch.

Step 2: Record atomic commands for each function.

Step 3: Create templates by dragging atomic commands to satisfy business scenarios.

Step 4: Generate API keys for each template to allow programmatic invocation.

When creating an API, device hardware/software versions are grouped, and command groups are linked to functional requirements.

Design and Optimization Practices

1. Kafka‑Based Command Execution Queue

The orchestrator uses Kafka topics to separate spatial (IDC) and temporal (execution order) dimensions, ensuring ordered and reliable command delivery. Each IDC’s commands are produced to a dedicated topic and consumed by a clustered consumer group.

2. High‑Availability Cluster Design

Zookeeper cluster

Kafka cluster

Kafka consumer clusters per IDC

MySQL master‑slave (1 master, 2 slaves)

Web server active‑active with Nginx load balancing

This design eliminates single points of failure; the system has run without downtime since launch.

3. Permission Management

API‑level permissions for CRUD and scenario execution

Menu‑level UI permissions

Backend token authentication linking each request to a user

4. Real‑Time Command Feedback

The front‑end displays execution status (success, failure, rollback, login failure, etc.) with color‑coded borders and captures switch echo output for operator reference.

5. Atomic Command Library

Supported switch brands (HW, H3C, RUIJIE) are cataloged by vendor, model, version, and patch. Devices with identical command sets are grouped, and atomic commands are stored with functional and parameter templates, dramatically reducing manual entry effort.

6. API Exposure and Second‑Stage Development

The system offers fine‑grained, atomic‑level APIs for switch operations, enabling both internal automation and external system integration (e.g., Pangu server provisioning, UXR projects).

Conclusion

After extensive testing, the orchestrator has been applied to multiple business scenarios, cutting IDC network build time from days to hours and raising deployment success from 80% to 99%. It also provides real‑time hybrid‑cloud network bridging via the user console.