Virtual Environment for Isolated Branch Testing in Service Mesh

The infrastructure team at Yuanfudao builds engineering platforms such as a multi‑cloud private cloud, monitoring, service governance, message queue middleware, and a front‑end engineering platform to support core business stability and development efficiency.

Key team members include Gao Wei (service mesh), Jiang Yangyadi (service framework, virtual environment, core libraries), and Li Yabin (message queue).

TL;DR: The virtual environment is an isolation solution that uses traffic coloring and routing tags to allow independent testing of different branches, supporting HTTP, RPC, and MQ protocols, thereby greatly improving testing efficiency and reducing cost.

Background and Purpose: Traditional testing suffers from a single shared environment, causing scheduling conflicts and integration issues when multiple branches or services need simultaneous testing. A virtual environment (also known as a lane or isolated environment) was developed to allow independent testing of multiple services or branches without interference.

R&D Process Change: The previous flow (development → review → merge → test environment → production) required coordination due to a single stable test environment. The new flow enables each branch to be deployed and tested in its own virtual environment, with traffic isolated, before merging into the mainline.

Advantages of Virtual Environments: Aligns with Gerrit workflow, allowing deployment per change without merging or branching. Improves mainline stability by ensuring all changes pass self‑test, integration test, and functional test before merge. Boosts development efficiency by eliminating wait times for test environments and preventing deployment interference. Provides finer‑grained change control and continuous deployment confidence. Automatically deploys on each push for the same change‑id.

Implementation Principles: A virtual environment is a weakly isolated layer derived from a baseline environment, sharing stateful resources (e.g., databases) while isolating traffic via coloring tags. Concepts include baseline environment, virtual environment, stress‑test environment, traffic, traffic coloring, and instance coloring.

Isolation Types: Strong isolation creates completely separate resource sets for each environment, which is costly. Weak isolation isolates only traffic while sharing underlying resources, achieving logical isolation with lower overhead.

Traffic Coloring and Routing: Traffic (HTTP, RPC, MQ) is colored with tags; routing rules direct colored traffic to matching colored instances, falling back to baseline instances when no match exists. Specific rules handle normal, stress‑test, and untagged traffic.

HTTP Traffic: Coloring is performed via domain names containing the tag (e.g., mytest--service.rz-venv.domain.com ). The tag is injected into HTTP headers and propagated via ThreadLocal.

RPC Traffic: Implemented with Thrift; the color tag is added to Thrift headers, and service discovery uses the tag to select appropriate instances.

MQ Traffic: Uses Alibaba Cloud MQ where consumer groups are suffixed with the virtual environment tag. Both baseline and virtual environments consume the full topic stream, applying filters to ensure only appropriately colored messages are processed.

Deployment: Automated deployment configures ingress and creates resources transparently for users. Virtual environments have a three‑day lifespan and are destroyed automatically after expiration.

Open Issues: Java native thread pools and asynchronous calls lose coloring information; a custom thread pool is planned to address this.

Recruitment: The team seeks engineers to explore advanced topics such as multi‑cloud Kubernetes scheduling, edge computing, high‑concurrency gateways, unified observability, service mesh, hybrid messaging, hybrid client frameworks, and DevOps integration.