How Vivo Achieved Parallel Multi‑Version Environments with Full‑Link Management

Background & Problem

In software development, environment issues—instability, chaotic test setups, and severe resource contention—are critical bottlenecks that dramatically reduce development and testing efficiency. Vivo data from 2020 shows that 67% of test‑to‑release delays were caused by environment problems, and the acceptance phase experienced more than ten postponements due to the same.

Scenario 1: Urgent integration blocked when dependent services fail.

Scenario 2: Test environment occupied by another version, forcing schedule delays.

Scenario 3: Configuration differences cause missed bugs in production, leading to further issues.

These issues stem from four main factors: unstable environments, chaotic test setups, aggressive environment抢占, and low resource utilization. In a micro‑service architecture with rapid iteration, the challenges intensify on two dimensions:

Architecture layer: Service splitting multiplies the number of services, each requiring independent deployment and maintenance, lengthening call chains and increasing failure impact.

Process layer: Fast‑track business demands parallel version development (e.g., version A testing, version B feature development, version C hot‑fix), creating resource contention.

Traditional environment management can no longer meet these demands, prompting the need for an innovative method that enables safe, isolated, and efficient parallel testing and release—akin to “parallel universes”.

Solution Idea

Vivo proposes a “Full‑Link Multi‑Version Environment Management” model built on three core capabilities:

Full‑Link Ability: All components of a service version—from front‑end, gateway, micro‑services, databases, caches to message queues—must be launchable and ready with a single click, ensuring the entire chain mirrors production data and configuration.

Parallel Versions: Multiple complete environments can be created simultaneously, each running in an isolated sandbox, eliminating resource抢占 and enabling minute‑level response for hot‑fixes and new features.

Automated Environment Management: Lifecycle automation—from provisioning, elastic scaling to idle reclamation—reduces manual effort, errors, and costs while improving resource efficiency.

With these capabilities, urgent issues can be validated in a dedicated environment within minutes without affecting other versions.

Key Technology Implementation

1. Environment Orchestration

The CI/CD platform acts as a visual orchestrator. Developers define services to deploy; the system automatically discovers dependencies, determines parallel or sequential steps, and completes the entire environment setup with a single click. This provides:

Full‑Link Support: End‑to‑end integration from code commit to automated verification, ensuring consistent configurations across environments.

Parallel Version Support: Automatic branching triggers independent build‑deploy pipelines, creating isolated environments with environment tags for reuse and isolation.

2. Elastic Resources

Vivo’s container platform supplies the execution backbone. Applications and all dependencies are packaged into standard container images, guaranteeing identical behavior across development, testing, pre‑release, and production. Containers share the host OS kernel, dramatically improving utilization—multiple isolated environments that previously required ten servers can now run on three to four.

The platform also offers automatic scaling: environments automatically expand under load and shrink when idle, achieving on‑demand resource usage.

3. Traffic Isolation & Traffic Coloring

After environments are ready, precise traffic routing ensures isolation. Two concepts are introduced:

Traffic Isolation: A unified traffic‑gateway maintains a routing table that maps environment tags to service instance groups, directing requests only to the correct version.

Traffic Coloring: Each request carries an environment label (e.g., env_tag=feature1) added at the gateway, enabling the system to recognize and route traffic accordingly.

Implementation covers three traffic types:

HTTP traffic handled by Vivo’s unified access platform.

Dubbo RPC traffic routed via Dubbo service governance, with tags injected through RpcContext.setAttachment("dubbo.tag","feature1").

MQ messages routed through an mq‑proxy component that injects the environment tag into message properties, ensuring consumers only receive messages from their own environment.

Environment tags are automatically propagated across the full call chain (HTTP → Dubbo → MQ) using a Java agent in Vivo’s distributed tracing system, achieving “one‑time coloring, end‑to‑end effect”.

Business Practice & Effect

After deploying the full‑link multi‑version environment, Vivo observed significant improvements:

Environment provisioning efficiency: From manual, two‑person‑day effort to a developer‑triggered, minute‑level automated process.

Version concurrency: Previously limited to 2‑3 serial test versions, now supports nine or more parallel feature environments.

These gains accelerated development cycles and enhanced business responsiveness.

Future Planning

Vivo’s roadmap focuses on two parallel tracks: standardizing R&D efficiency environments and optimizing resource cost.

5.1 Standardized R&D Efficiency Environments

Build‑as‑a‑Service platform offering templated environments for functional, performance, and security testing.

Full lifecycle management from request, approval, creation, monitoring to reclamation.

Multi‑layer observability covering infrastructure, middleware, and application metrics.

Automated compliance checks for naming, resource configuration, security, and data protection.

5.2 Resource Cost Optimization

Automatic reclamation of inactive environments (idle sleep, reminders after 7 days, auto‑delete after 14 days).

Cost visualization and attribution analysis by project, team, environment type, and time.

These measures aim to create a virtuous cycle of higher development efficiency and lower resource expenditure.