Stabilizing Test Environments with a Trunk‑Based Strategy

1. Background

To address the instability of the test environment, we conducted extensive interviews and data analysis, discovering that teams are gradually abandoning the daily environment and moving directly to pre‑release environments for testing. Demands for multiple pre‑release environments and isolation are increasing, raising concerns about online data safety and the risk of pre‑release environments becoming another unstable daily environment.

2. Project Solution

The project team focuses on the daily environment and presents an overall solution, emphasizing the environment plan.

2.1 Environment Solution

The core idea is that the trunk environment serves as the default dependency for testing. The trunk environment is automatically deployed based on the officially released version. The rationale for choosing the trunk environment includes:

Daily environments suffer from poor code quality, high release frequency, and user debugging, making global stability hard to achieve.

The trunk environment offers higher code quality, lower release frequency, and no user debugging, and most non‑integration test scenarios can rely directly on the trunk version.

The trunk environment has already shown good results on the business platform.

Implementation and promotion of the trunk environment are described next.

Completeness of dependent services

We identified dependencies across departments and deployed the trunk environment for all applications within a department, ensuring that dependent services are complete in the daily environment.

Stability of dependent services

The trunk environment deploys the highest‑quality trunk code.

Deployment frequency is low because updates occur only after production releases.

Debug ports are generally closed (No debug).

Real‑time monitoring provides self‑healing or alerts.

2.2 Test Scenarios

Various business test scenarios illustrate how the trunk environment is used:

Scenario 1: Single‑branch test (Project Environment → X). A new project environment is created, which by default calls the trunk environment.

Scenario 2: Backend application test (X → Project Environment). Developers can route their requests from the trunk environment to their backend application without affecting other users, using IP‑based routing.

Scenario 3: Multi‑application single‑branch integration (Project Environment → Project Environment). Two applications that need joint testing are linked within a single integration circle.

Scenario 4: Multi‑application multi‑branch merge integration test (Project Environment → Integration Environment). When applications are at different testing stages, an integration environment copy is created for isolated testing.

Scenario 5: Integration Environment → Integration Environment.

Scenario 6: Integration Environment → Trunk Environment. When the downstream daily environment is unstable, an entry‑point application can route all downstream calls through the trunk environment.

2.3 End State of Test Environments

Scenarios 5 and 6 have limitations and do not fully solve daily environment stability. A bolder idea is to isolate daily environments completely. By turning each application’s daily environment into an isolated circle that defaults to the trunk environment, the call relationships become more stable.

When integration is needed, all parties use integration environment copies, avoiding daily environment interference.

Scenario 5 (revisited): Integration Environment → Integration Environment using a private copy.

Scenario 6 (revisited): Integration Environment → Trunk Environment, similar to Scenario 1 but within an integration branch.

The final architecture is: the trunk environment provides services externally, while single‑branch project environments and multi‑branch integration project environments are used for daily testing. No environment other than the trunk is shared by everyone.

3. Implementation Plan

Application dependencies span multiple departments, so deploying the trunk environment as the default for all group applications is essential. A small‑scale pilot will be conducted first.

3.1 Identify Department Contacts/PM

Determine a PM within each department to coordinate and execute the environment governance.

3.2 Define Scope and Goals

Work with department PMs to decide which teams and applications to include, prioritizing those with many dependents. Goals are to improve test environment stability and increase usage rates.

3.3 Identify Applications for Trunk Deployment

Analyze dependency graphs to select applications that need trunk deployment.

3.4 Communicate with Stakeholders

Notify owners of affected applications; for few applications, contact owners directly; for many, coordinate via department leads.

3.5 Deploy Trunk Environments

For a small number of applications, owners perform deployment using a one‑click DevOps tool; for larger sets, DevOps handles deployment centrally.

3.6 Track Deployment Progress

Consolidate deployment status across all dependent applications.

3.7 Monitor Trunk Stability

Monitor containers and processes; attempt self‑healing first, then alert owners if recovery fails.

3.8 Promote Project Environment Usage

Encourage teams to use project environments (which depend on the trunk) before pre‑release, highlighting risks of bypassing testing such as low code quality, resource contention, instability, and data contamination.

3.9 Promote Test Environment Usage

Based on the solution, create department‑specific usage guidelines to boost overall test environment adoption.

3.10 Weekly Test Environment Metrics

Provide departments with metrics on usage rates, stability trends, issue summaries, and resolution times to enable data‑driven governance.