Scaling Automated Testing & Remote Collaboration for SequoiaDB in a Pandemic

Network Infrastructure

The development environment is divided into an external network (Internet‑connected) and an isolated internal network. Engineers access the external network via office desktops or VPN‑connected laptops, then use Remote Desktop to connect to a virtual desktop server where all source code, documentation, and test cases reside. The virtual desktop can SSH into internal development and test servers for compilation, submission, and testing.

Management Cluster

The management cluster hosts the core collaboration services:

Jira for issue tracking and workflow management.

Confluence for design documents and knowledge sharing.

GitLab for source control, code review, and merge operations.

An internally built resource‑management service that automates allocation and release of compute resources.

All code changes undergo GitLab‑based review before merge.

Resource Allocation

Nearly a thousand virtual and physical servers are managed internally. An automated resource‑management framework dynamically provisions machines for test cases that may require dozens or hundreds of servers, and releases them when no longer needed.

Development and Test Clusters

Each engineer is assigned at least three VMs for compilation and unit testing. Before committing code to CI, developers must pass a rapid standard test suite of ~2,800 cases (~20 minutes) to ensure basic robustness.

The test cluster is split into three parts:

Continuous Integration (CI) : Jenkins orchestrates ~300 VMs that run ~20,000 test cases 24/7.

Performance Testing : Physical x86, OpenPower, and ARM servers run benchmarks such as sysbench, tpccrunner, and other custom workloads.

Functional & Chaos Testing : A shared cluster executes functional test suites and fault‑injection scenarios.

Testing Levels

Commit Build + Baseline Test : Developers manually run the rapid 2,800‑case suite before committing.

Daily Build + Integration Test : Nightly CI triggers a full compile and runs ~20,000 cases across 300 VMs.

7‑Day Stability Stress Test : Long‑running workloads (TPCC, sysbench, custom scenarios) run for 168 hours, monitoring performance degradation and resource usage.

Multi‑Scenario Performance Comparison : The same workloads are executed on the latest and previous releases for 6 hours; a >5 % regression is treated as a critical fault.

Fault Injection & Reliability Test : Chaos testing injects random OS/network faults (disk I/O errors, packet loss, etc.) and manual fault‑injection tests verify data integrity under failure conditions.

Remote Collaboration

Teams in Beijing, Shanghai, Guangzhou, Shenzhen, and Chengdu use two video‑conference systems (XiaoYu Yilian for presentations and Maxhub for whiteboarding). Instant messaging relies on RTX (no external network needed), and Skype is used for frontline communication.

Team Organization

Projects are broken into small virtual teams (2‑5 members) focused on specific modules (SQL engine, optimizer, storage, etc.). Each team includes developers and a test engineer who participates from the design phase, writes test cases, and reviews them before implementation. Teams follow a Planning‑Implementation‑Retrospection cycle; after a feature is merged, the team dissolves and members are reassigned.

Issue Reporting and Fix Flow

Failed test cases trigger automated emails to the responsible team. The test engineer downloads logs, reproduces the issue, and opens a Jira ticket. Developers fix the bug, run the rapid test locally, then commit via GitLab. CI rebuilds and runs the full test suite to verify the fix before the next release.

Toolchain Summary

Key tools include Jira, Confluence, GitLab, Jenkins, libfiu for fault injection, and TestLink for test case management.