Yuque’s Quality Engine: Coach Mode, CI/CD, and UI Automation Across Platforms

1. Overview

Hello, I am a QA from Yuque. I am happy to talk about Yuque’s quality system and automation technology. The following sections will give a full picture of the quality system.

1.1. Yuque Product and Development Features

Yuque is a document and knowledge‑base tool from Ant Group, launched publicly on 8 January 2018, serving hundreds of thousands of enterprises and millions of users. From a development perspective, Yuque is a large full‑stack multi‑platform application (web, mobile, desktop) built on a One CodeBase model similar to Google’s monorepo. It relies on platforms for development governance, task tracking, code services, and CI pipelines. The QA team consists of only two people, with a test‑to‑dev ratio of about 1:20, making quality assurance a major challenge.

1.2. Coach Mode

Our solution is “Coach Mode”. QA acts as a coach, using quality‑guarantee mechanisms to raise developers’ quality awareness and provide comprehensive quality services, thereby improving product quality and development efficiency. We aim to make quality outcomes sustainable and long‑term through engineering practices.

1.3. Quality Architecture

The top‑level quality architecture consists of foundational platforms (development governance, task tracking, code services, CI pipelines), quality infrastructure (technical foundations like SkyTest, Lark‑Reliable, Macaca suite, and “invisible” quality culture), and quality capabilities (continuous integration, defect convergence, online inspection, UI automation, long‑term mechanisms, and development efficiency).

2. Quality Support Tools

Each quality team usually has support tools. Yuque uses SkyTest and Reliable.

2.1. SkyTest

SkyTest is an npm‑based CLI tool that provides a lightweight, easy‑to‑integrate testing toolbox for QA, reducing maintenance cost compared with web services.

2.2. Reliable

Lark‑Reliable is a quality data persistence and insight service that works with SkyTest to cover most testing needs. It is built on the open‑source Macaca Reliable suite. Test and coverage report viewing Coverage insight

3. Establishing Mechanisms

We create mechanisms to form a long‑term quality loop, automating QA workflows so quality engineers can focus on sustainable engineering.

3.1. Weekly Iteration Duty

Two QA engineers rotate weekly to ensure release quality, handle UI regression checkpoints, release execution, and hot‑fix retrospectives. Release frequency was adjusted from twice a week to once a week to balance agility and quality.

3.2. Defect Convergence

Offline Defect Reporting

Defects lingering in the backlog are automatically reported to development groups via DingTalk, establishing daily/weekly reports that drive convergence and keep defect count low.

Online Log Defect Governance

Online logs are continuously collected and analyzed; defects are automatically created in the defect management system and assigned to owners based on platform (web, mobile, desktop).

3.3. Coverage Governance

Incremental Coverage PR Gate

We add incremental coverage checks in PRs using macaca-istanbul to generate coverage reports, encouraging developers to add unit tests.

Full‑Project Coverage Tracking

Weekly CI coverage data is aggregated and broadcast via DingTalk, restoring long‑term coverage tracking.

3.4. Online Unavailability Governance

We monitor browser compatibility and white‑screen issues, ensuring no ambiguous browser versions, guaranteeing supported versions, and enabling automated regression on mainstream browsers.

3.5. CI Issue Tracking

Automated detection of CI failures (logs, defects, coverage, errors) and automatic assignment via a project defect platform, with DingTalk notifications.

4. Continuous Delivery

CI pipelines trigger on PR merges, performing automated deployment and testing for web, desktop, and mobile platforms, providing detailed notifications and reports to developers.

4.4. CI Efficiency Improvements

We reduced CI time by using Docker layer caching, a self‑developed job_keeper for automatic retries, shell script error handling, and task splitting, achieving full regression within 15 minutes.

5. UI Automation

We follow the testing pyramid, with QA handling UI automation. Our UI automation has discovered over 70 critical issues, preventing many hot‑fixes. Automation replaces repetitive manual work, saving time. Boosts regression efficiency for agile development. Enables 24/7 core functionality stability testing and inspection.

5.1. Macaca Suite

Macaca provides a cross‑platform automation solution based on the W3C WebDriver standard, offering a standardized driver layer for browsers and devices.

5.2. Multi‑Browser Support

Our UI automation covers Chrome, Firefox, Safari, and Edge, completing full UI regression in about 30 minutes with over 2 k test cases and a 51 % automation rate, using macaca-playwright as the driver.

5.3. Desktop UI Automation

We built macaca-macos driver and the AppTester framework to automate desktop applications.

5.4. Mobile UI Automation

Mobile UI automation runs on Ant Cloud Test platform with real devices, using our macaca app inspector, Macaca Reporter, and Reliable for persistence.

5.5. UI White‑Screen Inspection

Pixel‑level comparison of online pages triggers alerts within 10 minutes for web and mobile, providing early detection of unavailable pages.

5.6. Recorder

We developed macaca-recorder , a plugin‑based UI recorder that generates test scripts with customizable templates, reducing maintenance burden.

6. Summary

Through the “Coach Mode” and the described solutions, Yuque has advanced to Quality 3.0, aligning product quality with engineering quality.