How QQ’s Hodor System Stops Performance Degradation Before It Happens

QQ’s performance‑degradation prevention system, named Hodor (Hold the Door), was built from scratch starting in October 2021 and has reached industry‑leading maturity. The goal is to catch performance regressions early—before code merges or releases—by treating stability as a core quality gate.

Why Prevent Degradation?

Large code bases (IM, short video, etc.) with rapid bi‑weekly iterations generate many performance issues that are hard to trace after release. Traditional post‑release fixes are insufficient, so QQ introduced a set of "door‑keeping" mechanisms:

Automatic checks before branch merges for stable metrics.

Proactive issue tickets generated during development.

Comprehensive performance dashboards for "god‑view" monitoring.

Alert bots that trigger on defined performance thresholds.

System Implementation

1. Design Overview

The system records performance data per commit, compares it against a baseline, and provides automated reports, alerts, and ticket creation. Core capabilities include performance reporting, data analysis, intelligent scheduling, issue ticketing, device management, and test‑case management.

2. Data Collection

Dynamic performance data is captured using Apple’s xctrace (the CLI for Instruments) to record detailed trace files without invasive instrumentation. The collected traces are exported to XML for further analysis. Static scanning extracts symbols from compiled Mach‑O binaries to detect duplicate Objective‑C categories, +load methods, and native C symbols that can affect startup time.

Signpost APIs are used for business‑level instrumentation, allowing performance data to be correlated with specific app scenarios.

3. Task Scheduling

Git events trigger the creation of performance‑test tasks. Tasks are prioritized by type (crash, main‑flow, custom, idle‑time) and dispatched to appropriate device pools. The scheduler ensures urgent tasks run first and monitors device health, issuing alerts on failures.

4. Data Processing

Because raw Instruments data can be gigabytes in size, the client performs on‑device symbolization and extracts key metrics (CPU, memory, I/O, thread count). Processed data is uploaded to the server, where it is aggregated, stored, and used to generate dashboards, performance reports, and automated tickets.

Different metric types are handled separately: baseline performance data yields peak, average, and end values; signpost‑tagged data adds duration calculations; custom business data is ingested via dedicated APIs.

5. Management UI

The Hodor portal provides:

Performance dashboards filtered by time, branch, test case, and scenario.

Branch performance reports that compare each push against the mainline.

Automated ticket generation based on white‑list and rule filtering.

Alert notifications for metric violations.

Test‑case management for custom scenarios.

Visualizations include commit‑level status, metric trends, and detailed stack traces for regressions.

Key Insights and Outcomes

Hodor has dramatically improved developer efficiency by shifting problem detection left in the pipeline. Notable achievements include:

Automatic detection of a cold‑start regression caused by a dyld perfect‑hash collision, reducing startup time by 700 ms on iPhone 11 after fixing colliding symbols.

Continuous integration of performance tests for every commit, with real‑time alerts for crashes and high‑frequency logs.

Scalable architecture supporting QQ’s multi‑platform clients (mobile and desktop) and handling billions of performance samples.

The system continues to evolve in 2024, expanding to desktop clients and integrating additional efficiency metrics.

Overall, the Hodor system demonstrates how a large‑scale mobile client can embed performance guardrails into its CI/CD pipeline, providing continuous visibility, rapid regression detection, and automated remediation workflows.