AI-Driven Large-Scale Refactoring: Qunar's 70% Efficiency Boost with Harness & Loop

Introduction: AI Coding Beyond Small Tasks

AI Coding has entered daily R&D workflows for small, well-scoped tasks. This article explores whether AI can upgrade from "code assistant" to "engineering execution system" for large-scale refactoring. Qunar tackled a 150K-line high-concurrency core system reconstruction with strong dependencies, complex call chains, and strict performance/stability requirements.

Results Achieved

Delivery efficiency: Total effort reduced from ~100 person-days to ~30 person-days (~70% improvement)

Performance gains: Search main-path P50/P90 latency both dropped ~70% (P50: 191ms → 59ms; P90: 644ms → 195ms)

Resource savings: Machine resource cost reduced ~45%

Release outcome: Zero QA involvement, smooth production launch with zero incidents

Why Large-Scale Refactoring Is Hard: System-Level Uncertainty

The gap between small tasks and large refactoring is not just code volume but the shift from "local feature delivery" to "system behavior control." Four uncertainties determine success:

Behavioral uncertainty: Request/response/exception semantics consistency across edge cases, historical configs, special data paths

Impact uncertainty: Whether changes leak to main flows, branches, caches, configs, or external dependencies

Verification uncertainty: Manual tests and limited regression suites cannot cover real-traffic combinations

Risk uncertainty: Ability to detect, locate, contain, and rollback quickly when differences appear

Traditional refactoring suffers from high manual effort, incomplete system knowledge (reliant on few experts), insufficient test coverage, and poor artifact retention — leading to "code changed but no one can prove it's safe."

Prerequisite: Turn Uncertainties into Engineering Constraints

Before involving AI, four engineering conditions must be established: goals decomposable, boundaries constrained, verification automated, failures rollback-able. Only then can AI operate in a controlled execution environment.

Methodology: Harness & Loop

Harness = constraint system defining goals, boundaries, context, rules, available tools, stop conditions. Loop = feedback system: plan → modify → verify → analyze → record → judge next step (continue, retry, escalate). Together they prevent AI from filling gaps with hidden assumptions and from runaway changes.

Four-Phase Closed Loop

Planning: Define refactoring goals, scan system call chains, establish metric baselines — answer "why refactor, what defines success."

Design: Technical solution, milestone breakdown, Phase/Task planning, dependency mapping — answer "how to refactor, in what order."

Execution: Run AI Coding per Task; every change verified, recorded, judged; failures enter analysis/fix loop.

Acceptance: Aggregate test results, diff conclusions, metric effects, release evidence — judge readiness for next phase or production cutover.

Human-AI Division of Labor

Human: Goal definition, boundary setting, architectural trade-offs, risk acceptance, release decisions. Cannot delegate: direction judgment, solution choices, go/no-go.

AI: Code scanning, solution assistance, code modification, test execution, diff analysis, report generation. Cannot replace: business goal confirmation, risk boundary decisions.

Task Decomposition: Milestone → Phase → Task

Milestone: Stage-level business goals (e.g., behavior consistency first, then performance, then canary cutover).

Phase: Group of tasks sharing a common acceptance goal, enabling stage-gate Go/No-Go.

Task: Minimum unit AI can stably execute — near small-task granularity, independently verifiable, independently rollback-able. Must specify: inputs, outputs, modification scope, non-modification scope, acceptance criteria, risks, rollback plan.

Task Files: Constrain Inputs, Retain Evidence

Each Task file defines: what to do (goal, context, deliverables, success state), where to change (impact scope, modules, file list, dependencies, expected outputs), what NOT to change (protected call chains, interface semantics, configs, external side-effects, legacy compatibility), how to verify (compile, unit tests, automation tests, diff, regression, performance metrics), failure handling (rollback plan, failure strategy, human gates, stop conditions). Post-execution artifacts: execution record (files changed, reasons, diff summary, non-changes), verification results (compile, unit, automation, diff analysis, performance data), risk judgments (residual risks, failure causes, need for human confirmation), artifact links (design docs, code changes, test reports, deployment records).

Project Overview File & Automated Execution

Acts as navigation, status dashboard, context-recovery entry, and automation entry point. Contains: project definition (background, goals, scope, constraints, acceptance criteria), artifact links (design, architecture, task breakdown, task graph, verification docs, deployment records), execution status (milestone/phase/task progress, pass/fail, blockers, next actions), context recovery (enables AI to resume in new sessions without re-learning entire system). Automation picks next executable Task from overview + task graph, runs Harness+Loop, writes back records/results/links.

Release-Grade Quality Control

Task-level verification ≠ production readiness. Release validation requires:

Shadow dual-run: New/old paths run simultaneously; diff core results without affecting live traffic.

Canary cutover: Gradually increase new-path real-traffic share; avoid big-bang switch.

Monitored ramp: Continuously observe success rate, latency, error rate, resource usage, core business metrics; decide continue/stop.

Automated fallback: Auto-revert to old path when metrics exceed thresholds; preserve forensic evidence.

Case Study: 150K-Line Core System Reconstruction

Systems Involved

Search Engine System: ~105K lines, 10K QPS, I/O + CPU intensive

Packaging System: ~49K lines, 1.6K QPS, CPU intensive

Booking System: High stability requirement, 80 QPS

Pain Points

Multiple downstream data sources maintained separately → inconsistent data calibers, business consistency hard to guarantee

Core business rapid iteration accelerated system decay → high subsequent iteration cost, testing/maintenance difficult

Large payloads + high QPS → heavy I/O pressure, frequent single-node issues

Refactoring Goals

Merge search core chain systems

Unify third-party data sources, resolve cross-system business caliber inconsistencies

Systematic performance optimization: search P50 190ms → 120ms, save 30% server resources

Milestone Breakdown: Consistency First, Then Optimization, Then Cutover

Order driven by risk convergence, not ease of modification. Principle: guarantee behavioral consistency first, then optimize structure/performance, finally canary cutover. Benefits are not first priority; system trustworthiness is prerequisite.

M1: Scope & Guardrails

M1 goal: enable election system with local packaging capability, prove old/new path behavioral alignment via consistency verification. Guardrails:

Modification scope fence: Only touch relevant chains and implementation integration; leave other search, cache, election code untouched.

Behavioral semantics fence: Request/response/exception semantics unchanged; core chain diffs must converge.

Runtime safety fence: Shadow dual-run, canary cutover, auto-fallback; no big-bang replacement.

Side-effect fence: Shadow path must not write MQ, Redis, or external systems; new verification must not impact main path stability.

Execution Plan: 9 Phases, 34 Tasks

Tasks typed as MANUAL, CODE, CONFIG, VERIFY. CODE tasks are primary AI Coding targets; MANUAL and critical VERIFY require human judgment. Execution follows dependency order, not random selection. Project overview file tracks progress, dependencies, status; Task execution records capture changes, test results, deployment info, risk judgments.

Diff Closed Loop: Real-Traffic Differences Drive Next Fixes

Diff is not a conclusion but input for next repair cycle. Shadow dual-run puts old/new under real traffic; diffs enter loop: fetch diff logs (enrich with traceId, time, env) → cluster analysis (group similar diffs, identify high-frequency anomalies) → root-cause localization (combine module, config, code paths) → generate fix proposals (AI-assisted change suggestions + additional verification ideas) → human confirmation (adopt? how to canary? rollback?) → execute fix via AI Coding → re-verify. AI assists clustering, localization, suggestion; human owns risk decisions.

Verified Outcomes

Efficiency: 100PD → 30PD (~70% gain)

Performance: Core chain P50 191ms → 59ms, P90 644ms → 195ms (~70% reduction)

Resources: ~45% cost reduction

Release: No QA involvement, zero-incident launch

These results stand on prior behavioral consistency verification and release-grade quality control. Efficiency, performance, resource gains only have engineering value when system behavior is trustworthy and production risk controlled.

Summary: AI Coding Value Beyond Code Generation

AI's real contribution is embedding a high-risk, long-cycle, verification-heavy system engineering project into a continuously executable engineering closed loop.

1. AI Handles High-Frequency Execution; Human Owns Key Judgments

AI suits repetitive, context-clear work: code migration, interface adaptation, test supplementation, log analysis, documentation retention. Human retains goal judgment, solution trade-offs, risk identification, final confirmation. Clearer constraints → more stable AI output; richer verification → easier trust in automation.

2. Decompose Complex Problems into Verifiable Tasks

Task is the key unit for stable AI Coding operation. A good Task states not just "what to change" but why, where, how to verify, risks, and what evidence remains. With clear Tasks, AI execution relies on structured inputs, automated verification, continuous retention — not on single-session improvisation.

3. Ultimate Asset Is Team Capability

Post-refactoring, retain: task decomposition patterns, project overview files, verification scripts, diff analysis flows, release checklists, risk handling experience. These assets let the team reuse a mature working method for future large-scale changes: turn uncertainties into engineering constraints, hand constraints to AI + automation for repeated execution, verification, correction. AI Coding in large-system refactoring does not replace engineering governance — it makes governance finer-grained, more continuous, and more executable.