How Ctrip's DevMaster Ensures Quality Delivery in the AI Coding Era

1. AI-Era Delivery Challenges

AI coding tools accelerate code generation, but Ctrip found that delivery quality depends on what happens before and after generation. Five core problems emerged:

1.1 Raw requirements unfit for AI: Real requirements scatter across product docs, prototypes, meeting notes, and chats — inconsistent, incomplete, and version-mismatched. Developers fill gaps through communication; AI only sees current input, so missing or stale context causes drift and rework.

1.2 One requirement spans multiple projects: Medium-large features touch frontend, multiple backend services, and QA. Splitting requires senior engineers' structural knowledge and implicit conventions — hard to reuse, error-prone at scale.

1.3 Runnable ≠ production-ready: Generated code must meet team standards, architectural design, scenario coverage, and pass compile/build/test. These must become pre-generation constraints and post-generation checks, with human sign-off on key decisions.

1.4 Siloed understanding across frontend, backend, QA: Each role interprets requirements separately; discrepancies in API fields, state transitions, and error handling surface only at integration/test. A shared spec is needed for collaboration.

1.5 From point tools to full delivery chain: DevMaster uses structured specs as the connective tissue across four stages: requirement structuring → application-level splitting → spec-driven development → test feedback.

Core artifact chain: Raw requirements → TRD (cross-app Technical Requirement Document) → ATRD (per-app Technical Requirement Document) → design, tasks, incremental Specs → code and test verification.

Three guiding principles: (1) Specs as shared basis for product, dev, QA; (2) Humans confirm boundaries, technical decisions, acceptance; (3) Process measurable — track spec coverage, code match, test coverage, fix cycles.

2. Structuring & Standardizing Natural-Language Requirements

2.1 Why Raw NL Fails for AI Development

Missing info, version conflicts, and untracked changes make AI deviate; changes don't propagate to design, code, tests. Need stable, traceable, incrementally updatable specs before development.

2.2 TRD Design Goals

Unified structure: Stable format for triggers, actors, business actions, exception branches, expected results.

Traceable sources: Every rule links to original requirement or confirmed clarification.

Manageable changes: Versioning and diffing to identify impact on specs and downstream artifacts.

Auditable conclusions: AI extracts, organizes, raises questions; product/dev/QA leads make business decisions.

2.3 Core Process (Four Steps Sharing One Source/Version)

2.3.1 Aggregate Multi-Source Context

System ingests product docs, review records, domain knowledge, historical clarifications — preserving provenance. Conflicts resolved by: current confirmed conclusion > current project rules > historical experience (reference only). Undetermined items go to clarification backlog. Verified generic knowledge and clarification patterns enter a knowledge base with scope and source tags to prevent out-of-context reuse.

2.3.2 Express Verifiable Rules via EARS

Adopts EARS (Easy Approach to Requirements Syntax) — condition, action, result. Example: "When user meets condition and submits, system shall validate and return result; if validation fails, retain state and explain reason." Keeps natural language but makes contentious conditions, boundaries, results explicit. Uniform structure aids reading, downstream generation (design, code, tests), and per-rule implementation checks.

2.3.3 Human-AI Collaborative Clarification

System generates clarification questions from missing conditions, rule conflicts, incomplete boundaries — with source, rationale, impact. Candidate answers drawn from meeting records, domain knowledge, historical conclusions; human approves before entering formal spec. Goal: replace scattered, repetitive checks with a clear question list. Humans judge business rules/exceptions; system organizes evidence, records conclusions, turns valid patterns into reusable clarification standards.

2.3.4 Detect Changes & Incremental Update

On re-import, system diffs versions, locates added/deleted/modified content, identifies affected structured rules. Unchanged confirmed content retained; changed portions trigger new clarification questions for owner review. Incremental updates avoid full regeneration and preserve version links among requirements, clarifications, TRD — downstream systems use this to decide if design/code/tests need re-check.

2.4 Results & Limits

Applied to 100+ requirements, accumulated hundreds of human-reviewed clarification standards. Structured requirements improve clarification completeness and stabilize inputs for code/test generation. Effectiveness still depends on raw material quality, knowledge-base freshness, human review quality; complex/high-risk boundaries still need owner sign-off.

3. Application-Level Requirement Splitting

3.1 Architect's View of Splitting

Structured requirements express what but not which app does what . Business boundaries ≠ application boundaries. Devs must answer: which apps involved? What changes per app? Frontend/backend/upstream-downstream dependencies? Cross-app/team collaboration contracts? These are architectural structural decisions requiring code organization, service dependencies, data flows, historical technical constraints — not inferable from PRD alone.

3.2 ATRD: What Each App Receives

After boundary splitting, DevMaster generates an ATRD (Application Technical Requirement Document) per app — not a restatement of requirements nor code comments, but a re-organization from the single-app perspective into actionable engineering input.

ATRD answers four questions:

Context — why change? App's role in overall solution, upstream/downstream relations.

Collaboration touchpoints — who to sync with? External APIs, upstream dependencies, async messages, data-sharing boundaries — shared interaction contracts.

Change scope — what exactly to modify? New capabilities, logic adjustments, deprecated paths — guides AI coding entry points and code-review focus.

Acceptance criteria — what defines done? Given-input/execution-path/expected-output format, directly convertible to test cases and API automation inputs.

Core value: each app proceeds independently on clear contracts, replacing repeated alignment meetings with parallel execution.

3.3 Full Exploration Flow

Select requirement, tech stack, relevant repos to start exploration.

Recall/build architecture map, trace data flows, locate relevant code logic.

Multiple-choice interactive clarification — maximum "laziness" for engineers.

Design review: confirm app boundaries, collaboration contracts, change scopes.

Confirm and hand off downstream.

3.4 Team Memory & Self-Evolution

Philosophy: Turn each exploration's cognition into reusable team assets.

Two-layer retention:

Project-level memory: Post-exploration, AI writes key insights back to knowledge base — module boundaries, data-flow paths, past misunderstandings. Next exploration on same project auto-injects, no cold start.

Team-level knowledge: Cross-requirement, cross-project patterns — impact-scope rules for requirement types, middleware conventions, recurring misinterpretations. Becomes collective experience read by AI every exploration.

Three feedback directions: (1) Exploration more accurate — start from known skeleton, locate change points faster; (2) Design more reliable — reference historical same-type conclusions, avoid repeat judgment errors; (3) Coding less rework — team-accumulated constraints/rules prevent violating "unwritten rules" again.

3.5 Summary

"Requirement understanding isn't prep for AI coding — it's the hardest step of AI coding." Per-app splitting and outline design are the core artifacts; they are the starting point for every AI-generated line and the prerequisite for moving from "code written" to "code correct."

4. Harness-Driven SDD Development & Quality Assurance

Specs declare what to implement but don't enforce adherence during development. To turn SDD from personal habit into engineering system, three problems must be solved: Spec generation, execution-time enforcement, post-change maintenance. Ctrip introduced Harness (constraint & detection engine) and CodingMaster App (desktop GUI for the workflow). Harness doesn't replace AI coding; it constrains and checks so specs actually participate in design, implementation, verification, archival. CodingMaster App lets developers use the capability without CLI.

4.1 Overall Architecture

Spec source: Adopted OpenSpec's open spec-management scheme — Propose/Apply/Archive incremental change model. (Reference: https://github.com/Fission-AI/OpenSpec/)

Workflow redesign: Added Init project initialization; redesigned design/task/spec templates; added task dependencies & parallel management.

Quality enhancement: Validate, Lint/Build, Verify, Coverage gates; experience retention; continuous drift detection.

System roles: DevMaster manages requirements & project assets; CodingMaster App hosts redesigned SDD flow; Harness handles constraints & detection.

OpenSpec provides spec-management foundation; Harness and engineering extensions built atop team R&D requirements.

4.2 How Harness Constrains & Detects Across Four Phases

4.2.1 Init: Establish Trusted Project Baseline

Constraints: Structured project description (goals, stack, coding/arch standards, test approach, domain, external deps). Baseline Specs in uniform format — Requirement, Scenario, SHALL/MUST — no free-form unparsable descriptions.

Detection: Scan existing code → generate baseline Specs; check format parsability; verify each Spec accurately covers current code behavior.

Output: Structured project description + machine-parsable baseline Specs — starting point for incremental changes.

4.2.2 Proposal: Produce Executable Design & Tasks

Constraints: Frontend/backend-separated design templates requiring explicit data flows, APIs, error handling, changed files, dependencies. Tasks as structured metadata: unique ID, status, Do/Don't, done criteria, modified files, pre-dependencies. Contract-type pre-changes split into pre_tasks; Spec increments strictly in Delta format.

Detection: Check task dependencies for cycles/dangling refs; check same-parallel-batch modified files for conflicts; strict validation on Proposal completion — Delta format, Scenario/Requirement name clashes, cross-section conflicts.

Output: Design doc, structured task plan, Spec increments; only tasks with satisfied deps and non-conflicting files may run in parallel.

4.2.3 Apply: Quality Gates During Coding

Constraints: Pre-coding loads historical pitfall experiences to avoid repeat errors. Execute per task dependency order and file scope. Load project Skill rules (code style, naming, exception handling).

Detection: At coding session end, Stop Hook auto-triggers: frontend runs format & static rules; backend adds compile/build per project conditions. Optional Skill-rule-based code review. Post-implementation checks: requirement scenarios implemented? Code deviates from Spec? Unrelated changes present?

Output: Code passing basic engineering checks + Spec consistency checks, with verification reports; failures return to current task for fix — cannot proceed to Archive.

4.2.4 Archive: Closure, Coverage, Experience Loop

Constraints: Pre-archive must pass implementation verification & test coverage check; only then Delta merges back to main Spec. Pre-commit re-checks archive completion — enforce archive-before-commit.

Detection: Test scenarios classified PASS/WARN/FAIL — judge concrete implementation evidence. Extract & filter valuable pitfall experiences from dev conversations. Record process metrics: code changes, scenario matches, test coverage, change traceability.

Output: Updated main Spec, verification reports, reusable pitfall experiences, process metrics; retained experiences reload next Apply cycle — cross-cycle closure.

4.2.5 Post-Archive Continuous Detection: Patrol Agent

After Archive, later iterations may modify code, causing archived Specs to drift silently — gradual, high manual audit cost/low frequency. Patrol Agent extends Harness post-archive.

Constraints: Archived Specs must not be silently modified; any drift-causing code change must be detected. Fixes only via PR; Spec-changing PRs require code review — no full auto-merge.

Detection: Periodic or on key code changes: auto-compare Spec vs code. On drift, auto-generate fix suggestions. Continuously report drift trends.

Output: Spec-Code alignment upgrades from "one-time archive acceptance" to "continuous assurance" — drifts caught and blocked early.

4.3 Why CodingMaster App?

Pain points: Early SDD workflow CLI-only — developers memorized init/propose/apply/verify/archive commands, understood directory structures and phase artifacts. Flow worked but learning curve hurt adoption.

Positioning: CodingMaster App = graphical entry for SDD + Harness, not a new code generator.

Three modes: Spec Mode (full SDD for formal requirements), Quick Dev Mode (small-scoped changes), Chat Mode (design discussion/debugging, no SDD binding).

Spec Mode usage: Step-by-step shows spec, design, tasks, code diffs, terminal results. Developers advance via "Next" — no command memorization. Key artifacts still require human review.

4.4 Verification & Repair Self-Loop: Test Issues Back to Original Dev Session

Pain points: Traditional: dev done → submit test → QA files bug → dev locates → fixes → notifies re-verify — all manual handoffs. Info fractures between stages; devs re-reconstruct original intent.

Solution: CodingMaster + test platform integration → verification-repair auto-loop: test-found issues auto-link to original dev session, auto-trigger fix, fix completion auto-notify re-verify, loop until cases pass.

Mechanism: CodingMaster retains dev session per requirement — Spec, design, tasks, historical experiences, code changes. Test failure carries requirement info; system locates original session for continued repair — avoids re-explaining requirements/design context. vs cold-start (only issue desc + current code), original session shows full design decisions and already-avoided pitfalls — fix aligns with original constraints.

Closure: Fix result re-enters test verification; if not pass, returns to original session until related cases pass. Extends Harness from code checks to test-result validation.

4.5 Results & Summary

Harness value isn't more code generation — it's spec execution, earlier problem exposure, traceable failure causes. CodingMaster App lowers adoption barrier.

Since deployment: 100+ requirements completed, 100k+ lines produced. Code contribution rate & SDD new-line count grow steadily — team reliance on and trust in AI-assisted dev rising.

Three measurable outcomes:

Quality assured: Harness shifts checks left; test coverage from <30% to ~70%.

Process measurable: Match rate, coverage, iteration rounds, contribution rate guide improvements.

Results traceable: Specs link requirements, code, test cases; Patrol Agent continuously detects drift.

5. TestPilot: Test Case Generation & Management

5.1 Test-Side Problems

Non-standard cases: Formats, hierarchies, styles vary across teams — cross-team alignment overhead, historical cases hard to reuse.

Fragmented authoring/execution: Writing, assignment, result stats scattered across tools — many stages, error-prone; progress & defect linkage not real-time.

Domain knowledge person-dependent: Historical logic & test methods orally transmitted — high new-hire ramp; raw requirement docs unstructured, cross-iteration scattered, generic search inefficient.

Case library bloat: Obsolete & conflicting expectations coexist long-term; same test point may have multiple contradictory cases — current effective rule unclear.

Test-dev disconnect: Alignment via manual comms; requirement/implementation changes often not sensed by other side — same issue re-confirmed repeatedly.

5.2 Core Design

5.2.1 Unified Case Specification (Five Dimensions)

Templates: Separate frontend/backend hierarchical templates defining required fields; supports per-business-line customization.

Terminology: Unified page/module names & relations — prevent same feature called differently across cases.

Nodes: Standard node tree constrains case hierarchy; mandatory fields & format validation ensure structural completeness & machine parsability.

Components: Common scenarios packaged as reusable components — multi-branch experiments, multi-end consistency checks, execution result markers — referenced during authoring.

Tags: Manage test type, priority, automation linkage — enables filtering & statistics.

On unified spec, platform supports template-driven manual authoring with live terminology/component references and real-time structural validation. Three assistive capabilities:

Historical case search: For new features — retrieve similar cases by requirement/page/test-point to reduce duplicate writing.

Recall case recommendation: For requirement-unchanged/tech-adjusted scenarios — recall relevant historical cases by impact scope to quickly form regression sets.

AI associative completion: Input partial case content → auto-suggest & complete remaining related cases — boosts authoring efficiency & coverage completeness.

5.2.2 Multi-Source AI Case Generation

Post-spec, system generates cases from varied inputs covering different R&D stages:

From PRD → requirement cases: All production lines; structured requirement-coverage cases.

From frontend technical requirements → frontend cases: Lines with structured requirements; structured frontend cases.

From technical design & interface contracts → backend cases: Lines with structured design; cases covering param combos, boundaries, exception scenarios.

All three inputs undergo parse/clean → requirement-point extraction → historical case alignment → case generation → coverage check. Structured backend scenarios add quality gate: auto-repair on insufficient coverage; support multi-app parallel generation then aggregate.

5.2.3 Task Management & Execution

Platform organizes cases into execution tasks with owners & cycles; supports task creation, auto-assignment, progress stats, report push.

5.2.4 Effective Case Governance

Effective Case Tree = current-version, full-functional-coverage valid case set — answers "what is current business logic", not historical aggregation. Goal: identify duplicates, conflicts, stale cases; converge to single voice; provide baseline for case generation & knowledge QA.

Governance = continuous cross-iteration diff + model analysis + human confirmation:

Consolidate & compare: Baseline = confirmed valid cases; group new cases by page/module/test-point → form old-vs-new comparison sets.

Model analysis: Per group judge relation — duplicate: keep better version; conflict: keep more reasonable version with rationale.

Write back: Tag each case valid/deprecated/unscanned; deprecated points to retained valid case.

Human review: QA confirms model conclusions; can adjust keep/deprecate/partial-keep.

Duplicate/conflict judgment based on text similarity, logical contradiction, cross-iteration duplication, temporal order — newer versions usually preferred.

5.2.5 Business Knowledge QA

Business logic & historical decisions scattered in personal experience & fragmented docs. Generic QA tools on unstructured cross-iteration content miss or hallucinate — three root causes: raw docs loose structure/unstable extraction; same feature info spans multiple iterations/docs — retrieval not concentrated; answers lack explicit provenance — unverifiable.

Governed cases have clear page-module-testpoint structure — natural knowledge base for QA with concrete citations.

Early QA: rule-orchestrated — fixed rules identify intent → single retrieval → concat results. Brittle to phrasing changes; answers unvalidated. New architecture: model-autonomous retrieval planning — model understands question → decomposes into concrete retrieval plan → calls appropriate tools (stats tool for statistical Qs, case retrieval for logic Qs) → validates results → streams "think-retrieve-conclude" so answers map to specific cases, not unverifiable summaries.

5.3 Quality Loop with Dev Side

Test case governance & generation integration with dev-side quality loop detailed in §4.4 "Verification & Repair Self-Loop".

5.4 Results & Limits

Deployed on dozens of business lines; tens of thousands of cases accumulated; AI generation >50%. Structured-design-generated cases show markedly higher accuracy & coverage vs raw-requirement generation. Model-autonomous QA improves answer quality; effective case governance in gray-scale on some lines.

System effectiveness still depends on upstream spec quality; case conflict judgment may misclassify — high-risk scenarios need human review; business QA cannot fully replace human judgment.

6. Work Practice Changes & Conclusion

6.1 Post-Adoption Workflow Shifts

Before: product/dev/QA separately understood requirements; quality checks concentrated post-dev; defects relied on manual handoff. DevMaster connects requirements, design, code, tests via shared specs — five shifts:

Requirements structured first: Product requirements clarified & human-confirmed → unified acceptance rules.

Collaboration boundaries aligned first: Full requirement split to relevant apps; frontend/backend/QA share interface contracts & acceptance standards.

Spec-driven dev & test: App-level specs generate design, tasks, test cases — reduce three-party re-interpretation.

Quality checks shifted left: Continuous checks during coding — team standards, compile results, design consistency, requirement coverage, test coverage.

Issues return to original context: Test failures link to original requirement, design, dev records; fix completes → re-verify.

6.2 Outlook

6.2.1 Cloud-Native SDD Flow

Future: code gen, run, debug, verify in cloud — reduce local env prep; developers focus on key design & result approval.

6.2.2 Automated Test Verification Integration

Future: introduce automated test tools for business functional verification, integrated into dev loop; failed cases auto-trigger fix & re-run; human intervenes only on high-risk or doubtful results.

Appendix

Glossary

(Image-based glossary omitted.)

References

Alistair Mavin, Philip Wilkinson, Adrian Harwood, Mark Novak: "Easy Approach to Requirements Syntax (EARS)", IEEE International Requirements Engineering Conference, 2009. DOI: 10.1109/RE.2009.9

Fission-AI: OpenSpec project. Official site: https://openspec.dev/; GitHub repo: https://github.com/Fission-AI/OpenSpec/.