From Personal to Team AI Coding: HuoLaLa's Process-Driven Implementation at Scale

HuoLaLa's engineering team of over 1,000 engineers across 10+ tech stacks and dozens of business lines found that simply distributing AI coding tools did not translate into measurable team-level efficiency gains. Five problems emerged: inconsistent individual usage, duplicated project context preparation, divergent code styles, inability to quantify AI's impact, and uncontrolled cost and risk. The core realization: personal efficiency gains do not automatically aggregate into organizational efficiency.

Three-Phase Evolution

Phase 1 — Popularization: Raising the 80% Floor

The team observed a wide gap between the top 20% and the remaining 80% of engineers using the same tools. They built AiBox , a unified workbench that centrally manages four configuration types:

Rules — team coding constraints

Skills — reusable multi-step workflows

Commands — shortcuts for high-frequency actions

MCP — external service and data source integrations

Configurations are maintained centrally and pushed to every IDE. The platform also handles tool admission, usage statistics, compliance checks, and auditing. Training covered AI coding fundamentals (where AI is reliable vs. where it hallucinates), IDE capability boundaries (completion, chat, agent modes and their trade-offs), unified environment setup (Rules, project initialization, MCP), and a structured onboarding flow. Practical scenario guides with runnable examples were created for five high-frequency scenarios: code comprehension (order module analysis), field and query interface addition, code change review, and unit test generation without containers or real databases.

Phase 2 — Spec-Driven Development (SDD): Making AI Follow Specs

Popularization solved "can use" but not "stable use." Two drivers: average engineers struggle to write good designs, so structure and template the spec for AI generation; at scale, the scarcest resource shifts from code volume to structured domain knowledge. The team adopted the industry-common five-step flow (clarify, design, task breakdown, implement, review) — referencing GitHub Spec Kit, AWS Kiro, BMAD-METHOD — and integrated OpenSpec's spec/delta structure (suitable for large legacy codebases) and Superpowers' Skill-based methodology (scalable to thousands). Four custom integrations were added: requirement system (PMIS) connection, automatic spec retrieval, post-use scoring write-back, and full-chain trace.

Step 1 — Context Assembly (critical): Auto-pull requirement from PMIS with PRD; AI retrieves candidate specs from tech/domain/dev/cr_spec libraries; package specs, code snippets, historical designs into a context checklist; engineer removes irrelevant items and adds missing ones to produce proposal.md. Changes are split into N atomic deltas, each validated before and after, so failures roll back only the offending step.

Spec lifecycle: Every retrieval records hit/miss and misleading rate; low hit-rate or high-misleading specs enter backlog for revision; long-unused specs are retired. Each step's artifact serves both human review and downstream AI consumption.

Phase 3 — AI Native: Full Lifecycle Closed Loop

Post-delivery experience and production issues must feed back into subsequent requirements. Closure hinges on two post-release loops: (1) development-phase insights (spec updates, context improvements) and (2) production feedback (incidents, performance regressions). The next requirement then arrives with historical assets pre-loaded.

Organizational and process shifts envisioned (some in progress, some directional):

Team boundaries: frontend/backend/client → business-domain aligned

Domain personnel: single-stack individuals → multi-stack + AI collaborators

Engineer role: implementer → goal setter + result auditor

Hiring bar: coding fluency → context construction ability

Documentation: ad-hoc → fully spec-structured

Workflow: experience-based → spec end-to-end

Review: human-only → agent first pass + human second pass

Metrics: lines/hours → FPY / adoption rate / spec coverage

Requirements, designs, tasks, and review conclusions must use unified structures with explicit boundary conditions and acceptance criteria. Tribal knowledge previously relying on team rapport must be documented for AI consumption.

Measuring AI Effectiveness: Three Metrics

1. AI Code Ratio

AI Code Ratio = (AI-generated lines finally merged to main) / (total new lines in main)

. The phrase "finally merged" is key — IDE-reported AI lines differ vastly from what survives. Three-layer collection ensures auditability:

IDE telemetry: hash signature per AI-generated line

Commit diff layer: at git commit, verify each line still matches its signature

Merge statistics layer: count only lines that reach main

2. Full-Chain Trace

End-to-end trace of a single requirement (example: batch order export, 1.5 hours, 20 spans). Shows proposal generated 3 times, adopted once; tech design generated twice, adopted once; tasks split into 7; review ran two rounds before passing. The gap between generations and adoptions signals improvement headroom.

3. First-Pass Yield (FPY)

FPY = (commit final lines - diff(v1, commit) changed lines) / commit final lines

, computed per task then weighted by line count. Wide FPY variance within a single requirement reveals specific tasks needing better specs or context. Two caveats: FPY measures generation quality, not code quality (high FPY could mean AI got it right or humans rubber-stamped); a single trace illustrates algorithm mechanics, not fleet-wide baseline. All three metrics settle at commit time — they answer "how well did AI generate" but not "did the system improve post-merge." Next metrics: post-merge survival rate, rework rate, change failure rate.

Three Pitfalls & Lessons

Pitfall 1: Overkill Agent for Simple Tasks

Initially gave strongest Agent to everyone. Costly: single-function completion triggered full Agent flow — slow and token-heavy. Solution: route by task step count, not tool strength. ≤2 steps, single file, clear semantics → Vibe Coding (IDE tab completion, zero learning curve). Cross-file, multi-step, ambiguous semantics → Agentic Workflow via AiBox with full trace. Hard threshold (currently 2 steps) prevents subjective routing; will adjust with model capability.

Pitfall 2: Context Overload Degrades Quality

Early approach: stuff 100k tokens per prompt. Two problems: attention dilution lowered generation quality; token cost soared. Switched to precise retrieval with three mechanisms:

Scenario context packs: retrieve by PRD type and business domain, not monolithic dump

Keyword-triggered knowledge injection: inject relevant spec only on keyword hit

Three-dimensional evaluation: track coverage, accuracy, auto-injection rate together to avoid single-metric over-optimization

Gatekeeping rule: every added spec must define its injection condition and hit rate; unanswerable specs are noise. Post-change: context volume dropped, key recall rate held or improved (qualitative observation, no public numbers).

Pitfall 3: AI Hallucinations Need a Second Pair of Eyes

AI invents non-existent APIs, fields, dependencies — logic reads clean but facts are wrong, easily missed in human review. Implemented dual-agent adversarial review : primary Agent implements per spec with source-code grounding; secondary Agent independently reads real source code and runs tests, solely verifying API/field/dependency existence. On mismatch, secondary rejects with feedback; primary fixes and re-verifies until pass.

Critical: secondary Agent must not reuse primary's context, or it co-hallucinates. It must independently return to source code.

Summary & Outlook

Why Put Specs in the Platform

For a thousand-person team, the challenge is getting specs into hundreds of legacy services and wiring them into existing requirement, development, and review flows. Platform integration lets engineers fetch relevant specs during requirement handling without manual document hunting or rule configuration.

Two Judgments (Not Industry Consensus)

Spec maintenance and governance need continued refinement. Even with SKILL.md/AGENTS.md formats, teams must define: who writes rules, who reviews, how expired rules retire, how conflicts resolve. Currently using usage feedback to drive revision or retirement.

R&D effectiveness measurement is splitting into three layers. Post-AI, lines and hours are insufficient; must add change failure rate, lead time, production incidents. This year added adoption rate, FPY, spec hit rate to pinpoint process bottlenecks. Next: post-merge metrics — how long AI-generated code survives, whether it causes rework or incidents.

Longer Time Horizon: L1 → L2 → L3

Internal three-level human-AI division: L1 fully manual (slow, controlled); L2 human sets goals, AI drafts, human final review (current state); L3 AI self-closes, human spot-checks only. Transition is gated by two hard constraints beyond model capability:

Delivery stability: AI accelerates change inflow; downstream test and feedback loops must keep pace or get overwhelmed.

Decision-right boundaries: which steps AI can close autonomously vs. require human sign-off. Must be drawn per scenario; wrong lines cause accidents or fake automation.

Over the past year, personal best practices were codified into shared team configurations, specs, and processes. The next test: are requirements delivered faster, is rework reduced, and does time saved by AI actually show up in team delivery outcomes.