Why Personal AI Coding Gains Don't Scale: HuoLaLa's Organizational Implementation Journey

Background: Personal Efficiency ≠ Organizational Efficiency

HuoLaLa's technology center comprises 1,000+ engineers across backend, mobile, frontend, algorithm, and testing roles, using 10+ mainstream tech stacks and dozens of business lines. After rolling out AI coding tools, five problems emerged: inconsistent usage patterns, duplicated project context preparation, divergent code styles, inability to quantify AI's efficiency impact, and uncontrolled costs and risks. The core realization: individual productivity gains do not automatically aggregate into organizational improvement.

Three AI Coding Modalities

The team classifies AI coding into three forms by human-AI division and runtime location:

Completion-style : Human leads, AI completes (IDE-internal) — e.g., GitHub Copilot, Tongyi Lingma.

Conversational : Human prompts, AI drafts (IDE-internal) — e.g., Cursor Chat, Qoder, Trae.

Agentic : Human sets goal, AI executes multi-step loop (IDE or CLI) — e.g., Cursor Agent, Claude Code, Codex.

The practice focuses on in-IDE usage; async cloud agents (task → sandbox → PR) are still under exploration.

Phase 1: Popularization — Raising the 80% Floor

A stark split appeared: 20% deep users leveraged context assembly, MCP + templates, Skills for code navigation, and Agent-driven test/CR automation, while 80% ordinary users started coding without context, let AI hallucinate designs, manually searched codebases, and skipped test execution and meaningful code review.

Unified Workbench: AiBox

AiBox centrally manages four configuration types and pushes them to every IDE:

Rules : Team coding constraints.

Skills : Reusable multi-step workflows.

Commands : Shortcuts for high-frequency actions.

MCP : External service and data source integrations.

Tool admission, usage statistics, compliance checks, and auditing are also centralized.

Training & Practical Scenarios

Four training pillars: calibrate expectations on AI reliability and hallucination; clarify IDE capability boundaries (completion, chat, agent); unify environment setup (Rules, project init, MCP); and codify onboarding flow. Daily development was broken into high-frequency scenarios with runnable examples: code comprehension (order module analysis), field+query addition (interface, DB column, query in one pass), code change review (structured CR report), and unit test generation (branch coverage without containers/DB).

Phase 2: Spec-Driven Development — Making AI Follow Specs

Popularization solved "can use"; spec-driven solves "stable and usable." Two drivers: average engineers write poor designs, so structure and template them for AI generation; at scale, the scarce resource shifts from code volume to structured domain knowledge.

Reference Frameworks

The team adopted OpenSpec's spec/delta structure (each change gets a change directory; delta marks track increments on legacy code — suited for large brownfield systems) and Superpowers' Skill methodology (no clarification no code; tasks sized to 2–5 minutes; enforced red-green-refactor TDD — suited for scaling methodology across thousands).

Five-Step SDD Pipeline

Requirement intake : Auto-fetch from PMIS (internal requirement platform) and link PRD.

Spec recall : AI retrieves candidate specs from tech/domain/dev/cr_spec libraries.

Context packaging : Specs, code snippets, historical designs bundled into a context manifest.

Human curation : Developer removes irrelevant items, adds missing ones, outputs proposal.md.

Atomic delta execution : Changes split into N atomic deltas; each validated before/after; failure rolls back only that step.

Each stage produces artifacts for human review and structured data for the next AI step (e.g., requirement → business intent + boundary constraints; PRD → structured PRD with functional goals, business rules, acceptance criteria; tech design → module boundaries, interface contracts, architectural constraints; task breakdown → per-task input/output/dependencies/acceptance; coding → deliverable code + CR report + new interfaces/rules; context write-back → updated knowledge assets for next iteration).

Spec Lifecycle Management

Specs are scored on recall hit-rate and mislead-rate after each use; low performers enter backlog for revision, long-unused specs are retired.

Phase 3: AI Native — Full Lifecycle Closure

Post-delivery, accumulated experience and production incidents must feed future requirements. Two loops:

Asset precipitation : Every process step auto-archives; pitfalls become new Rules and CR check items, effective next cycle.

Production feedback : Alerts trace back to process archives; root causes generate rules auto-applied to similar future requirements.

Target Organizational & Process Shifts

Team boundaries: frontend/backend/client → business-domain aligned.

Domain personnel: single-stack individuals → multi-stack + AI collaborators.

Engineer role: implementer → goal setter + result auditor.

Hiring: coding fluency → context construction ability.

Documentation: ad-hoc → fully spec-structured.

Workflow: experience-based → spec end-to-end.

Review: human-only → Agent first-pass + human second-pass.

Metrics: lines/hours → FPY, adoption rate, spec coverage.

Implicit tribal knowledge must be documented with explicit boundary conditions and acceptance criteria for AI consumption.

Measuring AI Effectiveness

AI Code Ratio (Three-Layer Signature)

AI Code Ratio = (AI-generated lines finally merged to main) / (Total new lines in main)

. "Finally merged" is critical — IDE-reported AI lines differ vastly from what survives. Three-layer collection:

IDE telemetry : Hash-sign each AI-generated line.

Commit diff layer : At git commit, verify each line still matches its signature.

Merge statistics layer : Count only lines that reach main.

Full-Chain Trace

End-to-end trace captures generations vs. adoptions per stage (e.g., a batch order export requirement: 1.5 hours, 20 spans; proposal generated 3× adopted 1×; tech design 2× adopted 1×; tasks split into 7; review took 2 passes). The gap between generations and adoptions signals improvement headroom.

First Pass Yield (FPY)

FPY = (commit final lines - diff(v1, commit) changed lines) / commit final lines

, computed per task then weighted by line count. FPY measures generation quality, not code quality — high FPY may mean accurate AI or lazy human approval; the two are indistinguishable from the metric alone. Low-FPY tasks pinpoint where specs and context need reinforcement. Current metrics settle at commit time; post-merge survival, rework, and change failure rates are next targets.

Three Pitfalls & Lessons

Pitfall 1: Overkill Agent for Simple Tasks

Initially gave strongest Agent to everyone; cost and latency high for trivial tasks (single function, ≤2 steps, single file, clear semantics). Solution: route by task step count, not tool strength. ≤2 steps, single file, clear semantics → Vibe Coding (IDE tab completion, zero learning curve). Cross-file, multi-step, ambiguous semantics → Agentic Workflow via AiBox with full traceability. Hard threshold (currently 2 steps) prevents subjective routing.

Pitfall 2: Context Overload Degrades Quality

Early "more context is better" led to 100k-token prompts, diluting attention and ballooning token cost. Shifted to precise recall with three mechanisms: scenario-specific context packs (by PRD type and domain), keyword-triggered knowledge injection (only on hit), and three-dimensional evaluation (coverage, accuracy, auto-injection rate) to avoid single-metric optimization. Gatekeeping rule: every added spec must define its injection condition and hit-rate; otherwise it's noise. Context volume dropped while key recall rate improved (qualitative observation, no public numbers).

Pitfall 3: AI Hallucinations Need a Second Pair of Eyes

AI invents non-existent APIs, fields, dependencies — logic reads fine but facts are wrong, easily missed in human review. Implemented master-slave dual Agent: Master implements per spec with source-code grounding; Slave independently reads actual source code and runs tests, solely verifying API/field/dependency existence. On mismatch, Slave rejects with feedback; Master revises and re-verifies until pass. Critical: Slave must NOT reuse Master's context, or it will co-hallucinate; it must re-read source independently.

Summary & Outlook

Why Spec in Platform

For a thousand-person team, the challenge is injecting specs into hundreds of legacy services and connecting them to existing requirement, development, and review flows. Platform integration lets developers pull relevant specs on demand without manual document hunting or rule configuration.

Two Judgments (Not Industry Consensus)

Spec maintenance needs maturation : With SKILL.md/AGENTS.md formats, governance remains — who writes rules, who reviews, how to retire expired rules, how to resolve conflicts. Currently using usage feedback to drive revision/retirement.

R&D efficiency metrics splitting into three layers : Post-AI, lines and hours are insufficient; must track change failure rate, lead time, production incidents. This year added adoption rate, FPY, spec hit-rate to diagnose process bottlenecks. Next: post-merge metrics — code longevity, rework, production impact.

Longer-Term Evolution: L1 → L2 → L3

L1 : Fully manual, slow but controlled.

L2 : Human defines goal, AI drafts, human final review (current state).

L3 : AI self-closes, human spot-checks only.

Transition is gated by two hard constraints: (1) Delivery stability — AI accelerates change inflow; downstream test/feedback must keep pace or be overwhelmed. (2) Decision-right delineation — which stages AI can close autonomously vs. require human sign-off; must be drawn per scenario, else accidents or false automation result.

The past year turned personal AI tricks into shared configs, specs, and processes. Next step: verify if delivery is faster and rework lower; time saved by AI must genuinely reflect in team delivery outcomes.