Skip to content
Multi-Agent R&D Pipeline: 3 Human Gates, 335 Reqs Delivered in 106-Min Median
R&D Management 35 min read

Multi-Agent R&D Pipeline: 3 Human Gates, 335 Reqs Delivered in 106-Min Median

Tencent's Social Security Center built a multi-agent end-to-end delivery platform that processes production requirements through nine automated stages with humans only confirming at three gates (requirement review, technical design, acceptance), achieving 335 deliveries with a 106-minute median cycle time and zero production incidents, using structured Specs, state machines, quality gates, and a knowledge base to eliminate handoff losses and context gaps.

Smart Era Software Development
Smart Era Software Development
Smart Era Software Development

Problem: Coding Fast but Delivery Not

Coding agents have become fast, yet the overall lead time from requirement to production remains days or weeks. The team identified that most stages exist to compensate for human limitations, not because the problem demands them. Adding AI pointwise to each stage yields only additive gains; letting agents own entire stages yields multiplicative improvement.

Three Root Causes

Translator role: Developers act as translators between vague requirements and agents. Example: "post list slow" requires knowing that author info comes from another service with only single-query API; a naive agent would add concurrency and hit rate limits.

Telephone game: Requirements pass through 7-8 hands, each rephrasing. Meetings (review, alignment, integration) exist mainly to confirm shared understanding.

Zero-based questioning: The team asked: is this stage needed because of human limits or because the problem requires it? They compared traditional stages before and after AI holds full context:

Frontend/backend split – existed because one person can't master both; with AI holding full context, both ends change in one pass, alignment meetings disappear.

Requirement review – existed due to long chain of misunderstanding; agents pass a structured Spec, so alignment meetings vanish.

Integration testing – existed because two developers work at different paces; agents deliver frontend and backend in the same pipeline, serial wait eliminated.

Rebuilt Process: Nine Stages, One Spec, One State Machine

Nine Stages with Three Human Gates

A real requirement: "Add multi-select policy when creating task" (no PRD).

Requirement Review (human #1): Review agent paraphrases requirement, lists uncertainties @ product, product answers, then confirm.

Feasibility Assessment: Only answers: can this be done end-to-end? If not, route to manual scheduling.

Repo Binding & Technical Design (human #2): Agent matches repos, generates design; tech lead confirms direction. Even with no questions, human must click confirm to proceed.

Development, Review & Fix, Deploy to Test: Fully automated by agents.

Acceptance & Release (human #3): Same afternoon, owner accepts and marks release.

Humans spend only minutes at each gate. The process halts for human decision when assessment says not end-to-end, review fails four rounds, or agent detects missing info.

Four Human Judgment Types

Choose design: Trade-offs (maintainability vs performance, cache worth it). AI lists options with pros/cons; tech lead decides.

Supply context: Undocumented legacy baggage, team conventions. One sentence from human often beats prompt tuning.

Spot anomalies: Detect when AI goes off track.

Define value: Which requirements are worth doing and priority. Human sets value; AI executes.

Case Study: Tag-Splitting Missed External Cron Job

An admin backend needed "tag splitting". AI modified frontend, backend API, storage; compile, unit tests, code review all passed. At acceptance, human discovered an external cron job writing to the same table and calling the changed interface. The cron job was not updated. Linters and unit tests cannot catch this; AI sees only inside the repo. After fix, the lesson was codified as a repo-level knowledge card so future agents know to check external dependents.

This shows human review should focus on intent and system-external impacts, not implementation details.

Structured Spec as Baton

Nine agents pass a structured Spec containing: refined requirement, acceptance criteria, technical design (files + steps), open questions with answers, and injected knowledge cards. The Spec is progressively completed. Two key designs:

Requester need not write everything upfront: review agent iteratively clarifies and writes back to TAPD.

Agents do not chat directly; they read/write fixed Spec fields to avoid information loss.

Why Multi-Agent Short Loops Over Single Long Loop

Context: Single agent context grows, gets compressed, loses info, attention drifts. Multi-agent: each agent high-cohesion, takes only needed fields, context length constant.

Waiting: Single agent wait lives in session; process restart loses it. Dozens of concurrent waits invisible to platform. Multi-agent: wait recorded as DB state, resume after a week knowing exact step.

Pause: Single agent can only idle. Multi-agent: context persisted, restart next month from same point.

Division: Single agent tuning conflicts across stages. Multi-agent: review agent tuned by PM, dev agent tuned by tech lead with dev skills and team norms, independent.

Therefore, the natural form of an R&D process is a state machine.

Three-Piece Toolkit

State Machine: Answers "next step? who can block?" 42 states total; each step has "working", "waiting human", "failed". 9 human-wait states are the gates. Granularity enables precise replay: "fixing" vs "reviewing" separate so crash knows which to re-run. Top-level state changes require ~10 mapping updates, so states added conservatively.

Context Minimization: Answers "what to feed each agent?" Agents exchange structured fields and deltas, not chat logs. Max 8 knowledge cards per invocation. Dev agent doesn't know how many rounds review agent spent with PM.

High-Cohesion Agents: Answers "who does this step, how to swap?" Each agent does one thing. Deterministic tasks (read ticket, match repo) use fixed I/O workflows; reasoning-heavy stages (review, dev, review) use planning agents. Skills iterate per agent without cross-impact.

These three pieces are business-agnostic and reusable.

Running Stably in Production

Agents on Cloud Dev Machines

Dev agent runs tens of minutes (clone, edit, test, push MR). Running locally would saturate CPU. Agents run on cloud dev machines, headless, horizontally scaled. They interact with platform only via MCP and HTTP: claim task, heartbeat, progress, read details, submit result. Every step's state, assignee, progress stored in DB as single source of truth. Management UI reads same records. Key designs: one agent per task, agent pushes progress, tool calls use one-time credentials. Agent is stateless; if it dies, platform still knows progress.

Crash Recovery from Checkpoints

Agents are processes that can die anytime. Like game saves: all intermediate states persisted, resume from latest checkpoint. Four failure types handled:

Process crash or deploy restart: On startup, scan stuck demands, replay by state, launch new agent to retry.

Multiple instances race for same task: Optimistic lock (CAS-style) ensures only one claims successfully.

Process alive but silent (deadloop, tool failures): Agent heartbeats with progress; timeout triggers re-queue for one retry, then alert for human to diagnose system vs agent issue.

Human wants full redo: In management UI, roll state back to phase entry. E.g., design already at code review, product wants change: revert to "requirement reviewing" and restart.

Recovery != re-run from start. If "fixing" crashes, replay "code review" not "fixing" because fix may have been submitted; re-review reveals current code state.

Four Quality Gates

Four quality gates each answering one question
Four quality gates each answering one question

Requirement Gate – Question: Is requirement clear? Key: AI paraphrases first, then iterative deepening clarification (not checklist). Four must-haves before pass: what, for whom, rules, how to verify.

Design Gate – Question: Can we code directly from this design? Key: Steps ordered by dependency (create table -> write API -> write logic -> callers). Step >5 files split for atomicity. Design agent uses stronger model; coding agent uses cheaper model due to small granularity. Design agent then self-critiques assuming overconfidence to surface uncertainties early.

Coding Gate – Question: Any over-engineering or out-of-scope changes? Key: Simplicity first (three duplicate lines better than premature abstraction). Scope discipline: only edit files listed in design; out-of-scope findings recorded in "found but not touched" section. AI tends to "drive-by refactor" non-standard legacy code; each such change risks hidden bugs.

Review Gate – Question: Any critical issues? Key: Must-fix: bugs, crashes, privilege escalation, data corruption. Style preferences only advisory, not blocking. Evidence traced from design intent to code. Six security hard lines: UID from login not trusted param, new APIs authenticated, SQL parameterized, no secrets in code/logs, external input validated at protocol boundary, resource access checks ownership.

First two gates prevent vagueness; last two prevent over-engineering. Every gate shifts detection left: AI dev consumes department budget, earlier fix cheaper. Review gate tiering does not lower standards; it separates must-block from advisory, aligning with Google code review principle: if change improves health, don't block for imperfection. Humans still hold acceptance and release gates.

Automation Guardrails: Knowing When to Stop

Quality gates catch bad outputs; a subtler problem is infinite spinning.

Repair non-convergence: Review-fix-review loop, fixing one thing breaks another, consuming tokens endlessly. Cap at 4 rounds, then escalate to tech lead.

Missing context/knowledge: Agent guesses blindly when missing lookup tables or configs, wasting tokens. Correct behavior: tag "needs info" and @ human.

These guardrails don't make AI smarter; they make the system stop when confused and hand back to humans. Confidence comes from knowing any problem triggers a stop and alert.

Knowledge Foundation: Deciding Scale

Three concentric circles: inner = AI end-to-end, middle = AI stalls needing context, outer = infra gaps
Three concentric circles: inner = AI end-to-end, middle = AI stalls needing context, outer = infra gaps

After stabilizing flow, bottleneck shifted to context/knowledge. AI's capability equals the team knowledge fed to it. Audit by requirement type: inner circle (AI end-to-end) ~30% of Security Center demands; outer circle (infra gaps, external systems) needs platform work; middle circle (AI stalls halfway) is the real opportunity. Data: of demands not fully automated, <5% due to AI generation failure; 34.7% stalled at requirement clarification and repo binding. Fill context, middle circle expands outward.

Knowledge for Agents vs Humans

Who writes: Human wiki relies on voluntary docs or management push. Agent knowledge: auto-harvested at process nodes, human only corrects. Human-supplied context at gates becomes knowledge.

Format: Human wiki = long docs. Agent knowledge = structured cards, small precise context, human- and machine-readable.

Consumption: Human searches when needed. Agent: pre-injected before action; agents don't know what they don't know, won't search proactively.

Maintenance: Human wiki = write-only. Agent knowledge = TTL, auto-expire. Humans judge stale docs; agents trust them blindly.

Traditional wiki aims for completeness; agent knowledge aims for precision and removability.

Three Knowledge Sources

Repo cold-start: On onboarding, distill repo profile: service paths, architecture, key conventions. Profile indexes code (still ground truth), accelerates AI understanding. Incremental refresh on main branch commits.

Process auto-harvest: Every gate (review, design, review, release) emits knowledge. Only knowledge from released, repo-linked demands is referenced. Most valuable: review rejections. Repeated same pitfall becomes repo-level constraint, injected pre-dev, checked at review.

Human feedback loop: Many demands still manual. Team provides Skill and CLI; after manual work in IDE, developer invokes Skill to reverse-harvest context and norms back to platform. Next similar demand, AI uses it directly.

Knowledge Cards: Instructional, Source-Tracked, Expirable

Real knowledge card from code review: binary vs base64 encoding
Real knowledge card from code review: binary vs base64 encoding

Example card from a review: an image API must return both base64 and binary. Review found that re-encoding binary branch causes JPEG lossy mismatch. Card contains:

One-sentence conclusion: Concise, single point.

Do/Don't as instructions: "Note encoding consistency" is a reminder; "Prohibit re-encoding for binary" is an executable instruction agents follow.

Source fact: Which demand, which repo. If source changes (business shift, code change), card must retire; otherwise it becomes noise worse than no knowledge.

Injection method: initially agents queried themselves; they almost never did. Now platform pre-retrieves by repo and stage, max 8 cards, injects before agent acts.

Injection Hit Rate Dropped as Volume Grew

Monthly injection hit rate: September highest volume, lowest hit rate 16.9%
Monthly injection hit rate: September highest volume, lowest hit rate 16.9%

September: ~2000 cards injected (record high), hit rate 16.9% (record low), far from 60% target. Like an overstuffed bookshelf: more books, harder to find the right one. Root cause: production side (auto-harvest) too strong, duplicates abound; governance side (recall, quality) too weak. Recall only considers relevance, not quality. More injection = more noise = agents misled. Four fixes in progress:

Deduplication: merge semantically identical cards with different wording.

Weighted re-rank: incorporate duplicate recall count and human audit results.

Inject only high-confidence cards.

TTL expiry: expire unused cards after validity period.

No public results yet. Advice: build these four from day one; don't wait until scale exposes the problem.

Results and Boundaries

Throughput Up 30%, Not Headcount Reduction

Monthly end-to-end delivery trend: April pilot, May-Jun backlog clearance, Jul-Sep quality focus
Monthly end-to-end delivery trend: April pilot, May-Jun backlog clearance, Jul-Sep quality focus

Team measures demand throughput, not headcount saved. AI replaces zero-value work. Excluding pure strategy/config demands, department throughput increased ~30%: same 20 people, from 100 to 130 demands/month. Saved effort redirected to architecture, low-level optimization, and AI infrastructure.

Monthly trend in three phases:

April: Canary, low volume.

May-June: Platform opened to PMs/ops; quickly cleared long-standing P2/P3 backlog, volume spiked.

July-September: Backlog cleared; team deliberately shifted from volume to quality while building knowledge base. As knowledge filled, scope expanded, September rebounded.

Additional numbers: 48.5% of platform demands completed fully automated (target 60%). AI delivers, but accountability stays with repo tech lead. Note denominators differ: "~30%" = share of Security Center total demands AI can handle end-to-end; 48.5% = share of platform-submitted demands fully automated; 30% = throughput gain after excluding config demands.

Boundary: Context Completeness, Not Size

Single criterion for AI suitability: can all relevant repo change contracts be defined upfront? If context complete, even large work fits (one demand estimated 3 days by AI review, actually done in 1 hour). If context incomplete, even tiny demand goes to human.

End-to-end: Changes within single repo, design/dependencies/norms definable upfront. Mainly admin backends, config, query features. Low-risk systems, large workload okay.

Human: External system integration (AI can't reach), architectural rewrite (human involvement >70%, not worth end-to-end), security core logic (policy forbids AI), C-end demands (wide verification surface, now human-AI collaboration).

Boundary is dynamic: each knowledge patch expands the capability circle.

Next Bottleneck: Verification

Dev cycle compressed from days to hours; verification still waits for human availability. Two experiments:

End-to-end auto test: Internal admin UIs can use Computer Use, Playwright for automated click-test; backend services and client apps not yet fully coverable.

Shift-left verification: Who owns release verifies. Backend services verified by developer per risk tier; admin features verified by requesting PM/ops in test env before release.

Trade-off: shift-left only for internal low-risk (~30% of demands). Zero-tolerance C-end scenarios still require rigorous testing.

Adoption Advice

Start with a concrete low-risk scenario, get it running: Team began with Security Center's admin backend: high volume, low risk, fast verification. Running beats upfront design; then expand to architecture design, multi-repo, C-end.

Copy the skeleton, tune parameters: State machine, context minimization, high-cohesion agents are business-agnostic. Number of states, agent skills/team norms, human gate placement, quality gate thresholds must fit your process.

Before building knowledge base, answer four questions: What scenarios? Who uses? What content? How to measure success? Knowledge injection hit rate is a key success metric.

When AI makes a stage redundant, that stage should disappear.
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

quality gatesstate machineknowledge basemulti-agentR&D automationend-to-end deliveryAI in software engineeringstructured specification
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.