AI Coding Deep Dive: Why Humans Must Retreat to Judgment, Not Code Review

The author, Huang Xun from PUBG Mobile backend, details a quarter of hands-on work building production-grade AI coding agents. The core thesis: machines provide verifiable facts, humans make judgments — don't trust declarations, verify facts.

Part 1: Prompt Engineering Hits a Wall — Constraints Must Sink into Frameworks

The team built a test agent (ut-tester) for a mature end-to-end test framework. After months of prompt tuning, marginal returns vanished. Two concrete failures illustrate why:

Mock restoration: Despite repeated prompt instructions to pair setup/teardown, the AI missed teardown ~50% of the time, polluting test environments. Solution: declarative mock with auto-restore in the framework — the framework automatically restores mocks after each test case. This eliminated an entire error class and freed the model's attention for the main task.

Trace timeout analysis: AI struggled with chain-timeout debugging, often exhausting context. The framework was enhanced to auto-classify timeouts (queued, business logic no response, rate-limited, response dropped) and return exact code execution paths with the failing line marked. The error channel became a precise prompt-injection channel — but noisy signals (e.g., warning on unknown protocol names) misled the AI more than no signal.

Key insight: Prompt-layer constraints make the model dumber as they pile up; framework-layer constraints make it stronger . The skill file (SKILL.md) grew to 400+ lines then stopped — each added rule fragmented attention. Meanwhile, framework commits outpaced skill changes 4:1. The criterion for sinking a rule: hard gate only for deterministic facts (path prefix match), warning only for proxy indicators ("read coding standard" ≠ "followed it") . Signals must come from outside the AI (e.g., coverage 0/0 lines catches fake tests), and false-positive gates force worse code to pass checks.

Part 2: Orchestration Ends at Runtime Sovereignty

The team evolved from a workflow-engine (state-machine spawning child agents) to Claude Code's agent teams (mesh of long-lived agents). Workflow-engine gave stability but three problems: token waste (re-reading same code), poor UX (opaque child agents), and idle main agent.

Two experiments:

Full flow on agent teams (experiments/2026-05-29-team-driven-dev): brainstorming → team-execution with 7 roles (planner, implementer, reviewer×N, test-planner, test-engineer, explorer, guardian). Failed due to: model untrained on the mechanism (constant prompt patches), interrupt livelock (main agent pinging planner every minute while planner needed >1 min warm-up), unreliable member lifecycle, and counter-intuitive mechanics (kill tool needs taskId but spawn returns agentId). Produced agent-teams skill — an ops knowledge base from 100+ failed sessions (idleReason semantics, failureReason routing, timeout-based liveness, communication discipline).

Narrowed to test flow only (/module-test): main agent writes spec (frozen verification matrix), two agents (test-engineer + reviewer) peer-review point-to-point, main agent only monitors. Stable but reviewer suffers confirmation bias — fixed only by spawning fresh reviewer each round (token trade-off accepted).

Conclusion: Skill patches mechanism imperfections but cannot fix mechanism defects . Three black-box incidents proved every agent-loop layer must be auditable and intervenable:

Claude Code telemetry poisoning for China-origin users.

CLI UTF-8 corruption when writing long Chinese files (byte-slice breaks multi-byte chars) — 36/37 docs damaged, 370 U+FFFD.

11-hour spin from compression bug: max_turns ignored, 100K threshold on 200K window, summary prompt treated as user request, critical "progress/todo" sections dropped to titles — 203 compressions = 203 restarts.

Runtime sovereignty = ability to replace: tool param stitching (outer), context injection/telemetry (audit), compression strategy (deep, inside loop). Evaluated options: trpc-agent-go (full rewrite), trpc-agent-go thin wrap (me-too), Claude/CodeBuddy SDK (black box), Codex CLI (open source but encrypted child-agent payload locks to OpenAI), pi agent (MIT, minimal core, extensible TUI, focus on context engineering), DeepSeek Harness (MIT, everything plugin via Cordis meta-framework, loop itself replaceable, append-only event log for full audit). Chose pi first (months of contact, low extension barrier, hits pain point), DSH later after self-verification.

Part 3: Scores Are Unreliable — The Problem Is the Test Set

Built a heavy 170k-line eval platform (107 days, 25 real tasks, 4-layer weighted scoring, LLM-as-judge, 6-container Langfuse stack). Result: only 2 complete runs , scores 69.9 and 51.9, one layer skipped, "attribution summary generation failed". Logs deleted 20 days later.

Four root causes:

Non-executable criteria: Reference-implementation diff penalized valid architectural alternatives. Switched to checklist → main metric became LLM semantic judgment (Cohen's Kappa 0.10–0.21 = "almost no agreement").

Judge bugs: Format parse error turned 98 into 0 on same artifact.

Pipeline maintenance = full-time job: Deadlocks, SIGTTIN hangs, zombie processes, plugin load failures — none related to AI coding ability.

Test set lagged the system: Platform moved 3×, scoring changed 3×, while the evaluated command was merged into another.

Contrast: a 16-run A/B (8 tasks × 2 retrieval strategies) in one day changed a production strategy still in use (routing doc → direct grep, 12→2 hops). Difference isn't engineering effort; it's whether criteria are executable and conclusions become a single commit.

Industry survey (50+ papers/benchmarks): SWE-Bench, Meta TestGen-LLM, TestGenEval, SWT-Bench — all paper-backed methods held; the sole custom method masked refill (delete test items, measure recall) collapsed. Construct validity study: fuzzer killed 100% injected bugs but 0% real bugs — synthetic defects are easier. V2 design: criteria must be execution results, not synthetic lists . Main metric = mutation kill rate (TestGenEval: GPT-4o coverage 35.2% vs mutation score 18.8% — coverage gamed by empty assertions).

V2 test-set construction (three layers): Scenario (6 biz domains), Prototype (6 failure modes from review checklist), Four-axis difficulty balancing (branch complexity, dep cost, side effects, visibility). 6 hard gates (15–80 LOC, branch density ≥3, mockable deps, ≥6 mutable points). Mutants = real bug taxonomy (operator flip, and/or swap, remove or 0 fallback, comment-out state write).

Human-machine division on the pipeline:

Human proposes → 6 Sonnet sub-agents fetch candidates from biz code (file map → verify → answer gates) → Human final pick (34→16, balance scenario/prototype/mock weight, log rejections) → Script generates mutants (operator quota) → 3-level verify (Sonnet drops obvious equivalents → execution verdict → human only resolves disagreements) → DeepSeek-V4-pro answers 3×, average taken.

Three critical split points: (1) Retrieval needs judgment ("check_" may be passthrough, "update_" real validation) — human must re-verify at function granularity. (2) "What counts as a real bug" is pure value judgment (log diff ≠ observable; killability by natural construction only). (3) Verifiers err: LLM missed short-circuit on and/or flip, single-branch analysis, wrong column on duplicate symbols; reference tests had bugs (pcall closure nil leak) — order fixed: LLM saves CPU, execution decides life/death.

Run discipline: 16 tasks × 3 runs = 48 runs, ~19h. Only mean; <6pp = no conclusion, 3–6pp = "possibly significant", >6pp = actionable (from Anthropic's ~6pp infra noise). Platform enforces 3-run default. Trust tool calls, not AI self-report (config declared Task tool removed but still present).

Eval self-review: missing bare-model baseline — but purpose matters: for optimization (find weaknesses), baseline adds no signal; for proving plugin value, it's a flaw. Design eval purpose first.

Five weakness clusters from surviving mutants: missing default fallback (largest), missing boundary equality, assert return only no state read-back, no illegal input, no enum constant iteration → each maps to a generic test-skill rule. Data gave weighting, not direction. Surprise: carefully designed prototype taxonomy did not predict scores (same label 91% vs 14%) — true driver: "does killing it require default/boundary/state construction?" Future difficulty calibration must follow this. A self-consistent taxonomy falsified by own data is the most valuable eval output.

Two final certainties: (1) Core eval work is building the test set, not the platform/pipeline — every evolution happened on the test items; scripts are deterministic plumbing, not architecture. (2) Eval helps improve agents but costs are prohibitive for product teams (~$500 + 10h calibration, 19h formal, thousands in tokens for test-set build). Daily iteration works via real-task replays + session analysis (as in Part 1).

Part 4: Humans Retreat to Decision Points

After AI takes coding, humans move to decision points (requirements + acceptance), not the production line. Two sharp questions:

Still need line-by-line review? antirez: "If you control the idea, line-by-line is suboptimal" — 5k lines/day unreadable, LLM good at local optimum not macro trade-offs, 8h better spent on direction + QA. Robert C. Martin: "Don't read agent code at all" — but wraps agent in strict gates (unit, Gherkin, QA, metrics, mutation testing , coverage). He drops implementation, holds criteria. Consensus: question isn't "review or not" but "what must humans see, what can machines gate?"

Can we trust AI-written tests? AI cheats to pass (weaken assertions, drop edges). Uncle Bob's mutation layer is the backstop. Two-pronged check: Offline = mutation kill rate (Part 3); Online = platform re-runs in independent containers, lays raw results on approval page (Part 5).

Some errors are in principle invisible to humans : CLI UTF-8 byte-slice corruption (36/37 docs, 370 U+FFFD) — files valid UTF-8, text reads fine, occasional missing char. No amount of staring catches it. Same for "tests pass but coverage empty" or "assertions silently relaxed". Hence: machines provide facts because some facts only machines can fetch . Gate added: scan for U+FFFD on ingest, hard reject.

Harness to coding agent = IDE to human programmer. Stronger programmers demand better tools, not fewer. Two historical levers: better languages (OOP, types, GC) and better toolchains (LSP, lint, formatters, static analysis, profilers). For AI: build AI-native toolchains — simple, auto-trigger, instant feedback (like the trace tool). Legacy stacks don't excuse skipping this; even the greenfield control-group platform needed it.

Must have out-of-session check pipeline : rule-based quality gates + semantic checklists. AI training is exam-oriented — it will escape sandboxes, copy test answers, mutate tests to pass. In-session constraints are bypassable; out-of-session pipelines are not.

Part 5: Turn Criteria into Daily-Running Process

Two unreliables: AI single-shot output (error count driven by volume, not per-shot accuracy) and human single-shot attention (fatigue, distortion). Solution: nested independent loops stacking reliability .

Inner loop: dev + self-test in one session (AI writes, AI runs, human watches).

Outer loop: auto code review, checklists, static checks, broader test verification — triggered by polling commits, independent of session; test verification uses platform's own containers, not producer's self-report.

Two design pillars: (1) Each loop independent — no causal link between trigger and verified object, so loop speaks new truth. (2) Outer loop doesn't eliminate attention load; it fragments it — single long session needs sustained divergent attention (human weakness); multiple loops = single-theme, concentrated attention, repeated exposure surfaces issues.

Gates (deterministic rules/checklists) consume zero attention — only exceptions reach humans. Gate criterion from Part 1: deterministic fact → hard block; proxy indicator → warning only.

Loop Interface: Only Facts Allowed

Three mechanisms grown from the test-case loop:

Producer ≠ Writer: AI submits candidates; platform is sole writer, per-repo serial queue. Flow: AI candidate → human review → serial ingest → asset. Review can edit, reassign, discard; ingest auto-merges or flags conflicts. Humans only on true conflicts.

Failure Must Have a Name: Ingest stuck in "merging" 4×. Root: optional repo capability "has actual diff" not plumbed; kernel defaulted to "has diff" → empty commit; ordinary errors didn't transition state; retry logic re-queued "merging" → self-sustaining loop. Fix: make capability required (compile fail if missing), skip commit on no-diff, explicit "merge failed" state (retryable), stop auto-retry on failure. State machine now 7 states — last one bought by incidents.

Single Fact Source, Derivable: Test-to-coverage linkage via path convention ucases/<target-path>/<func>_unit.lua — path doesn't lie, metadata does. (Legacy: 2k+ tests with empty owner field; code still trusts AI-filled value — half-fixed tech debt.) Coverage merge = line-wise union across test cases (single runs underestimate). "Not run" vs "run but 0%" strictly separated — denominator only includes executed functions. Built on wreckage of platform showing "collection failed" as "no data".

Loop Entry: Don't Trust Upstream Self-Report

AI runs tests and uploads coverage. Platform re-runs in its own container pool , puts per-case results on approval page. Results advisory, non-blocking — machine provides facts, human judges. Pool design: admins contribute dev containers; per-container serial slots, cross-slot parallel; append-only logs; failover to new slot. Manual runs on private pool, auto on public — no contention.

Assets Are Living Lifecycles, Not Files

Requirements, tests, knowledge enter as "candidates"; human judgment = issuance discipline. Unmanaged state: scattered AI tests (unknown runnable/stale), review feedback lost in chat, knowledge polluted with errors. Human judgment structured into DB (why discarded, why no-value, why rejected) → feedback signal to upstream prompts/skills. Calibrate human labels before auto-feedback — else dirty data trains producer (Part 3's broken scorer replay).

Asset must survive two versions: release + in-dev. Patch phase demands minimal diff, compat, data fix, observability — exactly the "value-oriented" constraints AI fails at (Part 1). Repo stores files; platform stores living context across two version cycles with different disciplines . Asset's life = loop's rotation.

Human Is a Node Inside the Loop, Not an Outside Inspector

Initial boundary: no cloud IDE (avoid Codespaces bloat). Overturned — built AgentUI (browser workbench, direct CLI attach, approvals/questions/interventions in-page). Reason: Human is a node inside the loop, not an external acceptor . Node must act — approve, finalize, review, talk to any agent — so platform must embed interaction. This replicates Part 2's control need (workflow-engine couldn't give it).

Core UX goal: lower the cost of human judgment . Three levers:

Markdown → HTML: 100+ line Markdown unreadable; HTML carries tables, SVG, collapse, nav, one-click share. Anthropic's Claude Code team same finding: HTML puts humans back in the loop. Critical: custom HTML must end with "export" (copy as JSON/prompt) — judgment loops back to AI. Done for test/coverage/trace reports; requirements/knowledge/reviews still Markdown.

Everything annotatable, annotations feed AI: In /dev plugin, human highlights spec → annotation becomes structured input to model. Human judgment = sole external input; mechanism must prevent loss, enable reuse.

Trace visualizations with numbers untouched by LLM: LLM only names, groups, explains; all numbers recomputed by deterministic scripts from raw evidence; conservation check fails → fall back to raw view. Visualization = absorb truth without sacrificing fidelity.

Loops Must Cover the Entire R&D Cycle

Same skeleton (trigger → produce → gate → human review → ingest → reuse) instantiated per phase. Near-term three:

Requirement pre-analysis: agent kicks off on ticket assignment — rough design, unspecified details, impact/effort estimate. Core output: clarification list, not the design (design is just one self-consistent AI take).

Bug pre-analysis: is it bug? client/DS/backend? cause? fix? Unique luxury: ground truth returns free when bug fixed — accuracy measurable, calibratable.

Review pre-analysis: agent pre-screens common bug patterns before human review, reserves human attention for true judgment calls.

Should a Product Team Build This Heavy Platform?

Two-layer answer:

Replaceable layers were never self-built: Containers (AnyDev), CLI (off-the-shelf), tickets (TAPD sync). Self-built = three things central platforms can't give: Criteria (mutation operators from our review checklist, patch discipline from our version policy), Asset model (path conventions, knowledge modules, requirement doc system), Loop wiring (who gets which exception when). Central platforms provide shell (compute, session, generic CI); they cannot provide the body . Same as Part 3: test set is body, criteria are body, asset model is body — bodies aren't generic; generics aren't bodies . When a generic platform truly does the shell better, swap it; self-built parts are pre-filtered for "irreplaceable" — not coincidence, principle.

Investment sink determines sink-or-swim: Platform's real output isn't code — it's criteria, test sets, asset models, loop designs, accumulated failure history. They travel with the team, not the code. Like Part 1's discarded specs: vessel disposable, clarity indispensable. Even if platform replaced tomorrow, zero body waste.

This platform is the runtime of the R&D system : humans, agents, assets woven into nested loops of differing speeds; humans step off the production line onto loop nodes to judge; machines serve verifiable facts at loop interfaces; assets flow and appreciate across loops.

Closing

Five parts, one repeated motion: swap "trust what it says" for "go verify it" . Don't trust state symbols → check for timeout messages. Don't trust live members → trust disk artifacts. Don't trust AI's self-run report → platform re-runs. Don't trust metadata ownership → derive from file path. Don't trust my own failure taxonomy → count killed mutants.

One sentence: Distrust declarations, trust facts; distrust self-discipline, trust mechanisms. It's the flip side of "machines provide facts, humans judge" — the machine half must emit verifiable facts, not plausible stories. Part 1 said "hard gates beat soft nudges"; this article drives it to the bottom.

One harder lesson, appeared four times this year: Part 1 — wrong coverage signal worse than none; Part 3 — format bug scored same artifact 98 vs 0; Part 3 — custom scorer logged "wrong file" as "not found"; Part 5 — platform showed "collection failed" as "no data". Same judgment, four separate failures. So "machines provide facts" isn't a slogan — the hard part isn't "humans judge", it's "is the fact the machine hands you actually true?" Fact-providing machines must themselves be verified.

Back to the opening platform: pure AI delivery (3 months, 7 services, 300k+ LOC, zero human-written lines) — and it still breaks. No contradiction — AI output capability is sufficient; precisely because it produces more and faster, safety nets are more necessary, not less. True for any stack.

Models strengthen quarterly; prompts, orchestration, eval numbers expire — they're flow . Constraint-heavy SOP compensating current model gaps get deleted on next model swap. But other things stay valuable across model generations: the process of thinking the problem through (even its spec output is disposable, the thinking isn't), ops discipline distilled from 100+ failed sessions, runtime sovereignty judgment, framework capabilities that don't tax model attention, acceptance levers, eval test sets, every criterion on the governance chain — they're constants . Doing AI engineering = finding and guarding constants in a torrent of flow.