AI's Scarce Resource Isn't Compute—It's Human Judgment: 12 Agent Engineering Talks
This conference report analyzes 12 talks from AI Engineer Shanghai 2026, revealing that Agent engineering's real challenges are verification, context management, sandbox infrastructure, LLM gateways, data layers, and cost optimization—with human judgment becoming the ultimate scarce resource as automation scales.
Conference Overview
The AI Engineer Shanghai 2026 afternoon session featured 12 talks that moved beyond model size to tackle the hard engineering problems of putting Agents into production. The talks formed a chain: long-horizon task execution → persistent runtime (sandboxes & gateways) → AI-native organization → memory & data layer → cost engineering → democratized creation (game development).
Talk Summaries
1. StepFun (Ailing Teng): Long-Horizon Execution as Product Ceiling
StepFun distinguishes generational model leaps from functional iterations. Their Step Code Bench focuses on coding scenarios, arguing that "what you test defines what the model becomes." Current Agents run 26–48 hours continuously; this duration sets the complexity ceiling. They open-sourced the Step Code harness, offer a cloud Studio with free sandbox credits, and plan a desktop app—aiming for 10× human productivity.
2. OpenClaw (Vincent Koc): From Intent Engineering to Verification Engineering
Koc runs 300+ parallel Agents, submitting 1,000–2,000 PRs daily from a laptop. The bottleneck shifted from speed to proof: "How do we know it really works?" He outlines three engineering paradigms: loop engineering → intent engineering (injecting human intent into frameworks) → verification engineering (writing verification conditions as code). OpenClaw's adaptive testing uses classifiers to predict regression risk and runs only the high-risk subset, targeting 90.5–98% CI cost reduction. A side experiment lets Agents play Doom to self-optimize latency, feeding back into framework speed. Key insight: when Agent output exceeds human review capacity, trust must come from automated verification, not manual diff reading.
3. AutoArk (Alex Zhou): On-Device AI's Three Bills & Intent Routing
Cloud AI faces linear token cost, network dependency, and privacy/compliance issues. On-device AI delivers millisecond response, offline certainty, data-on-device privacy, and zero token cost. AutoArk's model matrix includes Edge0 (on-device LLM), Audio8 (ASR/TTS, down to 0.1B for wearables), and a 150B MoE multimodal model running on 18–24 GB memory with >1000 tok/s prefill and >100 tok/s decode. The real engineering cost is full-duplex interruptible conversation pipelines; their EVA SDK provides 3A audio algorithms and cross-platform support. A headphone demo routes simple intents (heart rate) fully on-device, sending only open-ended questions to cloud—cutting token cost >92%. Audio8 models top Hugging Face speech leaderboards with 1.8M+ downloads.
4. Daytona (Ivan Burazin): Kubernetes ≠ Agent Sandbox
Sandbox semantics have drifted through four generations: running Python scripts → hosting entire Agents → spawning thousands for RL → computer-use Agents. Kubernetes optimizes for steady-state longevity; sandboxes need birth-fork-kill cycles that finish before K8s schedules. Daytona's solution: local snapshots (stop without delete, instant resume), snapshot distribution (push hot snapshots to nodes), delta-only transfer (database-style). Running 1,000 sandboxes for 176 hours costs 2.7× less than self-managed K8s. Warning: sandbox breaches can kill a product; delegate to specialists.
5. Manus (Fan Bin): Mono Repo + Public Tasks = AI-Native Organization
Context fragmentation across personal repos and internal systems blocks AI autonomy. Manus consolidates everything into a mono repo with integrated debug, log, and analytics tools. New hires onboard instantly; product managers query logs, fix simple bugs, and hand off complex ones with root-cause analysis attached. Full-chain OpenTelemetry tracing cuts bug localization to 20 minutes (5 minutes in emergency mode). Public tasks enable relay: decision-makers open a task link, see full context, decide, and the Agent continues. Plugin architecture and flat "work-assigned-to-person" org let 1–2 people own large modules; a product manager merged 160+ PRs with zero quality incidents. Digital twins auto-check service health (once beating monitoring systems) and auto-process GitHub issues—though PR creation outpaces review, risking token waste. Three honest pitfalls: (1) human importance rises—hybrid talent needed; (2) attention becomes the non-renewable resource—invest it in high-leverage judgments, codify routine decisions into skills/memory; (3) cost—one engineer spent $100k/month on tokens; complex tasks need human-in-the-loop, not autopilot. In a cyberpunk twist, 30% of Slack permission replies came from other Agents.
6. Exa (Jakub Hojsan): Search Layer for Personal Agents
Exa indexes 2T+ URLs, serves 100B pages at ~300 ms latency. A "Shanghai dinner" demo shows memory-injected preferences (budget, ambiance) driving search-first behavior: read highlights, decide what's missing, fetch only that. Dynamic Highlights selects the most informative tokens across results, saving ~95% tokens vs. full-page dump. The standout design: restraint. The Agent checks calendar, budget, seat availability before interrupting—"the scarce resource isn't information, it's permission to disturb." Advice: start with one repetitive task, add memory, then search, then teach silence.
7. Zilliz (Rentong Guo): Agent Memory = Next-Gen Training Data
Two exponential curves force architecture change: token consumption jumps an order of magnitude per intelligence tier; data entries grew 10× yearly for four years (1B → 10B → 100B → 1T). Zilliz Vector Lakebase merges database UX/performance with data-lake capacity/cost. Agent memory (dozens of reads/writes per trajectory, queried at step/span/trajectory granularity) is isomorphic to RL rollout storage—same operators for operational writes and analytical scans, no ETL. Hot/warm/cold tiers at $3 / $0.30 / $0.025 per GB/month; cold data surfaces in seconds. Cursor swapped databases four times in months—infrastructure evolves by business collision, not upfront design.
8. Optiver (Xinghong Fang): LLM Gateway as Utility Infrastructure
Gateway handles 3M requests/day across 4 data centers, 22 models, 95% traffic, 0.4% upstream error rate. Three "fake success" modes: streamed 200 OK that stalls mid-response, cache invalidated by protocol translation (cache hits cost 1/10th), silent drop of thinking blocks (e.g., Anthropic beta flags unsupported on Bedrock). Failover rules: session stickiness (same session → same provider), feature alignment (disable unsupported features), constant backup rehearsal (10% traffic always on backup), circuit breakers per error type. A costly lesson: centralized auth + full prompt logging in PostgreSQL caused 200 GB/day cross-Atlantic, 4 TB DB, 2 s auth latency. Fix: stream logs to S3, local auth replicas (2 s → 20 ms), eventual-consistency billing. Observability drills to per-tool-call traces cut debugging from minutes to seconds. They built a minimal custom gateway but may return to open source—"build auth once" and "build gateway once" are the same judgment.
9. TiDB/PingCAP (Li Li): Database Pushed Forward by AI Startups
Four infrastructure layers emerged from a single "need it tomorrow" request: (1) Agent state persistence (meta snapshots + events) scaling from TB to PB across 20+ shards; (2) cross-sandbox persistent filesystem for millions of isolated project spaces; (3) ten-million isolated database hosting; (4) hundreds of TB/day tracing with JSON-to-SQL column expansion and object-store spillover. The closure: bind meta snapshots to filesystem checkpoints to restore Agents to exact prior state, enabling branch-and-compare experiments. "No one plans this much AI Infra—user demand pushes the database there." AI Infra merit: can it resume (meta), restore (checkpoint), and A/B test (branch).
10. Techcombank (Do Tuan Anh): Inference Cost Efficiency Threshold
Three billing modes: seat subscription, per-token, reserved capacity (self-deploy). For GLM 5.3 on 16×B200 ($166k/month) vs. $2/M tokens, the breakeven is 83M tokens/month. Below that, reserved loses ~$3k/month; at 300B tokens, reserved saves 58% ($200k vs $500k). Hard constraints: GPU availability (biggest blocker), concurrency drives config (1×8×B200 ≈ 8 concurrent users), auto-scaling needs per-token fallback (~10% traffic). Numbers are "gross"—discounts, savings plans, ops labor shift results. Semantic/language model routing exploits tokenization differences (Vietnamese/Chinese efficiency) for another 40–50% savings. Cost is a solvable math problem: decompose into unit price × volume × routing efficiency.
11. Google (Tinsley Shi): Million-Sandbox RL Infrastructure
RL post-training (coding, etc.) is blocked by sandbox scheduling. Minimal RL loop: task → sandbox → execute → verify → LLM-as-judge. Frontier labs need 1,000 tasks × 16 rollouts = 16,000 sandbox starts/stops per task. Images >1 GB; Kimi K3 used 1.5M+ images. Infrastructure pollution: slow sandbox prep gets rewarded as task failure, corrupting model weights. Observed 80% sandbox idle time (waiting for model), only 15% active—massive GPU waste. Four-layer architecture: thin Python SDK, backend scheduling, control plane, data/storage. Engineering answers: dynamic placement (300k images across 4k CPU nodes with cache), optimized control plane (API server/controllers for scale), simplified network identity (10k sandboxes → 512 identities), high-density packing (60 sandboxes/node). Open-source Agent Sandbox: 1M concurrent stable 8 days, zero errors; first-instruction <1 s P99; 1M sandboxes prewarmed in 12 minutes; tested on 300k public images (swe-bench etc.); validated on TPU with Qwen 379B. Next-gen "agent substrate": snapshot-based sub-second start/stop to eliminate idle time; new data plane for control throughput.
12. Kimi/Moonshot (Tang Feihu): Game Dev When Code Isn't Hand-Written
Demos: fully model-rendered animation with structural understanding; non-dev sketches storyboard → playable animation. Long-horizon capability leap: engineers no longer write or review code ("too much to review"). Benchmarks like Software Marathon and Agent Last Exam demand end-to-end delivery in sandboxes (e.g., replicate RPG Maker's Tower of Magic, operate Blender). Prototype time collapses. Human role shifts to verify & decide. Three internal games: avg 4 days, ~3000 RMB tokens. Video-to-game: "Grandma GTA" video → playable open-source game in one week. Non-devs build 3A-style action games, port Diablo 2 to iPhone with touch redesign. Agent self-debug: screenshots own gameplay, understands bug, fixes, writes report. 3D assets fully model-generated. When idea-to-playable-prototype costs 4 days/$3k, bottleneck moves from throughput to taste & verification—echoing OpenClaw's verification engineering, Manus's non-renewable attention, and Kimi's own "human only verifies & decides."
Seven Cross-Cutting Patterns
Verification over loops: OpenClaw's adaptive testing (90.5–98% CI cut), Manus's public-task review, TiDB's checkpoint A/B tests—all replace "I trust it" with "system proves it."
Context as first-class resource (two directions): Mono repo solves AI's context blindness; public tasks solve human context blindness. Exa adds memory injection + search-first; TiDB builds Agent Data Layer (meta + files + DB + tracing + checkpoint). Output quality ceiling = relevant context visibility.
Exponential Agent data rewrites infra: Zilliz 10×/year for 4 years; TiDB PB-scale, 20+ shards, hundreds TB/day tracing; Google 1M sandboxes, 1.5M images. Convergent tactics: tiering (hot/warm/cold), delta-only, snapshots, prewarm pools—unlimited capacity with near-zero switch cost.
Cost engineered as calculable math: Five independent levers: on-device intent routing (>92% token cut), infra substitution (Daytona 2.7× cheaper), capacity mode switching (Techcombank 83M token threshold, 30–50% savings), selective test execution (OpenClaw 90%+ CI cut), routing/tokenization optimization (Techcombank 40–50% extra). From "skip one API call" to "buy one less GPU," every layer is quantifiable.
Reliability concentrates on shared state: Optiver's gateway textbook: cache sharing (protocol translation breaks it), session stickiness, feature alignment, backup rehearsal, logs out of hot path, local auth, eventual-consistency billing. Rule: hot-path shared state → local replica; cold data → out of DB.
Human role redefinition: Manus: stronger AI → bottleneck returns to human → need hybrid talent. Kimi: engineers don't review → human only verifies & decides. OpenClaw: factory manager → verification engineer. New role: define validity, judge worth, make non-delegable decisions. Fuel: attention (non-renewable vs. renewable tokens).
Creation & verification thresholds collapse & rise together: Kimi: 4 days / 3000 RMB for playable games; Grandma GTA playable in 1 week; non-devs build 3A clones. Making ideas real gets cheap; judging which ideas are worth making gets precious—back to pattern 1: verification & taste are the scarce commodities.
Breakthrough Table (17 Hard Problems)
Key entries include: verification bottleneck → adaptive testing (classifier predicts regression risk, tests subset); AI-side context fragmentation → mono repo + OpenTelemetry (20-min bug localization); human-side context fragmentation → public task links (decision-makers join mid-stream); cloud AI pain points → on-device intent routing (>92% token cut); sandbox/K8s mismatch → local snapshots + delta transfer (2.7× cost reduction); personal agent token bloat → Dynamic Highlights + search-first (95% token savings); personal agent over-notification → condition-gated interrupts; data explosion → Vector Lakebase hot/warm/cold ($3/$0.30/$0.025 per GB/mo, second-level cold recall); Agent memory/training data split → unified operators (no ETL); LLM gateway fake successes → session stickiness + feature alignment + backup rehearsal + circuit breakers; gateway shared-state cost → S3 logs + local auth replicas (2s→20ms) + eventual-consistency billing; Agent data layer → meta/files/DB/checkpoint quartet (resume, restore, branch); inference billing chaos → efficiency threshold formula (83M tokens/mo); further inference optimization → semantic/language routing (40–50% extra); RL sandbox throughput → 1M concurrent, 12-min prewarm, P99<1s, 60 sandboxes/node; RL data pollution → infra/eval decoupling (distinguish task failure from infra failure); game dev barrier → long-horizon Agent + video-to-game (4 days, self-debug, auto 3D assets).
Epilogue: The Structural Shift
Morning session said Agent success hinges on identity, iteration, evaluation. Afternoon translated those into sandboxes, gateways, data layers, bills. No talk discussed bigger models; all answered: "When model capability doubles every two weeks, what engineering velocity keeps up?" Six replacements: verification replaces loops (OpenClaw), context replaces communication (Manus/Exa), infinite capacity replaces finite cups (Zilliz/TiDB), math replaces gut feel (Techcombank), sub-second snapshots replace long waits (Google), on-device routing replaces full-cloud (AutoArk). Underneath all: tiering . Compute tiers edge/cloud (AutoArk), storage tiers hot/warm/cold (Zilliz/Optiver), session tiers across providers (Optiver), test tiers by risk (OpenClaw), inference tiers reserved/on-demand (Techcombank), sandbox tiers active/idle (Google). When growth is exponential, tiering is the only engineering answer—expensive resources for the tiny fraction that needs them, seamless switches so users never feel the seam (cold data surfaces in seconds, auth replicas sync in seconds, 10% backup traffic unnoticed).
The unwritten constraint across every tier: human attention . Manus: "Tokens renewable, attention not." Exa: product principle—interrupt only when all conditions met. Kimi: code volume exceeds review capacity → human only verifies & decides. Three independent paths converge: When throughput is no longer the bottleneck, the human role shrinks to two acts—define what counts as valid, and decide what's worth doing.
The deepest insight of the afternoon isn't a technology, but a position shift: In the AI era, the scarcest resource isn't compute, nor tokens—it's the few judgments a human can make. Invest those in the highest-leverage spots; hand the rest to Agents.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
