Uber's AI Software Factory: How Enterprises Turn Coding Agents into Measured Production

Key Insights

Uber's Software Factory practice reveals five foundational principles:

Verified work results are the production unit. Only merged PRs, effective reviews, alert investigations, or maintenance tasks with defined completion criteria, quality signals, and cost attribution can be continuously compared and optimized.

Real-task benchmarks are a prerequisite for model routing. Generic leaderboards only describe average model capability; enterprises must measure quality, reliability, latency, and cost on their own tasks, codebases, and defect samples to choose configurations for different jobs.

Agent cost is determined by execution trajectory, not just token price. Prompt Cache, Tool Search, Code Mode, and Context Graph each reduce redundant context, tool definitions, protocol polling, and error search; the goal is to lower cost per trusted result.

Managed Runtime turns scattered session experience into reusable, governable organizational capability. Once a task enters a managed environment, model, tools, execution environment, identity, acceptance criteria, traces, and budgets can be configured per task instead of relying on each engineer's ad-hoc choices.

Software Factory must establish a controlled evolution mechanism driven by execution feedback. Every failure, human correction, and anomalous cost should enter traces and, after regression evaluation and human governance, be converted into benchmarks, context, skills, rules, or routing updates; otherwise scaling only amplifies the same errors.

1. Define Measurable Software Work Units First

Uber categorizes agent usage into four layers by task specialization and organizational control:

Raw Sessions – Engineers use Claude Code, Codex, or OpenCode locally; platform only sees per-session cost.

Sessions with Skills – Local interaction but invoking 3,600+ internal skills.

General Agent (Cortana) – Cloud-hosted, same skills managed centrally; unit becomes a Query.

Specialized Managed Agents – Built around specific tasks with dedicated metrics:

Minion: Intent → PR, measured per merged PR

uReview: Code review on all PRs, measured per review

Agentic XP: Experiment execution to report, measured per readout

Conan AI: Alert → root cause, measured per alert

Fawkes: Scheduled code maintenance, measured per cleanup

Only at the top layer are inputs, execution flow, completion conditions, escalation paths, and output artifacts encapsulated, enabling per-result cost calculation and configuration comparison. Factoryization requires definable product, process, quality standard, and unit cost. Agent sessions and tokens are merely activity and resource consumption, not output. A work item becomes an operable production unit only when it can be expressed as "fix this CI failure," "review this PR," "complete this migration," or "analyze this alert," and the system can judge acceptance.

Uber's public data shows >70% of PRs from agents, 3,600+ skills, 30k+ daily executions, 7× WAU growth, 9.4× request growth, yet total AI spend stable; per-1k-request cost down ~34%, per-session cost down 52% from June peak.

2. Build Model Routing on Real-Task Benchmarks

Instead of buying frontier models by public benchmarks, Uber creates dedicated benchmarks per Managed Agent from real work. For uReview, they built an eval set from real PRs with known bugs, stratified Easy/Medium/Hard, tracking Precision, Recall, F1, cost per PR, latency, timeouts, and noise. Configurations are plotted on a quality-cost Pareto frontier; only frontier configs serve different risk tiers. Dominated configs (higher cost, lower quality) are retired.

Routing is not guessing prompt difficulty at request time; it classifies tasks first, then uses historical samples to judge real model performance on that task distribution. Same principle applies to subagents: "strong model plans, weak model executes" works only when task boundaries are clear and results verifiable by the planner or external validator. Retrieval, bulk formatting, fixed queries, and result aggregation suit cheaper models; architectural trade-offs, complex debugging, security reviews need stronger ones. Misclassification turns saved inference cost into retries and rework.

True routing depends on stable task definitions, representative samples, independent quality judgment, and continuously updated results. Without these, routers degrade to price arbitrage, erased by next model price drop. The accumulated asset is a judgment system answering "what work under what conditions by what config" – built from own codebase, toolchain, risk appetite – not purchasable.

3. Decompose Agent Cost into Observable Execution Trajectory

Uber breaks total spend into six multiplicative factors:

Total Spend = Users × Sessions/User × Turns/Session × Requests/Turn × Tokens/Request × Price/Token

First two = adoption & engagement (should grow). Middle three = agent execution trajectory (main optimization target). Last = vendor pricing + model choice.

This decomposition turns "AI cost too high" into a diagnosable engineering problem:

High tokens/request → oversized system prompts, preloaded tool schemas, uncompressed history, mismatched cache strategy.

High requests/turn → verbose tool protocols (e.g., DB query split into submit, poll, poll, fetch).

High turns/session → insufficient context (agent searches wrong locations), skills not codifying correct paths, unclear completion criteria causing rework.

Switching to cheaper models doesn't fix these; a cheaper model needing more turns may cost more overall. Uber's core metric in Managed Agents is Cost per Completed Task, observed alongside quality signals.

Metrics hierarchy: Portfolio (budget flow, tool usage), Unit Economics (per user, per 1k requests, per 1k sessions, per active hour, cache hit rate), Model Economics (request share vs cost per model), Driver Decomposition (cost change from adoption, engagement, input, output), Managed Agent Outcomes (cost per merged PR/review/alert/cleanup + Revert Rate, F1, MTTR).

Governance: real-time session spend in status bar, shared budget tiers, alerts at 50%/80%/100% with fast approval. Session-level analyzer reads local/remote sandbox traces, identifies 16 anti-patterns (e.g., using Opus for simple multi-turn, keeping 40KB MCP returns in context, long idle invalidating cache, preloading 100k tokens of system prompts/tools). It shows not just spend but causal behaviors and fixes.

Cost governance must prioritize default paths and system feedback. Engineers see cost and judge ROI; platform codifies verified optimizations as defaults. Any cost optimization must be observed with completion rate, rework rate, escaped defects, noise, human review time. Factory lowers cost per trusted result, not cost per failure.

4. Context & Tooling Determine Whether Agent Completes Task or Wastes Money

Six-factor equation shows enterprise agents spend most time finding info, understanding org relations, choosing tools, handling protocol overhead – not generating code.

1. Prompt Cache ROI Depends on Session Rhythm

Each request resends history, project context, tool results. Prompt Cache lets subsequent requests read cached prefix at lower price, but cache write has premium; different TTLs have different costs. Engineers' interactive sessions often pause >5 min; 5-min TTL causes repeated cache misses, forcing full-price prefix rebuild on resume. Solution: interactive sessions → 1-hour TTL; short-lived subagents → 5-min TTL. Same config yields different results on different workloads; no universal best practice, only pragmatic strategies matched to session rhythm.

Auto-compaction at 400k tokens (even if model supports 1M) because longer context increases cache burst cost, per-request duplicate input, and may lower info utilization. Default Reasoning Effort = Medium (output/reasoning tokens cost more; most tasks don't need max reasoning). Thresholds should be chosen per workload based on task duration, interaction interval, cache hit rate, post-compaction success rate, and cost.

2. MCP Value in Connecting Tools, Cost Also from Tools

Uber's MCP Gateway unifies 1,000+ internal/third-party MCP servers, centralizing auth and policy. Standard MCP direct access preloads tool schemas into session; 100+ tools → ~50-70k tokens just for schemas, repeated every turn. This "capability tax" pays for unused tools; tool count growth may lower selection accuracy.

Two complementary mechanisms:

CLI Tool Resolution – Maps MCP tools to shell commands; CLI resolves/executes via Gateway at call time, so internal schemas stay out of context.

Tool Search – Agent queries tool directory, loads only definitions needed for current task. Former solves call path, latter solves discovery path.

Code Mode – Traditional tool calls involve model in every protocol step (submit, poll 2-5×, fetch). Code Mode lets agent write a Python loop in subprocess to handle polling, returning only final summary to model. Five identical SQL queries: Code Mode cut >50% tokens single, >90% batch. Savings from eliminating schema init, multi-turn polling, redundant reasoning – not compressing results.

Factory must distinguish model-required steps (interpretation, planning, exception judgment) from deterministic steps (protocol polling, batch loops, format conversion, state queries) that should exit model loop. Involving model in every mechanical step is expensive and adds error surface.

3. Context Graph Turns Organizational Facts into Queryable Production Assets

Uber's codebase: hundreds of millions of lines, thousands of tables. Agents lack not syntax knowledge but: service ownership, which table is actually used, what deployment maps to what change, how similar incidents were resolved. Facts scattered across code, team dirs, incident records, PRs, architecture docs, deployment systems, query history.

AI Context Graph: 24M nodes, 80M edges, 86 node types, 117 edge types, connects 30+ internal systems, allows natural language queries.

Concrete comparison: Same prompt + model. With Graph Grounding: agent queries historical usage, finds table used by 50+ analysts, correct answer in 38 seconds. Without: agent cannot see table, spends 20 min checking service code, spawns two subagents, hits three errors, concludes data unqueryable. Difference not explained by "stronger reasoning"; missing key facts forces model to expand search in visible space, accumulating larger context and faulty assumptions. Stronger model searches smarter but cannot reliably infer a never-exposed internal table.

Context Engineering affects both quality and cost. High-quality context reduces invalid turns, subagents, tool calls, error recovery. Not every company needs Uber-scale graph; small teams may solve with unified search, repo rules, few structured indexes. Only when info is scattered, relationships complex, search cost persistently appears in traces does Context Graph investment make economic sense.

5. Managed Agent Returns Task Control to Organization

Interactive coding agents optimize individual work; engineer decides when to start, what context, which model, whether to accept. Execution paths stay dispersed; platform can't enforce same validation standards or replicate success across teams.

Managed Agent shifts organizational control over tasks:

uReview runs fixed flow on all PRs → unified benchmark, dedicated model, continuous F1/noise measurement, fleet-wide model migration.

Minion runs Intent→PR in cloud → platform controls sandbox, tools, identity, network, trace, exit conditions.

Conan AI handles alerts → system dictates readable info, actions needing escalation, valid RCA definition.

More sessions now triggered automatically by Managed Agents (code review, self-healing CI, visual-verified e2e PR, on-call alert triage, new bug debugging, ops tasks). Work entry shifts from "engineer opens agent" to "system starts controlled flow when conditions met" – synchronous human delegation to asynchronous event-driven.

Coding agent toolbox provides execution capability; Software Factory decides when tasks start, what config, how to accept, who gets escalation on failure.

Managed Agents still need explicit human authorization and escalation. As tasks approach production, identity and accountability matter. Agent may act on behalf of an engineer, then call another agent/tool. If system only logs final service account, audit cannot answer: who authorized, which agent decided, which tool executed, did permissions expand during delegation.

Uber's "Solving the Identity Crisis for AI Agents" extends zero-trust to agents: Agent Registry, workload identity, short-lived tokens preserve human→agent→downstream tool actor chain. Principles:

Agents have independent identity, not mixed with human long-lived credentials.

Every delegation retains "on behalf of whom, doing what" relationship.

Downstream tools see full authorization chain, not just caller.

Permissions granted per task, expire after execution.

High-risk/irreversible actions cannot bypass approval just because they're in auto flow.

Agent may have execution rights but not simultaneous authority to define goals, modify acceptance criteria, and approve own results. Otherwise closed loop just stuffs conflicting duties into one probabilistic system, reintroducing outcome uncertainty.

6. Make Every Failure a Self-Evolution Input

Only when feedback changes next execution does Software Factory become a learning system. Uber's Learning Loop components:

Real PRs + known bugs → Benchmarks

Model/Harness changes replayed on historical tasks

Session Analyzer extracts cost anti-patterns from traces

Engineers distill high-frequency work into Skills

Planned: propose updates from skill papercuts and accumulated traces

Loop:

Real task execution → Results & Traces → Failure & waste classification → Benchmark, Skill, Context, or Rule updates → Regression eval → New config to production

Different feedback → different systems:

Model misses known defect → Review Benchmark

Agent repeatedly searches wrong directory → update repo Context or Skill

Tool polling creates excessive requests → convert to Code Mode

Model frequently times out on task class → adjust routing

High-risk action without proper auth → identity/policy issue

Attributing all failures to "prompt not good enough" lets real system defects persist.

Traces are raw material. Final diff shows what changed, not why agent took 20 turns, called irrelevant tools, or lost context after compaction. Without process records, cost anomalies and failure patterns aren't reproducible; without reproducible samples, improvements rely on few engineers' memory.

Risks of "agent auto-improves itself": single failure may stem from model randomness, upstream data error, transient service fault, or wrong acceptance criteria. Direct rule generation from single trace hardens flukes into global constraints. Lower-level rules have wider blast radius.

Continuous improvement needs at least four guards:

Candidate rules only after similar failures reach frequency threshold.

Changes have clear owner and version history.

New Skill/strategy must pass regression on historical benchmarks.

Post-deploy monitor quality & cost, with fast rollback.

Agent can help discover patterns and generate edits, but must not rewrite its own evaluation standards and deploy directly.

7. How Enterprises Should Start Building a Software Factory

Uber's scale (hundreds of millions LOC, thousands of tables, 1,000+ MCP servers, massive agent traffic) isn't directly copyable. Most companies starting with unified gateway, org-wide knowledge graph, thousands of skills will build expensive infrastructure lacking production load.

Feasible path: build one closed-loop production line first, then decide which shared capabilities warrant platformization.

Step 1: Pick a Factory-Suitable Work Unit

Prioritize high-frequency, clear boundaries, verifiable results, isolatable/rollbackable failures. CI fix, dependency upgrade, scoped migration, routine review, alert triage, repetitive maintenance > new product design or cross-org architecture decisions. Answer four questions: input source? what artifact = done? who/what independently accepts? worst failure consequence? If unclear, don't discuss full autonomy yet. This step only picks first line, not specific agent/model.

Step 2: Define Unit Result, Not Agent

After task selection, write a measurable, verifiable production contract for one task. Not "build a Review Agent" but define "one effective review": covered issue types, min Precision/Recall, max noise/latency, required evidence citations, escalation triggers. Similarly, "auto-fix CI": allowed file edits, mandatory re-runs, auto-PR on success, failure stop conditions. Clearer work unit definition → easier model/harness/vendor swap later.

Step 3: Build Benchmark from Real Work

Sample recent completed tasks; keep original input, accepted result, known defects, execution env, acceptance criteria. Cover common tasks and high-risk long tail; stratify by difficulty, language, repo, system type. Metrics beyond success rate: quality, cost, latency, timeout, retry, noise, human review time – all together. Prevent data leakage/overfitting: hold out validation set not used in daily tuning; regularly add new incidents/tasks.

Step 4: Establish Minimal Managed Runtime

V1 doesn't need full "platform". Must provide: unified model interface, isolated execution env, tool permissions, task state, trace, budget log. Probabilistic agent handles planning/exceptions; deterministic steps (Git, test, state migration, permission checks) stay in workflow control layer.

Before production: Agent Identity, short-lived creds, least privilege, human approval. Control plane decides when to start, how to migrate, who authorizes; execution env ephemeral, destroyed after task. Agent with code execution rights must not own its permission policies.

Step 5: Grow Context & Skills with Traces

Don't start with org-wide Context Graph. Observe where agent wastes time: repeated service owner searches → integrate service directory; missing historical changes → connect PRs & deployment records; repeated tool sequences → codify as Skill or deterministic script; bloated tool schemas → introduce Tool Search or CLI Resolution.

Context should deliver minimal high-quality info needed for current task at right moment; Skills codify verified execution paths. 3,600 skills is Uber's scaled result, not your startup KPI. Example: team early pushed Claude Code + Skills for complex biz system → useless skills everywhere.

Step 6: Verify Unit Result Improvement Before Expanding Autonomy

One line must simultaneously track three outcome types:

Delivery outcomes: throughput, lead time, first-pass rate, rework, failure recovery.

Business outcomes: model cost per trusted result, infra cost, human review time.

Learning outcomes: % of failures entering benchmarks/rules, whether same issues recur.

Only when quality stable, human review declining, failures isolatable/rollbackable → expand task scope & permissions. Else higher concurrency just creates bigger review queues.

This sequence differs from "attach AI to each SDLC phase". Don't build Plan/Design/Build/Test/Deploy/Maintain agents first; find one end-to-end closable work class, complete task definition, execution, verification, governance, feedback. When multiple closed-loop lines share needs, abstract Gateway, Context, Identity, Benchmark, Skill platforms.

Factory AI's five-layer framework (Entry Point, Agents & Orchestration, Quality Gates, Operations & Knowledge, Foundation) provides external reference; Uber's Managed Agents, Benchmark, Gateway, Context Graph, Skills implement similar control logic. That framework suits post-multi-line platform direction, not Factory starting point.

8. What Not to Copy Directly from Uber

Different scale economics. Unified MCP Gateway, 30+ system Context Graph, dedicated SWE Benchmark, Session analysis platform have significant fixed costs. Low task volume → buying mature tools or lighter repo-level Context may be more rational.

Different code/process standardization. Uber's long-term monorepo, dev platform, service catalog, unified identity give agents machine-readable boundaries. Scattered docs, flaky tests, manual-deployment orgs need platform engineering first, not expecting agents to bypass gaps.

Different workloads. uReview's Pareto-optimal model only holds for Uber's real PR distribution. Another company's languages, defect types, security requirements, code structure differ; copying model choice is meaningless. Transferable is the benchmark-building method.

Cost isn't the only constraint. High-risk work (payments, identity, privacy, infra, data deletion) cannot auto-expand permissions even if model/human cost drops. Needs stricter separation of duties and approvals. Factory optimizes production, doesn't remove accountability.

Some harness components lose value as models improve. Complex retries, orchestration, compression, multi-agent structures address current model capability gaps. Post-upgrade, old scaffolding may no longer improve quality, even add cost/failure points. Every component = verifiable, deletable hypothesis, not permanent architecture.

Counter-case to consider: if a single model stably dominates most enterprise tasks, long-context and native tool use keep improving, small teams may not need complex Router, Context Graph, multi-layer Harness. This doesn't negate Software Factory; it tightens its definition: factory value depends on organization's ability to use data to judge which components still create value and promptly delete expired ones.

9. Factory Output Isn't Code

Previous article discussed how trusted results flow through AI-Native SDLC: Artifacts preserve state, Harness produces evidence, Governance decides stage progression. Uber's practice answers operational layer: how to make each transition measurable, comparable, governable, continuously optimizable.

Software Factory's basic production unit is independently verified software work result, not agents, tokens, or PR counts. Only after defining boundaries, cost, governance rules for such results can enterprises improve production via Benchmark, model routing, Context, Skills, Traces – continuously lowering cost per trusted result.

What enterprises truly manufacture is not more code or more agents, but a system that continuously turns organizational intent into trusted results and makes the next production cheaper and more reliable.