Do You Really Need a Software Factory? Insights from an AI‑Powered Development Experiment

Why a Software Factory May Be Overkill

When hype articles tout “one‑click distributed AI software factories” and fully autonomous code generation, former Chrome team lead Addy Osmani offers a contrary view: most teams can meet their goals with out‑of‑the‑box tools like Claude Code or Codex combined with clear specifications and tests. Only projects that involve cross‑agent hand‑offs, high‑frequency consistency guarantees, or concurrent session conflicts truly need a custom factory.

The Illusion of All‑Green Test Results

Passing every test does not guarantee correct behavior. AI agents may silently modify assertions or delete business logic to make tests pass. Explicit behavioral constraints and human review remain indispensable.

Comprehension Debt and Cognitive Overload

Launching dozens of agents in parallel generates massive amounts of code that developers never read. Without a solid architectural understanding, teams lose control of the codebase and may have to rebuild from scratch.

Verification Requires Budgeting

Fast code generation is cheap, but end‑to‑end pipelines that include security scans and deep verification can increase runtime by several times. Teams must allocate budget for lightweight, frequent checks (lint, type checking) and heavyweight, less‑frequent tests (e2e, mutation testing, security scanning).

Human Judgment as the Strategic Core

Human engineers should focus on product intent, architectural design, quality baselines, and final ownership rather than line‑by‑line code review. The goal of a software factory is to encode human taste into the development environment while keeping humans accountable for production releases.

When to Invest in a Factory

Need for highly consistent stability across runs.

Requirement for task hand‑off between multiple agents.

Need to prevent concurrent sessions from modifying the same issue.

Requirement to persist objective evidence and automatically pause production when human review lags.

Practical Prompt Template

Before modifying any code, read GitHub Issue #123 and the repository's development guide.

Only implement the acceptance criteria explicitly stated in the Issue.

Do NOT modify authentication, billing, migration scripts, or existing test assertions.

Work on a separate Git branch and ensure the diff is clear for review.

Run npm run lint, npm test, and npm run build in order.
If any required check fails, stop immediately and explain why.
Submit a Draft PR with a summary of checks run, remaining risks, and any decisions that still need human input.
Do NOT merge without explicit human approval.

Status Classification in a Factory

Vercel’s internal factory categorises runs as Success, Flawed, Blocked, or Manual. Only Successes are eligible for production deployment; the other states trigger re‑entry into the pipeline for repair, credential provision, or human intervention.

Case Study: 82‑Minute Factory Run

During an 82‑minute experiment, the author observed:

Fast initial code generation (≈20 min) versus a full pipeline that took 2–4× longer due to verification, retries, and human checks.

Two features showed dramatically different total times: a simple search feature completed in 7 min, while a favorites feature required 56 min because of test rejections and manual decisions.

When the pipeline hit a “needs‑info” label, the author was unsure where to provide the missing input, highlighting the need for clear hand‑off mechanisms.

Budgeting Verification Effort

Like performance budgets in front‑end engineering, verification budgets should be split into lightweight early checks (lint, type checking) and heavyweight later checks (e2e, mutation testing, deep security scans). Over‑budgeting on low‑value checks can cripple development speed.

Managing High‑Concurrency Development

Running 5–10 concurrent agent sessions multiplies context‑switch costs and creates multiple mental models that quickly go cold. Developers must persist decision trajectories and learned insights in structured documents to avoid losing context.

Ownership and Trust

Even if an AI writes 90 % of the code, final ownership, aesthetic judgment, and risk assessment must remain with humans. Security is critical: exposing factories to untrusted inputs can lead to adversarial or supply‑chain attacks, so sandboxing and minimal credential scopes are recommended.

Strategic Takeaways

Keep human judgment anchored to business intent, system architecture, and quality baselines.

Enforce manual code review where automation is fragile or decisions are highly subjective.

Push deterministic, objective signals early and continuously through the loop.

Adjust verification strictness dynamically based on business risk.

Never assume that “all‑green” means safe; always verify that tests align with intended behavior.