Why Humans Must Keep a Light On in Fully Automated Software Factories

From Dark to Light Factories

AI agents are now able to generate, test, and submit code at unprecedented speed, forming what Addy Osmani calls a Software Factory . Two extremes exist: the Dark Factory , where code is produced and deployed without any human reading, and the Light Factory , where humans intervene at high‑value checkpoints to preserve understanding.

Core Concepts: Loop, Harness, Factory

Loop : a single agent repeatedly performs a task – collect context → act → check result → repeat until a condition is met.

Harness : the surrounding framework that provides sandboxing, tool access, memory, and gating for a loop.

Factory : many harnessed loops run concurrently, fed by a task queue and gated by a review gate, while humans supervise at the highest level.

The factory is therefore a collection of harnessed loops at scale , not a single super‑agent.

The Danger of Dark Factories

Chasing full automation inflates comprehension debt – the gap between the total amount of code in a repository and the amount humans can actually understand. The bottleneck is not code generation speed but the cost and reliability of verification . When verification is cheap and fast, loops can be granted autonomy; otherwise they must remain under human control.

Back‑Pressure Law

Any loop’s autonomy must be strictly limited by the verification capacity of humans or automated tests – even a single inch of extra autonomy is prohibited. Modern frameworks such as LangGraph enforce this by turning free loops into state‑machine‑driven graphs that embed back‑pressure routes.

Human‑Owned Outer Loop

Engineers should own the Outer Loop , focusing on system architecture, boundary validation, and final decision‑making rather than low‑level code edits. This shift moves judgment upstream to design and planning, reducing the need for massive post‑generation code reviews.

Historical Perspective

The idea dates back to Bob Bemer’s 1968 paper “The economics of program production”, which envisioned repeatable, measurable software production. Past attempts failed because creative insight cannot be mass‑produced like physical parts.

Current AI Agents

Even the most advanced coding agents (e.g., Claude Code, Codex) are trained on their own harnesses and toolchains, making them excellent at short‑term, greenfield tasks but blind to long‑term maintainability. They lack awareness of the “comprehension debt” that accrues in legacy (brownfield) systems.

Case Study: Four‑Month Fully Automated Factory

Running a four‑month fully automated code factory without any human reading led to catastrophic failure. The trade‑off was maximising token utilisation versus preserving human understanding – the latter collapsed to near zero.

Verification as the Ultimate Constraint

Cheap, high‑reliability verification (CI/CD, unit tests, static analysis, security scans) can scale to near‑zero marginal cost. However, the Review Gate remains the only expensive, non‑scalable component, representing human judgment and aesthetic taste.

product intent → demand queue → harness builds → automated checks → deploy → monitoring → feedback to queue

This pipeline shows that only the review gate incurs significant cost and cannot be naïvely automated.

Turning the Light Back On

In a Light Factory, agents still perform the bulk of work, but before any high‑risk decision the code must be reviewed by a human. The human’s judgment is moved upstream to architecture and design, dramatically cutting downstream review effort.

Classic Architectural Safeguards

Strong type systems and function signatures catch errors at compile time.

Test seams make system behavior observable.

Clear code layout and short call stacks aid readability.

Explicit component boundaries limit blast radius.

Dependency injection enables safe module replacement.

These practices become a safety net that prevents AI agents from introducing irreversible errors.

Responsibilities of the Outer Loop

Decision – determine the correct direction for a problem.

Verification – ensure AI‑generated solutions are robust.

Approval – sign off and deploy changes.

Accountability – bear the consequences of any mistake.

Evidence (diffs, test reports, logs) links the inner loop’s automated work to the outer loop’s human oversight.

Conclusion

Engineers must shift from being code‑writing laborers to architects and gatekeepers, using proven architectural practices to constrain AI agents. Only by keeping the “light” on at critical decision points can we avoid the silent collapse of a dark software factory.