Why AI‑Powered Coding Factories Fail: Hidden Maintainability Defects in Lights‑Off Software Factories

01 From Traditional Factory to "Lights‑off" Factory: Accelerating the Path

The term "software factory" was defined at a 1968 NATO conference. Until 2022 the workflow was simple: product managers and engineers decide what to build, push requirements into trackers like Jira, developers take tasks, write code, undergo manual review, automated checks, and possibly manual testing before deployment. Teams mitigated long build times with upfront planning, architecture reviews, and sprint planning.

Around 2025 most tech companies claimed to have built an "AI coding agent factory" where agents generate over 75% of code, cutting build time from hours to minutes. However, human code review still took hours or days, so agencies added AI‑driven review and regression testing to accelerate that bottleneck.

The "lights‑off" software factory concept emerged, where humans no longer read code; monitoring events and user feedback feed directly into an agent pipeline that automatically creates pull requests for fixes, leaving developers only to add more tickets.

02 Maintainability Nightmare: Ignored Long‑term Cost

Industry voices warn that this rapid acceleration is exposing hidden costs. A Faros AI report shows that since early 2025, after widespread adoption of AI coding tools, code‑review quality has sharply declined: review comments are longer but many pull requests are merged without any human inspection, leading to more incidents and higher defect density per developer.

The speaker recounts a July 2025 experiment where a team fully adopted the lights‑off mode. Within months the system suffered outages because agents produced obscure bugs that required developers to dig into three‑month‑old code they had never read, causing user frustration.

The root cause is that current large models cannot autonomously maintain and improve a codebase over long periods. Maintainability—changing a part of the code without breaking others—is compromised by “shotgun” modifications, a pattern identified by Martin Fowler as a code smell.

03 Why Models Never Learn "Good Design"

Understanding this issue requires looking at how coding models are trained. Claude Code achieved rapid commercial success by aligning its training directly with the Harness framework that will be shipped to users, unlike earlier tools such as Aider or CodeBuff that only offered read/write/search capabilities.

Large language models are next‑token predictors. Training an coding agent involves giving it a problem, generating many solution traces, scoring them on correctness (e.g., test pass), and using reinforcement learning to increase good behavior. Benchmarks like SWE‑bench Multilingual draw tasks from open‑source repos (Redis, Django, fastlane) and evaluate whether an agent can fix a bug without breaking other functionality.

These benchmarks only require tests to pass; they cannot penalize poor design or long‑term maintainability. Consequently, agents often add unnecessary try‑catch blocks or perform unsafe type casts simply to make tests green.

Validating maintainability is orders of magnitude harder than confirming that code runs. Bad architecture costs manifest over months or years, and reward signals for such long‑term effects are too weak for current reinforcement‑learning loops.

04 Benchmarks Evolve, Yet Remain Insufficient

The community is building more complex benchmarks to address this gap:

SWE Marathon (Abundant AI): tasks that consume tens of millions of tokens and span long time windows, such as cloning every feature of Microsoft Excel with complex reward channels.

DeepSuite (MindStudio et al.): selects large tasks from open‑source repositories that have never been built, thus never appearing in training data.

FrontierCode (Cognition): multi‑step tasks that penalize agents when generated tests pass on pre‑patch code, using a judge model to enforce code‑quality rules.

These efforts improve agents' ability to judge code quality, but limitations persist: if the judge model itself does not understand “good code,” it cannot drive improvement. Adding more review agents or tokens raises the baseline but cannot overcome the knowledge ceiling of the reinforcement‑learning stage. Human code reading remains essential.

05 Turn the Lights On: Bring Planning Back to the Stage

A pragmatic strategy is to reopen code review and use upfront planning to drastically reduce review effort and duration. The proposed four‑step practice is:

Product Review : clarify the problem, expected behavior, and even sketch model diagrams. Small tickets can go straight to the agent; larger features require this step.

System Architecture Design : define component contracts, data models, and constraints; produce a high‑level blueprint.

Program Design (the most underestimated layer) : dive into types, method signatures, layout, and call‑stack level details; these fine‑grained decisions drive maintainability in large codebases.

Vertical Slicing and Implementation Order : decide cross‑repo build order and how to verify correctness at each stage. Agents tend to produce horizontal, unlayered plans; humans must reshape them into vertically sliced, step‑by‑step verifiable pieces.

Thirty minutes of upfront alignment can save hours of review, making line‑by‑line reading feasible. Moreover, the real pain point is not the number of PRs but the proportion of “bad PRs.” A good PR is pleasant to review; a PR requiring >20% rework burdens both reviewer and author. AI‑assisted planning can provide the alignment information up front, accelerating both review and coding while keeping humans in the loop.

06 Conclusion: Engineer Constraints and Levers

Although the vision of never reading code is alluring, engineers must work within constraints: models excel at some tasks but fall short on others. The solution is not to abandon AI but to identify leverage points—using AI for speed while humans retain responsibility for design and quality.

The speaker’s team is building Human Layer , an AI‑collaborative IDE and platform that supplies the building blocks of a software factory and offers better quality validators, aiming to provide a structured planning, collaboration, and review experience similar to how Figma transformed design collaboration.

In the foreseeable future, the most practical victories will belong to those who can harness AI speed while actively shouldering design responsibility.