The Hidden Flaws of AI‑Driven “Lights‑Off” Software Factories

From Traditional Factories to "Lights‑Off" Software Factories

The term "software factory" dates back to a 1968 NATO conference. Before 2022, the workflow was simple: product managers and engineers defined requirements in trackers (Jira, Linear), developers fetched tasks, wrote code, performed manual review, automated checks, and optional manual testing before deployment. Planning activities such as architecture reviews and sprint planning were already used to reduce rework.

Around 2025, many tech companies claimed to have built "coding‑agent factories" where agents generate over 75% of code, reducing build time from hours to minutes. However, human code review still took hours or days, prompting the introduction of agentic review and regression testing to accelerate the bottleneck.

The "lights‑off" vision eliminates human code reading entirely: monitoring events and user feedback flow directly into the agent pipeline, which automatically creates fix pull requests. Humans only add more items to the backlog while agents handle implementation.

Maintainability Nightmare: Ignored Long‑Term Costs

Rapid adoption of AI coding agents has exposed serious issues. A Faros AI report (since early 2025) shows a sharp decline in code‑review quality: review comments increase in length, but many pull requests are merged without any human inspection, leading to more incidents and defects per developer.

The speaker shared a personal case: after fully adopting the lights‑off mode in July 2025, their team encountered unrecoverable agent errors within months. Developers were forced to dig into three‑month‑old codebases to understand "rotted" code, while services were down and users angry.

The root cause is that current large language models cannot maintain or improve code‑base quality over long periods. They frequently introduce "shotgun" modifications—what Martin Fowler calls "shotgun surgery"—that break unrelated parts of the system.

Why Models Fail to Learn "Good Design"

Training an coding agent involves feeding it problems, generating solution trajectories, scoring them on correctness (e.g., test pass), and applying reinforcement learning to increase good behavior. Benchmarks like SWE‑bench Multilingual draw ~15‑minute tasks from open‑source projects (Redis, Django, fastlane) and evaluate whether the agent fixes the issue without breaking other functionality.

These benchmarks only require tests to pass; they do not penalize poor design or maintainability degradation. Consequently, agents add unnecessary try‑catch blocks or perform unsafe type casts simply to make tests green.

Maintaining code quality is orders of magnitude harder than passing tests. Faults introduced months earlier may only surface later, making reward signals too weak for the model to learn long‑term health.

Benchmarks Are Evolving—but Not Enough

The community is building richer benchmarks to address this gap:

SWE Marathon (Abundant AI): tasks consuming tens of millions of tokens over long horizons, such as fully cloning Microsoft Excel.

DeepSuite (MindStudio et al.): selects large, never‑built open‑source repositories to avoid training‑set leakage.

FrontierCode (Cognition): multi‑step tasks with judges that penalize passing tests on unchanged buggy code and enforce code‑quality rules.

These efforts improve agents' ability to assess code quality, but limitations remain: if the judge model itself does not truly understand "good code," it cannot enforce it. Adding more review agents or tokens raises the baseline but cannot overcome the knowledge ceiling of reinforcement learning.

Practical Steps: Turn the Lights Back On

Given the constraints, a pragmatic strategy is to reopen code review and use upfront planning to drastically cut review effort. The proposed four‑step workflow is:

Product Review : Clarify the problem, desired behavior, and optionally produce model sketches. Small tasks can go directly to agents; larger features require this step.

System Architecture Design : Define component contracts, data models, and constraints; create a high‑level blueprint.

Program Design (the most underestimated layer) : Dive into types, method signatures, code layout, and call‑stack details—crucial for maintainability in complex codebases.

Vertical Slicing & Implementation Order : Determine cross‑repo build order and correctness checks at each stage. Humans must reshape agents' horizontal plans into verifiable vertical slices.

Thirty minutes of upfront alignment can save several hours of review, making line‑by‑line reading feasible again. Good pull requests are pleasant to review; bad ones impose heavy cognitive load, especially when >20% of changes require rework. AI‑assisted planning supplies the alignment information, letting agents focus on building while humans retain full code visibility.

Conclusion: Engineer Constraints and Levers

Although the dream of never reading code is alluring, engineers must work within constraints: models excel at some tasks but fall short on others. The path forward is not abandonment but leveraging AI speed while shouldering design responsibility—using tools like the "Human Layer" IDE to provide structured planning, collaboration, and quality verification.