Why AI Coding Agents Fail to Deliver Sustainable Software: The Lights‑Off Factory Dilemma

1. From Traditional Factories to Lights‑Off Factories

The term "software factory" dates back to a 1968 NATO meeting. Historically, product managers and engineers defined work items in trackers like Jira, developers wrote code, and humans performed reviews, automated checks, and occasional manual testing before deployment.

Since around 2025, most tech companies claim to run "coding‑agent factories" where agents generate over 75% of code, cutting build times from hours to minutes. However, human code review still takes hours, prompting the introduction of agent‑driven review and regression testing to accelerate the bottleneck.

The "lights‑off" vision eliminates the need for developers to read code: monitoring events and user feedback feed directly into an autonomous pipeline that creates pull requests overnight, leaving humans only to queue new requirements.

2. The Maintainability Nightmare

Industry reports (e.g., Faros AI) show a sharp decline in review quality after widespread AI‑code adoption: review comments increase in length, many pull requests are merged without any human inspection, and incident rates per developer rise dramatically.

The speaker recounts a July 2025 case where a team fully embraced the lights‑off mode, only to encounter unrecoverable agent errors within months, forcing engineers to dig into three‑month‑old codebases to fix outages.

3. Why Models Can’t Learn “Good Design”

Large language models are trained as next‑token predictors. When fine‑tuned for coding, they generate many solution traces, score them on test‑pass criteria, and use reinforcement learning to favor successful traces. Benchmarks such as SWE‑bench Multilingual evaluate agents on short‑term tasks (e.g., fixing a bug without breaking other functionality) but do not penalize poor architectural choices.

Consequently, agents often add unnecessary try‑catch blocks or perform unsafe type casts simply to make tests pass, without improving the underlying design.

4. Measuring Maintainability Is Harder Than Passing Tests

Long‑term maintainability defects manifest over months or years, making reward signals for reinforcement learning almost nonexistent. This explains why agents excel at short‑cycle tasks yet stagnate on code‑base health.

5. Evolving Benchmarks Still Fall Short

The community is building richer benchmarks:

SWE Marathon (Abundant AI) : massive token‑consuming tasks such as fully cloning Microsoft Excel, with complex reward pathways.

DeepSuite (MindStudio) : selects large, never‑built open‑source projects to avoid training‑set leakage.

FrontierCode (Cognition) : multi‑step tasks that penalize agents when generated tests pass on unmodified code, using a judge model to enforce quality rules.

These efforts improve agents’ ability to assess code quality, yet limitations remain because the judge models themselves must already understand “good code.” Adding more review agents or tokens raises the baseline but cannot surpass the knowledge ceiling of reinforcement learning.

6. Practical Steps: Re‑Opening the Lights

Instead of abandoning human involvement, the article proposes a four‑step workflow to reduce review effort while preserving code quality:

Product Review : Clarify the problem, expected behavior, and possibly provide model sketches. Small tasks can go straight to agents; larger features require this step.

System Architecture Design : Define component contracts, data models, and constraints, producing a high‑level blueprint.

Program Design (the most underestimated layer) : Dive into types, method signatures, layout, and call stacks—granular decisions that drive maintainability.

Vertical Slicing & Implementation Order : Determine cross‑repo build order and correctness checks at each stage. Humans reshape the agent’s horizontal plan into verifiable vertical slices.

Thirty minutes of upfront planning can save several hours of review, making line‑by‑line reading feasible again.

7. Human Layer – An IDE for AI Collaboration

The speaker’s team is building Human Layer , an IDE and platform that supplies the building blocks needed for a software factory and offers advanced quality validators, analogous to how Figma structures design collaboration for tools like Claude Code.

Conclusion

Engineers must acknowledge constraints: models excel in some areas but lag in others. The pragmatic path is to harness AI speed while retaining human responsibility for design and review, using structured planning and tooling to keep codebases healthy.