Software Engineering's Core Laws vs. LLM Limits: The Agent Boundary Question
The article examines software engineering's fundamental complexity laws, large language model limitations including context constraints and hallucination, Rice's Theorem implications, and questions the boundaries of AI agents in solving real software problems despite improved memory, tools, and context organization.
Software Engineering's First Law: Complexity
The article opens with the First Law of Software Engineering: the Law of Complexity . Essential complexity is inherent; accidental complexity cannot be eliminated.
Large Model First Law: Knowledge Compression and Probabilistic Uncertainty
The First Law of Large Models states that they are a form of knowledge compression characterized by probabilistic uncertainty.
Inherent Limitations of Large Models in Software Engineering
Context limits – fixed context window size.
Probabilistic uncertainty and hallucination – a p% hallucination problem.
Lack of tacit knowledge – the implicit, experience-based knowledge that engineers hold.
Lack of decision knowledge – the ability to make design trade-offs.
Linear understanding limitation – models process tokens sequentially, while software design is a three-dimensional thinking process.
The Software Engineer's Design Process
When designing software, engineers work within the limits of human working memory. The process is three-dimensional: abstract thinking is the foundation, structural construction is the method, constraints and trade-offs are the wisdom . Code is merely the final linear representation; the high-level mental model of the software resides in the designer's mind. This concept is explored further in the referenced article "Constructing Software's Mental Representation" .
Rice's Theorem and Its Implications
The article cites Rice's Theorem (referenced in "Rice's Theorem Tells You Why" ) to underscore fundamental undecidability results that affect what any automated system — including LLMs — can guarantee about program properties.
Agent and LLM Boundaries in Software Work
Given these essential limitations, the author asks: what is the true boundary of agents and LLMs in software work? Even if agent memory management is perfected (recalling programming history), tools are excellent (code checking, testing), and context organization is superior (code location, spec-driven approaches), which real-world software problems can actually be solved?
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
Thought Artisan
I think, therefore I am; recording insights from daily life and technology.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.