AI Won't Kill Software Engineering, But Cheap Code Generation Might Hollow It Out

Introduction: The Fields Medalists' Warning

On September 11, 2026, Terence Tao posted an open letter titled "Serious Misalignment of AI in Mathematics," co-signed by 25 Fields Medalists from Deligne (1978) to Deng Yu (2026). The letter does not oppose AI doing mathematics; it opposes rushing to announce results without writing proper papers, extracting new methods, or citing prior work. The key judgment worth posting on every software team's whiteboard: "Solving problems is merely a tool, a proxy metric for the primary goal—conceptual understanding and insight. Forgetting this in the AI world may cause the tool to turn against the primary goal."

Solving problems is merely a tool, a proxy metric for the primary goal—conceptual understanding and insight. Forgetting this in the AI world may cause the tool to turn against the primary goal.

Keras author François Chollet adds a harder-to-quantify observation: a pervasive mood among math students—"I don't want to be a mathematician anymore." AI may not functionally replace mathematicians, but could effectively cause near-extinction by crushing the younger generation's morale.

I. Software Engineering Was Born Because "Code That Runs" Was Not Enough

Software engineering did not emerge because programmers typed too slowly. The 1968 NATO conference became the discipline's starting point because the "software crisis" revealed a fact: people could write runnable programs, but increasingly failed to deliver systems on time, on budget, and with acceptable quality as scale grew.

Writing a program and building software well have never been the same thing. Brooks in No Silver Bullet distinguished essential from accidental difficulties: tools can remove expression and implementation obstacles, but conceptual complexity, real-world constraints, and continuous change do not disappear.

AI may indeed start touching some essential difficulties—this must be honestly acknowledged. But "the model completed the implementation" still does not imply "the engineering problem is solved." Take a refund interface: generating code takes minutes; the real difficulties are:

Will a duplicate request cause a duplicate refund?

If the payment channel times out, is the transaction failed or succeeded with lost response?

How to recover if the service crashes after funds are debited but before state is persisted?

Do old service version and new database schema maintain consistency?

Can audit logs reconstruct facts when disputes arise?

AI can answer these questions. The key is: they must first be identified, then codified as design constraints, verification evidence, and runtime mechanisms. A piece of code that runs does not prove this process occurred.

Understanding in software engineering is not an aesthetic preference; it is a risk management tool. Systems continuously evolve and fail. During evolution and failure, someone or some reliable mechanism must be able to take over. The entire legitimacy of understanding rests on this sentence.

Therefore, software engineering's true definition: it is the systematic answer to the responsibilities code assumes once it enters the real world.

II. The Real Misalignment: Treating Easy-to-Show Outputs as Hard-to-Measure Value

The open letter criticizes a familiar substitution: using problem-solving progress to observe understanding progress, eventually chasing only problem-solving scores. The software industry has its own long-standing proxy metrics:

Easy-to-show output: Lines of code generated → Real question: What value did the user gain?

Easy-to-show output: Tasks auto-completed → Real question: Did end-to-end delivery cost decrease?

Easy-to-show output: All tests pass → Real question: Are critical risks covered?

Easy-to-show output: Changes merged → Real question: Is the system still controllable?

Easy-to-show output: Benchmark scores → Real question: How does it behave in real environments under continuous change?

These metrics are useful. The problem: metrics provide evidence, they cannot replace goals.

We now have empirical data. In 2025, independent evaluator METR ran a randomized controlled trial: 16 experienced open-source developers completed 246 real tasks in their own familiar mature repositories. Beforehand, they expected AI to cut completion time by 24%. Afterward, they felt ~20% faster. Actual measured result: using AI tools increased task completion time by 19%. Time went into prompting, waiting, and reviewing AI output.

This study must be interpreted cautiously: small sample, early 2025 tools, senior developers working on their own large codebases—precisely where AI has least advantage. METR itself adjusted the experimental design in February 2026. Change the people, change the tasks, and the conclusion could reverse entirely.

But the gap it leaves is more valuable than the conclusion: perceived speedup +20%, actual slowdown -19%, a nearly 40-percentage-point cognitive bias. This is the most concrete form of "misalignment" in software: not that AI is useless, but that our ability to evaluate whether we are getting faster is degrading faster than we are actually getting faster.

From requirements to production, the path goes through clarification, design, implementation, review, integration, release, operation. Accelerating one link only moves the bottleneck downstream: implementation outpaces review, merge queues grow; merging outpaces verification, defects flow to production; releasing outpaces organizational absorption, maintenance costs rise. Time saved at the generation end often reappears as review, rework, and incidents at the other end of the lifecycle.

Deeper misalignment lies in incentive structures: tool vendors showcase automation, managers showcase efficiency gains, engineering teams bear long-term system consequences. The three parties are not inherently opposed, but only when lifecycle costs are accounted on the same spreadsheet can short-term incentives align with long-term value.

"90% of our code is written by AI" is merely a description of production mode. It is not a quality certificate, nor an efficiency report card. AI did not invent our obsession with speed; it merely made an old problem unavoidable: are we rewarding software getting better, or software getting more?

III. The More Hidden Burden in the AI Era: Cognitive Debt

Technical debt is familiar: choose an expedient solution today, pay higher modification cost later. AI-assisted development may amplify another burden— cognitive debt : the system has been delivered, but the organization has not built understanding and control capabilities matching its complexity and risk.

Technical debt is "we know we owe a debt here"; cognitive debt is "we don't even know where the debt is recorded."

Code may be well-written: clean structure, standard naming, thorough tests, detailed comments. Yet the team may still be unable to answer: Why was this consistency model chosen? Which business constraints must never be broken? What failure was that odd special branch guarding against? Which dependency upgrade would invalidate the design's premises?

An absurd state emerges: code is freshly generated, yet already resembles a legacy system with no one to take it over.

This is not AI's monopoly. Turnover, sloppy outsourcing, long-term lack of knowledge management all create cognitive debt. AI's special trait: implementation growth speed can far outpace organizational understanding growth speed.

Peter Naur in his 1985 Programming as Theory Building said: the product of programming is not just program text, but the programmer's "theory" of the problem, reality, and solution. The program can be perfectly preserved; the theory disappears with the person's departure—this is why many legacy systems have full source code yet no one dares touch them.

This explains a trap many overlook: AI-generated design documents cannot automatically prove design knowledge has been internalized. They may faithfully record trade-offs, or they may just be a fluent post-hoc rationalization of an already-made implementation—they read exactly the same.

Truly valuable design records answer only four questions: What alternatives were considered? Why were others rejected? What assumptions does this choice depend on? What changes would invalidate it?

Must emphasize: This does not require humans to understand every line of code. Modern software is built on compilers, databases, OSes, and cloud services—black boxes no one fully understands. We need understanding matched to risk : need not know database internals, but must know what its transaction isolation guarantees and does not; need not read every line of a component, but must know its interface contract, failure modes, and applicability boundaries.

Such understanding need not reside solely in human brains. It can be shared among AI, static analysis, contracts, and runtime guards. But understanding can be distributed, not skipped. Critical understanding should not wait until a 3 AM incident call to be demanded for the first time.

IV. Tests Pass, Formal Verification Passes—Still Not the End of Engineering

AI generates code, then generates tests, then explains why tests pass—is that a complete quality loop? Maybe, or maybe it's the same misunderstanding written three times . Implementation and verification, if built on the same wrong assumption, will perfectly match each other and jointly deviate from reality.

Software engineering distinguishes two questions: V&V

Verification: Did we build the system correctly according to spec?

Validation: Is the system we built the one actually needed?

A refund program can perfectly satisfy "cumulative refunds never exceed original payment" while completely missing account takeover, cross-currency settlement, and merchant-vs-platform liability division.

The Navier-Stokes controversy three days before the open letter provides a precise example: OpenAI's proof was formally verified in Lean—machine-checked without error—but controversy immediately shifted to the imposed smooth external force term, while the Clay Institute's standard formulation addresses the no-external-force case. The proof itself had no problem. The problem was the boundary judgment of "what the proof covers." And boundary judgment, the machine does not do for you.

This is the lesson the software industry most needs to remember: formal methods can provide extremely strong guarantees for specific properties on top of explicit assumptions and trusted foundations. They do not guarantee the spec is complete, nor that the formalized object is the one you actually want.

The proof didn't fail; the danger is we overestimated the proof's range. Software runs in open, changing, failure-prone environments, so trustworthiness must be built in layers: type systems and contracts constrain local behavior; testing and static analysis find defects; formal methods prove critical properties where needed; canary releases limit blast radius; monitoring, rollback, and runbooks catch residual risk.

This means software engineering need not merely "call for slowing down" like mathematics—it has the capacity to turn requirements into mechanisms:

High-risk operations require authorization independent of the generation process;

Acceptance criteria cannot be self-certified by the same pipeline that generated the implementation;

Changes traceable, impact containable, releases reversible;

Runtime continuous validation of key invariants, not just one-time CI checks.

These mechanisms can also heavily use AI. The point is not that all checks are manual, but whether evidence has sufficient independence.

The more autonomous AI becomes, the more engineering must specify: what it is allowed to change, what evidence proves safety, and how to reclaim control when things go wrong.

V. Deeper Than Headcount Changes: How Engineers Grow

"How many programmers will AI replace?" is an important question. But there's a longer-timescale question: If junior work is automated, where will future senior engineers come from?

Historically, a junior task produced two things simultaneously: a fixed bug, and a person who understands the system better . Engineers grow judgment by reading old code, locating problems, being rejected in reviews, participating in incident retrospectives: which requirements were actually unclear, which seemingly local change is actually dangerous, which abstractions withstand change, when not to add another service.

AI can deliver the first product, but does not guarantee the second follows.

If organizations only see junior roles as "cheap code capacity," the conclusion is almost automatic: code can be auto-generated, so no need to pay for these roles. But saving training costs and improving production efficiency are two different things . Automating the entire talent-development pipeline may be subsidizing today's efficiency reports with tomorrow's talent supply.

This does not mean preserving all drudgery. Mindless boilerplate doesn't build judgment; a newcomer struggling alone with an obscure error for five days may not be better than receiving one good mentoring session. What must be protected is not inefficiency, but necessary cognitive engagement .

AI can be an excellent mentor: explaining legacy code, constructing counterexamples, simulating failures, comparing design options, letting newcomers engage with full systems earlier. But training must maintain a sequence— judge first, then see suggestions, finally explain why you accept or reject .

Concretely, require newcomers to:

Write key constraints and failure conditions before generating implementation;

Given two working solutions, explain long-term trade-offs;

Design a test case that makes the model's program fail;

Modify, run, and recover the system under requirement changes and fault injection;

Formulate verifiable diagnostic hypotheses when no ready answer exists.

Returning to Chollet's quote. Math students say "I don't want to be a mathematician anymore"; software may hear "I don't want to be an engineer anymore." Morale is not a soft topic. A discipline can lose hearts while jobs remain—young people aren't laid off, they're discouraged.

They need to see that software engineering hasn't degenerated into "clicking confirm on model outputs." Identifying problems worth solving, mastering complex systems, creating better abstractions—these remain intellectually dignified work.

A discipline's continuity depends not only on how many products it delivers today, but on whether it's still cultivating people who can ask new questions tomorrow.

VI. When Code Gets Cheaper, Will Open Source Lose Control to Code Flood, and Public Code Assets Disappear?

A seemingly contradictory future is becoming imaginable: public code grows ever more abundant, yet truly dependable public software becomes harder to find.

Because open source's scarce resource was never code. It is maintainer attention , trustworthy contribution relationships , continuously accumulated engineering judgment .

1. Generating Code Is Free; Judging Whether Code Is Worth Merging Never Is

This is no longer hypothetical. curl maintainer Daniel Stenberg repeatedly expressed dissatisfaction with AI-generated security reports in 2024-2025: incoming vulnerability reports were well-formatted, used professional terminology, sounded confident—yet were fabricated. The team had to painstakingly verify each, because real and fake vulnerabilities became indistinguishable in text. In May 2025, he announced stricter AI disclosure and handling policies for HackerOne submissions—not because reports were too bad, but because they were too convincing, wasting the scarcest resource: attention.

The asymmetry: contributor clicks generate once; maintainer spends hours judging and communicating.

Low-quality contributions aren't AI's invention, and AI can help filter and review. But when input volume exceeds processing capacity, the community's rational response is to raise barriers, limit submissions, close channels. The irony: machines widened the contribution entrance, attention pressure narrowed the collaboration entrance. Genuine new contributors pay the price of this contraction together.

The 2024 xz-utils backdoor reminds us of another thing: when maintainers are exhausted, systemic risk enters from human relationships , not code syntax. The attacker spent two years gaining trust, taking over maintenance. The adversary never exploited insufficient code; they exploited not enough people willing to seriously review code .

2. Public Assets Won't Be Deleted; They'll Lose Maintenance, Provenance, and Discoverability

An unmaintained repo can still be cloned; code of unknown origin can still be copied; a three-year-unanswered issue still hangs publicly. But "accessible" and "safe to depend on" are two different things.

Public software's value lies not only in source files, but in these relationships:

Which version is worth using;

Which issues are known but unfixed;

Who can explain design boundaries;

Who publishes security fixes;

What rules govern contributions;

How newcomers become maintainers step by step.

Once these relationships break, code remains, but the effective value of the public asset shrinks drastically.

Another hidden erosion: when generation tools return only an answer without preserving sources, documentation entry points, and contribution paths, the user gets code but loses the road to that knowledge community . They don't know who to ask, nor that they could someday become the answerer.

License and training data issues require specific facts and legal judgments; labeling all AI output as infringement is neither accurate nor helpful. But maintaining provenance, fulfilling applicable license obligations, and paying for maintenance of critical dependencies are unambiguous engineering responsibilities.

3. Open Source Is Not Just Public Code; It's Public "Why"

Open source communities are humanity's largest distributed engineering school. People learn "how" from code, "why this way" from code reviews, "why not that way" from rejected PRs, "where past judgment failed" from incident retrospectives.

This exactly matches the open letter's heaviest judgment: if no one is willing to take up these ideas, develop them, integrate them into the canon, AI-conceived ideas will never truly come alive—that critical human transmission chain breaks.

Open source is the software industry's transmission chain.

Good news: this is not irreversible. AI can also improve documentation, explain historical designs, translate language barriers, help maintainers pick truly important signals from noise. Its potential to lower participation barriers is real.

But governance must keep up: require submitters to state real problems, verification methods, and ongoing maintenance intent; rate-limit and mandate disclosure for high-frequency automated submissions; fully preserve design discussions and contribution attribution; let the companies benefiting most pay for maintenance of critical projects, not just sponsor new-feature launch events.

Open source won't die merely because there's too much code. The real cause of death: code grows without limit, while people willing to understand, filter, maintain, and transmit it continuously decrease.

VII. Software Engineering Must Redefine Success, Not Defend Old Work Habits

Must block an easy-to-slide-into conclusion: human-written code is inherently reliable, AI-written code inherently dangerous. This does not hold. Humans equally ignore edge cases, create vulnerabilities, leave systems no one can explain. And AI is concretely helping us chew legacy code, expand test coverage, construct counterexamples, lower the cost of using formal methods.

What needs changing is not the tool allow-list, but the definition of success .

1. For Enterprises: From Output Metrics to Lifecycle Outcomes

Evaluating AI tools cannot just look at generation speed. Must examine end-to-end delivery time, review and rework costs, production defects, incident recovery time, and what the end user actually received.

The METR "perceived +20%, measured -19%" gap gives the first, plain action item: don't use surveys to measure efficiency. Self-perception is the most unreliable metric of all.

Delivery standards cannot stop at "feature complete"; they should be risk-tiered, including critical constraints, verification evidence, operational responsibility, and recovery plans.

One most easily overlooked: allow "writing less code" to count as achievement. Rejecting an unnecessary requirement, deleting a redundant dependency, merging two duplicate services—these may be more valuable than auto-generating ten thousand files.

When generation becomes nearly free, judging what should not be generated becomes the most expensive capability.

2. For Education: From Producing Answers to Forming Judgment

Both teach students to use AI and test whether they possess the foundation for independent judgment.

Assessment should expand from one-time answer submission to requirements clarification, solution comparison, code explanation, fault localization, and continuous evolution. Reference: AI Era: Reshaping Software Engineering Education .

Retaining necessary unassisted training is right, but the purpose is not to prove humans can repeat all machine work—it's to ensure when the machine is unreliable, humans can still recognize problems and take over .

Simultaneously, organizations must budget real resources, mentors, and opportunities for newcomer training, rather than assuming senior engineers will forever be hired from elsewhere. Reference: The Teacher's Joy May Come from a New Mission .

3. For Research: From Generation Capability to Trustworthy Evolution Capability

Code generation is only a small piece of software engineering research. More urgent to study:

How to turn vague intent into machine-checkable constraints?

How to limit an agent's modification permissions and blast radius?

How to keep verification evidence continuously valid across versions, not just pass once?

How to detect shared wrong assumptions among multiple agents?

How to long-term measure AI's impact on maintenance cost, collaboration quality, and talent growth?

How to distill generation results into reusable, explainable, transmissible knowledge?

These questions won't disappear because models get stronger. The more tasks models can handle, the more urgent they become.

What software engineering must build is not a proof that "machines will never learn engineering," but a methodology: letting increasingly capable machines enter engineering while keeping systems trustworthy, organizations controllable, and knowledge accumulating.

Conclusion: Code Can Be Auto-Generated; Trust Cannot Be Auto-Established

The open letter ends: "Mathematics is the first discipline forced to the front to explicitly write down its values. It will not be the last."

Software engineering is likely next. And our situation is slightly more awkward: mathematics at least debates "this is wrong"; the software industry has already put generation volume on page one of quarterly reports.

But we have advantages mathematics lacks. Facing AI companies, mathematicians can only publish open letters—they have no enforcement power, AI companies have no obligation to slow down. Software engineering is different: review rules, entry barriers, release processes, acceptance standards, hiring budgets—these are already in our own hands.

So this question was never a prediction waiting for technology to reveal the answer; it is a system design question .

It concretely exists in: a report that treats generated lines as productivity; a design review cancelled to meet a deadline; a decision "no new grads this year"; a maintainer's never-ending submission queue.

AI will not inevitably destroy software engineering just because it gets better at writing code. It may even help the discipline more fully deliver on its promise. Reference: Software Engineering 3.0 Three-Year Journey: From Concept to Large-Scale Practice .

But if we reduce engineering to generation, review to clicking confirm, training to tool tutorials, open source to free code sources—then the discipline will be hollowed out. Repositories still flourish, deliveries proceed as normal, but understanding and transmission no longer grow .

The real dividing line is not whether code is written by human or machine, but whether technology is used to enhance engineering capability or to exempt engineering responsibility.

So the question the whole industry must answer is not "how much code can AI write for us," but:

When generation gets cheaper and cheaper, are we still willing to pay for judgment, verification, maintenance, and transmission?

The answer is not in the next model's parameters. It's in today's budget spreadsheets, course syllabi, review rules, and every organizational choice.

Code can be auto-generated; trust cannot be auto-established.