Why Software Engineering Has Never Been Truly Engineered – How Large AI Models May Finally Deliver Real Engineering

1. Software engineering has never been truly engineered

Classic engineering fields (mechanical, chemical, electrical, automation) succeeded by converting low‑order human cognition into physical control loops powered by energy (e.g., steam‑engine centrifugal regulators, chemical plant thermostats, power‑grid dispatch systems). The output of these loops is deterministic given the same energy input, eliminating uncertainty.

Software development still requires high‑order cognition (abstraction, reasoning, creation) performed by humans, so the engineering loop is missing: "energy → high‑order intelligence".

2. Large language models (LLMs) introduce the missing link

LLMs consume compute (energy) and produce understanding, reasoning, and code generation – the first instance of "energy → high‑order intelligence". This makes a true engineering loop possible, but LLMs also bring new uncertainties: hallucinations, model drift, and lack of explainability.

3. Two mind‑sets for AI in software engineering

Efficiency view : AI makes people faster but keeps them in the same loop.

Attention view : AI frees human attention from repetitive tasks so humans can focus on scarce, high‑value decisions.

The article argues that the attention view is the true goal of an AI‑centric software engineering paradigm.

4. Why the current "human‑center + AI‑assist" (Copilot) model fails

In the Copilot workflow humans remain the primary decision makers; AI only suggests code. The feedback loop looks like:

Human uncertainty → AI training data → AI output (statistically "looks correct") → Human review (still using original standards) → Uncertainty cycles back.

Consequences:

Productivity gains of 20‑40% in coding speed do not translate into lower bug rates.

The model learns the organization’s standards only weakly because the feedback is unstructured.

5. Closed‑loop, node‑first migration strategy

Instead of gradually replacing individual nodes, build a tiny end‑to‑end AI‑controlled loop on a well‑defined sub‑domain and then expand.

Step 1: Build a tiny closed loop (requirement → design → code → test → deploy) on a narrow domain.
Step 2: Iterate, accumulating deviation cases.
Step 3: Expand the loop’s boundary.
Step 4: Eventually cover the entire software pipeline.

The "code + test" node is the best starting point because verification mechanisms (compilation, unit tests, contracts) are strongest there.

3️⃣ Deterministic judges for each engineering node

Code generation : compiler + unit tests + type system – "code runs and tests pass" is an objective fact.

Contract validation : contract checker – "output conforms to proto specification" is objective.

Deployment : gray‑scale monitoring metrics – "no runtime errors" is objective.

Design : performance simulator – "performance benchmarks met" is objective.

Only nodes that have an objective, deterministic judge can be closed‑looped first.

6. Hierarchical control and a coordination bus

AI‑driven organizations should keep the classic "divide‑and‑conquer" hierarchy (product domains, engineering teams) and add a coordination bus that orchestrates AI‑driven production lines. The bus is not a technical message bus; it is an organizational middleware that distributes tasks, encodes domain knowledge, and arbitrates cross‑layer deviations.

7. The hardest barrier: scenario‑driven implicit knowledge distillation

Experts often cannot articulate ~90 % of their tacit knowledge. Only concrete, problem‑driven interactions surface it. Four challenges:

Experts don’t know what they know.

Formulating the "right" questions requires domain expertise.

Distilled knowledge must be stored in a scenario‑based format, not as plain statements.

Existing knowledge bases (Wiki, RAG) capture explicit facts but miss scenario nuance.

Practical steps:

Force the AI to ask "why was I corrected?" after each human edit.

Continuously observe expert work to generate targeted queries.

Store the result in a scenario knowledge store with fields such as context, decision, rationale, exceptions, and provenance.

Example entry:

scenario_id: SCN‑001
context:
  - domain: cross‑border payments
  - trigger: refund > single‑limit
  - upstream_state: settled payment, merchant balance insufficient
decision: split‑refund
rationale: improve user experience, control risk
exceptions: blacklist → direct reject
provenance: distilled from expert X on 2026‑05‑23

8. New high‑value roles

AI Production‑Line Architect : designs end‑to‑end AI‑driven pipelines.

Cognitive SOP Engineer : codifies domain procedures into repeatable AI workflows.

Deviation Detection Engineer : builds automatic verification and counter‑example generators.

AI Production‑Line Optimizer : tunes prompts, knowledge bases, and model selection.

Cognitive Boundary Guard : handles cases where AI cannot self‑correct.

The most scarce position will be the Production‑Line Designer , who must combine deep business insight, control‑theory expertise, and the ability to extract tacit knowledge.

9. Strategic take‑aways

Shift from "human‑center + AI‑assist" to "AI‑center + human‑assist".

Make deterministic judges (compilation, testing, contracts) mandatory for every AI output.

Adopt a closed‑loop, node‑first rollout, starting with the highly formalizable code‑test node.

Build a coordination bus that mirrors the classic divide‑and‑conquer hierarchy.

Invest heavily in scenario‑driven knowledge distillation – the toughest but decisive barrier.

Prepare talent pipelines for roles focused on production‑line design and boundary guarding rather than manual coding.

10. Comparison: Copilot vs AI‑centered architecture

Feedback frequency : Copilot – low; AI‑center – high (automatic verification loops).

Feedback density : Copilot – 1 bit (accept/reject); AI‑center – rich (verification reports, deviation rules).

Feedback target : Copilot – trains generic models for the vendor; AI‑center – builds a team‑ and domain‑specific rule base.

Improvement driver : Copilot – AI vendor; AI‑center – the organization’s own production line.

11. Node formalization suitability

Each pipeline node can be evaluated on formalization degree and verification automation:

Requirement analysis : low formalization, low automated verification – difficult to close.

System design : medium formalization, medium verification – partially feasible.

Code + test : high formalization, extremely high verification (compiler + tests) – ideal first target.

Release & ops : high formalization (monitoring metrics), high verification – suitable after code‑test.

12. Six‑stage evolution of the AI production line

Stage 1: 70 % of a single domain’s requirements implemented with 0 % human‑written code (AI writes most code, humans still review).

Stage 2: 90 % of a single domain’s requirements implemented with 0 % human‑written code (most advanced teams).

Stage 3 ★: 70 % of a single domain’s requirements achieved with 0 % human intervention – the quality jump from "probabilistic correct" to "engineered reliable".

Stage 4: 80‑90 % domain coverage, AI runs the whole pipeline autonomously.

Stage 5: Build a coordination bus that connects multiple autonomous units into a full production line.

Stage 6: Replicate the proven "code + test" pattern to other nodes (release, problem‑location, system design, finally requirement analysis) – each subsequent wave is faster because the method is already codified.

13. Scenario‑driven knowledge distillation – the decisive bottleneck

Why it is extremely hard:

Experts often cannot articulate their tacit knowledge; only concrete problems surface it.

To extract it, AI must ask the "right" questions, but asking the right questions requires domain knowledge that the AI lacks – a dead‑lock that must be broken by proactive human observation.

The distilled knowledge must be stored in a scenario‑based format (context + decision + rationale + exceptions), which current RAG or wiki systems do not support.

Three practical avenues:

Make AI ask "why was I corrected?" after each edit, then store the hypothesis and expert confirmation as a scenario rule.

Observe experts in real time , let AI flag anomalies (e.g., unexpected pauses, divergent actions) and generate targeted questions.

Develop a scenario‑oriented knowledge store with fields for context, decision, rationale, exceptions, and provenance (see the example above).

14. The future human role – from "human compiler" to "production‑line designer"

Historically many programmers acted as "human compilers", translating vague business intent into precise machine code. With AI taking over that translation, humans will move to higher‑level responsibilities:

Define value functions and strategic trade‑offs.

Design and maintain the AI production‑line architecture.

Detect and resolve deviations that AI cannot self‑correct.

The most scarce role will be the Production‑Line Designer , who must possess deep business understanding, control‑theory knowledge, and expertise in extracting tacit knowledge.

15. Summary of the full reasoning chain

Software engineering lacked an energy‑to‑high‑order‑intelligence loop. LLMs provide that loop but introduce model uncertainty. To truly engineer software we must:

Adopt an AI‑centered paradigm (AI drives the loop, humans assist).

Enforce deterministic judges at every AI output point.

Start with a closed‑loop on the most formalizable node (code + test).

Iteratively expand the loop, building a coordination bus that respects the classic divide‑and‑conquer hierarchy.

Overcome the hardest barrier – scenario‑driven implicit knowledge distillation – to give AI the domain‑specific rules it lacks.

Transition human talent from manual coding to production‑line design, deviation detection, and boundary guarding.

Executing this roadmap will turn software engineering into a discipline comparable to classic engineering, with deterministic, energy‑driven processes and a new generation of high‑value roles.