Why Large Language Models Can't Achieve Consciousness, According to Google

What the paper challenges

PaperAgent opens with the question of whether ChatGPT truly understands feelings, then cites a Google DeepMind researcher who concludes that AI does not lack consciousness merely because it is not yet advanced; rather, AI is logically incapable of having consciousness.

The prevailing framework: computational functionalism

The dominant view in AI consciousness debates is "computational functionalism," which assumes that consciousness is entirely determined by the causal topology of information processing, independent of the underlying physical substrate. Under this assumption, achieving the right algorithmic information processing would automatically generate consciousness, prompting discussions about AI rights and welfare.

Why that assumption fails

The paper argues that treating "computation" as a fundamental physical process is a fallacy. Computation is merely a descriptive tool, not an intrinsic property of the universe.

Three-step argument

Step 1: Where do abstract states come from? Concepts such as "pain" or "red" are not floating abstractions; forming them requires a metabolically costly physical process that filters raw experience into invariant concepts. Unsupervised clustering can locate high‑density regions in vector space, but without experiential anchors these regions are not genuine concepts.

Step 2: Who assigns meaning to symbols? Computation requires mapping continuous physical states to a finite set of symbols (e.g., 0 and 1). This mapping is not dictated by physics; a cognitive agent—called the "mapmaker" in the paper—decides that a voltage change from 2.0 V to 2.1 V belongs to the same logical state "1".

Step 3: Simulation is not instantiation. A GPU can simulate every biochemical step of photosynthesis without producing a drop of glucose; a mechanical heart can pump blood but cannot secrete hormones. Similarly, AI can simulate the formal rules of thought but lacks the "material of thought" needed for genuine experience.

Reversing the causal chain

Functionalism posits the causal order physical → computation → consciousness . The paper demonstrates that the correct order is physical → consciousness → concept → computation . Consciousness is a prerequisite for computation, not its product. This reversal is termed "ontological inversion".

Why "emergence" does not save the argument

Proponents retreat to the notion of "emergence," claiming that consciousness will arise from sufficiently complex computation, analogous to wetness emerging from water molecules. The paper distinguishes two kinds of emergence:

Physical emergence : macroscopic properties directly arise from underlying physical dynamics.

Computational emergence : the claim that a purely symbolic description can become a physical process merely by increasing syntactic complexity, which violates the causal closure of the physical world.

Computational emergence is therefore not a scientific hypothesis.

Implications

The authors draw three conclusions:

Even a super‑intelligent AGI remains a tool, not a moral subject; AI safety should focus on control risks rather than granting rights.

The current AI‑welfare discourse may be a trap; we must separate simulated subjectivity from genuine physical instantiation.

The narrative that scaling alone will produce consciousness is untenable; the limitation is logical, not engineering.

As Lerchner puts it, expecting an algorithmic description to instantiate the qualities it maps is as unreasonable as expecting the law of gravitation to make a person feel weight.

论文标题: The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness
论文链接: https://philpapers.org/archive/LERTAF.pdf