Why Do Language Models Hallucinate? Insights from OpenAI’s New Study

Many users have experienced large language models giving a perfectly confident yet completely wrong answer to obscure questions, such as asking for Adam Tauman Kalai’s birthday and receiving three different incorrect dates.

OpenAI’s recent paper systematically reveals that the root cause of hallucination is that standard training and evaluation reward guessing while providing no incentive for the model to honestly express uncertainty.

Root Causes of Hallucination

1. Statistical Inevitability

Generation can be framed as a binary classification "Is‑It‑Valid?"; any error of the classifier leads to a generation error (Theorem 1).

2. Data Scarcity

Facts that appear only once in the training data (singletons) are inevitably memorized incorrectly; the error rate is at least the proportion of singletons (Theorem 2).

3. Model Expressiveness Limits

If the model family cannot learn the underlying pattern, the hallucination rate has a non‑zero lower bound (Theorem 3).

Post‑training “Exam” Mechanism Amplifies Hallucination

After reinforcement learning from human feedback (RLHF), a well‑calibrated pre‑training model becomes noticeably over‑confident (see Figure 2). Binary scoring (1 point for a correct answer, 0 for a wrong answer) forces the model to avoid a blank response, encouraging it to guess.

Key findings from the two stages:

Pre‑training: Even with 100 % correct data, the density‑estimation objective pushes the model to generate errors. Analogy: a teacher only teaches correct answers, but the final exam forces students to fill in everything.

Post‑training: Binary scoring (1 point for correct, 0 for wrong) makes the model reluctant to answer "I don’t know", even when uncertain.

Proposed Solutions: Make "I don’t know" an Option

1. Explicit Confidence Threshold

"Only answer when confidence > t; wrong answer penalized t/(1‑t), I don’t know gets 0."

2. Make Abstention Optimal

When the model’s true confidence is below t, saying "I don’t know" yields the highest expected score, while lying incurs a penalty.

Misconceptions Clarified

Misconception 1: Improving accuracy alone eliminates hallucination. In reality, many real‑world questions are unanswerable, so perfect accuracy is impossible.

Misconception 2: Hallucination is inevitable. Models can choose not to answer when uncertain.

Misconception 3: Only larger models can reduce hallucination. Smaller models may better recognize their own limitations.

Misconception 4: Hallucination is a mysterious defect. It stems from statistical mechanisms and reward structures.

Misconception 5: A good hallucination metric alone solves the problem. Existing metrics often penalize cautious answers and reward guesses, so they need redesign.

OpenAI hopes this statistical perspective clarifies the nature of hallucination and corrects common misunderstandings.