Readout Bottleneck: LLMs Know Answers But Fail to Output Them

When large language models answer logic questions incorrectly, we typically assume they lack the reasoning ability. However, a new study from Peking University and Yixin AI Lab, accepted at EMNLP 2026 (15.4% acceptance rate), challenges this assumption by introducing the concept of a Readout Bottleneck : models may internally represent the correct answer in their hidden states, but systematic biases in the output layer prevent that knowledge from being expressed in the final token probabilities.

1. The Confusion: Wrong Prediction ≠ Missing Reasoning

Standard multiple-choice evaluation computes logit scores for each candidate answer, picks the highest, and reports accuracy. The authors argue this conflates two distinct failure modes: (a) the model never derived the correct reasoning, and (b) the model derived it correctly but the output mapping distorted the scores. To disentangle these, they designed a three-stage diagnostic pipeline probing different "readout" nodes across difficulty gradients: in-distribution (id), deeper reasoning chains (depth), and lexical out-of-distribution (lexical-OOD) splits.

Key question: When a model answers a logic question incorrectly, is it because the model never figured it out, or because it figured it out but failed to express the correct answer?

2. Diagnostic Results: Hidden States Know, Output Layer Collapses

Using linear probes on hidden states at the prompt-end and after the "Answer:" token, the authors found near-perfect decoding accuracy even on the hardest lexical-OOD split. For Qwen3.5-9B, probe accuracy reached 0.830 . However, when the same information passed through the model's vocabulary projection (same-position label logits), accuracy dropped to 0.574 . Aggregating full-string log-probabilities for each candidate caused a further catastrophic collapse to 0.333 — equivalent to random guessing among three options.

Strikingly, on 1,000 test questions Qwen3.5-9B assigned the same label ( unknown) to 999 questions , revealing a massive global bias that flattened instance-level logical differences. The phenomenon persisted in the base model without instruction tuning (Qwen3.5-9B-Base: probe 87.3%, sequence scoring 36.1%).

3. Minimal Correction: Two Global Scalars Recover Reasoning

If the raw sequence scores are merely a globally biased version of the true logic signal, the relative ordering among candidates for each question might still be intact. The authors decompose the candidate score into a sample-dependent logic signal and a label-dependent structural bias. They then apply an extremely constrained intervention: add a single global scalar offset per candidate class (only 2 free parameters for three-class tasks), fitted on unlabeled context scores to match a uniform prior, then frozen and evaluated on a held-out test set.

This intervention cannot learn new task mappings (only 2 parameters) and cannot change within-class relative ordering (global constant shift). Yet it produced dramatic, consistent gains across four reasoning benchmarks:

Synthetic logic tasks : Qwen3.5-4B/9B accuracy jumped from 33.3% (random) to 57.0% / 60.2% , rescuing >230 questions.

ProofWriter deductive reasoning : from 33.3% to 65.3% / 67.8% , rescuing >320 questions.

FOLIO & ANLI : 9–29 percentage-point gains on complex first-order logic and adversarial NLI.

Cross-model validation : OLMo-2-1B 0.362→0.566, Llama-3.1-8B 0.333→0.477 (permutation tests p < 0.001).

Sample efficiency confirmed the method merely reweights existing signal: 25 unlabeled samples sufficed to recover most accuracy, saturating by 1,000 samples — ruling out covert relearning.

4. Rigorous Controls: Ruling Out Shortcuts and Luck

Two stringent controls eliminate alternative explanations:

TF-IDF-missed hard slice : Removing all questions solvable by surface lexical overlap (via a TF-IDF classifier) left only semantically demanding items. Calibrated accuracy remained high ( 62.2% / 65.1% for 4B/9B), proving recovery relies on genuine semantic deduction.

Count-preserving permutation null : Randomly permuting the calibrated predictions while preserving class counts yields a null baseline. True calibrated predictions significantly exceed this baseline (gap +0.20 to +0.30, p < 0.001), confirming the method corrects instance-level errors, not just global label balance.

5. Limitations and Boundary Conditions

No internal representation : Pythia models (410M/12B) failed permutation tests, indicating they never formed the correct internal representation.

Ceiling effect : When native accuracy is already high, little latent signal remains to recover.

Uncontrolled label imbalance : The method assumes a knowable prior (e.g., uniform); extreme long-tail or unknown distributions remain an open challenge.

6. Implications: Rethinking Benchmarks and Chain-of-Thought

The study establishes that internal reasoning capacity and external expression pathway are distinct . Compressing multi-step logic into a single classification token forces the static unembedding layer to bear the full readout burden, where geometric misalignment and accumulated scoring bias create the bottleneck. This offers a mechanistic explanation for why Chain-of-Thought helps: by emitting intermediate tokens, the model distributes the readout load across the sequence trajectory, bypassing the single-step bottleneck.

For practitioners, the message is clear: do not equate near-random benchmark scores with absence of reasoning . Probe the hidden states first — the model may have solved the problem on its "scratchpad" but stumbled when filling the answer sheet.

Paper : "Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores" by Qiyao Yan, Chenpeng Wang, Liangming Pan (Peking University & Yixin AI Lab). arXiv:2608.31068 (https://arxiv.org/abs/2608.31068)