Why LLMs Miss Simple Addition: Geometric Mechanism Behind Arithmetic Errors

Large language models (LLMs) excel at complex reasoning but frequently err on basic multi‑digit addition. Probing analyses have shown that even when the final answer is wrong, the hidden states often retain the correct result, suggesting that the failure lies in the read‑out stage.

Probe Versatility

The authors train lightweight probes on the residual stream of Qwen3‑4B while it adds three 10‑digit numbers (10,000 examples). Probes can decode multiple arithmetic variables from the same hidden state, including ground‑truth digit, model output, correctness, raw sum, input carry, and carry potential, demonstrating “probe versatility”.

Iso‑Raw‑Sum Trajectory (IRST)

Using UMAP, the final‑layer hidden states are visualized with digit token embeddings as anchors. The states form clear “digit basins” around the ten digits. Within each basin, representations further split by input‑carry status, creating a hierarchical geometry.

Samples with the same raw sum but different carry states lie along continuous lines that cross adjacent basins. This continuous line is defined as an Iso‑Raw‑Sum Trajectory (IRST): a set of internal representations sharing the same raw sum while differing in carry potential.

Noisy Quantization Model

The paper introduces the concept of Carry Potential, a continuous real‑valued signal representing accumulated lower‑bit sums. Discrete carries are obtained by quantizing this signal. Errors occur when the carry potential is near an integer boundary; small noise can push the quantized result across the boundary, causing ±1 mistakes. This phenomenon is termed “geometric slippage”. Experimental curves of error rate versus carry potential support the model.

Dual‑Stream Consistency Check

To exploit internal signals, the authors propose a decoding‑time correction method that extracts both the local raw sum and the global carry potential from the same hidden state. If the model’s output is inconsistent with these signals, a new candidate is generated by recombining the decoded values. This dual‑stream approach achieves the highest token‑level accuracy compared to the raw output and several baselines.

Conclusion and Outlook

The study reframes LLM arithmetic as a geometric problem, showing that probes read out information because it is geometrically separable. Errors stem from noisy quantization rather than missing information. Future work should focus on stabilizing the mapping from continuous representations to discrete tokens and leveraging the self‑monitoring signals already present in hidden states.