Can Code Really Boost Large‑Model Reasoning? A Re‑Examination of New Experiments

Clarifying What “Code” Means

Many discussions conflate any text containing code snippets with "pure code". The authors distinguish Code (executable functions, scripts, algorithm implementations) from Code‑NL (mixed code, natural‑language explanations, formulas, and step‑by‑step reasoning found in notebooks, tutorials, and Q&A).

How the Study Was Conducted

The team built a ~10 T‑token pre‑training corpus covering Web, Code, Code‑NL, Math, Wikipedia, Books, and multilingual data, and trained Mixture‑of‑Experts (MoE) models under different data‑mix configurations. Instead of a simple "with code / without code" comparison, they kept the total token budget constant and replaced one data type with proportionally more of the others, mimicking realistic pre‑training trade‑offs.

Finding 1: Pure Code Boosts Programming but Not Complex Reasoning

Removing Code reduces programming ability, confirming its importance for coding tasks. However, on complex mathematical reasoning benchmarks (Minerva‑Math, OlympiadBench, MATH, CollegeMath, MathBench) the full‑data model with Code performs 14.38 % lower on average than the model without Code, while simpler tasks (GSM8K, CMath) are only mildly affected.

This phenomenon is termed negative coupling : adding one data type improves its own domain but can crowd out learning signals needed for another domain.

Finding 2: Math and Code Are Not Simple Complements

Adding Math data benefits algorithmic coding tasks (CodeForces + 37.11 %, LiveCodeBench + 11.26 %) but harms other benchmarks (CruxEval ‑ 17.30 %, MBPP ‑ 6.12 %, HellaSwag ‑ 2.94 %). Thus, math data provides specific symbolic‑reasoning signals that help certain code‑style tasks but can interfere with tasks that rely heavily on program execution semantics.

The Real Takeaway: Structured Reasoning Scaffolds

To isolate the beneficial signal, the authors define cognitive scaffolds —structured reasoning samples that contain clear intermediate steps, sub‑goals, and explicit symbolic operations. Using a lightweight FastText classifier trained on 400 k samples, they filtered 188 678 validation samples achieving 96.96 % accuracy, 99.98 % positive precision, and 96.65 % positive recall.

Replacing a portion of generic Math data with these scaffolds (while keeping the overall Math budget fixed) yields substantial gains on hard math benchmarks: overall math reasoning improves by 17.56 % (CollegeMath + 30.05 %, MATH + 23.17 %, OlympiadBench + 47.78 %, MathBench + 14.51 %). Simpler tasks (GSM8K, CMath) see slight drops, indicating that scaffolds help difficult problems but are not universally beneficial.

Model‑Internal Evidence

Analysis of MoE expert routing shows that removing Code or Math changes the activation patterns for the corresponding tasks, confirming that data composition affects internal computation paths. In contrast, adding scaffolds perturbs routing only mildly and in a more distributed way, suggesting they act as a gentle, cross‑task structural signal.

What the Study Ultimately Says

Pure code is valuable for learning programming constructs, but the reasoning improvements observed in prior work likely stem from the mixed Code‑NL content that contains structured reasoning traces. Future data‑optimization should focus not on the raw proportion of "code vs. math vs. web" but on selecting samples that carry transferable, well‑structured reasoning signals.