MIT Researchers Embed Generalization in Harness: Short-Task Training Unlocks 32× Length Extrapolation

Problem and Motivation

When a language model processes very long sequences, the accumulated tool results and intermediate states cause the input to drift away from the distribution seen during training, leading to poor performance on tasks that are many times longer than those used for fine‑tuning.

Recursive Language Model (RLM) and Harness Design

MIT CSAIL researchers built a Recursive Language Model (RLM) that keeps the core language model unchanged and introduces a “harness” that mediates between the model and the external environment. The harness performs two key functions: (1) context off‑loading , which stores large task data as variables in an external program so the root model only sees the high‑level task structure, and (2) programmatic sub‑agent calls , which treat tools and sub‑agents as function calls whose results are kept in variables rather than appended to the main context. This design aims to keep each model invocation “locally in‑distribution” (LID), i.e., close to the training input distribution, even when the overall task trajectory is far outside the training distribution.

Experimental Setup

All experiments share the same 30‑billion‑parameter base model (Qwen3‑30B‑A3B‑Instruct‑2507). Three variants were trained with reinforcement learning: a standard transformer baseline, an RLM, and an RLM with decomposition prompts. Training was performed only on short tasks (≈64K tokens). Evaluation tasks were stretched to 8–32× the training length, up to about 2 million tokens.

Six benchmark tasks varied the length of inputs, outputs, or instructions. Additional long‑context baselines included MRCRv2 (trained on 64K tokens) and GraphWalks (trained on <128K tokens).

Key Findings

On short tasks, RLM and the transformer baseline achieved comparable training‑reward gains.

On long‑task evaluation, RLM’s improvement was roughly ten times that of the transformer baseline, demonstrating strong length extrapolation.

When the same RLM harness was applied to Qwen3‑30B, performance on four long‑task benchmarks approached or exceeded the reference results of GPT‑5.5.

Training cost for RLM was 1.5–3× that of the transformer baseline because each long task required multiple sub‑agent calls.

Scaling the model further (e.g., using eight H100 GPUs) would make direct long‑trajectory training of a transformer prohibitively expensive.

Cross‑Domain Transfer

Three cross‑domain transfer groups were evaluated: (1) TREC question aggregation → SPAM/HAM aggregation, (2) writing‑style retrieval → mathematical‑reasoning retrieval, and (3) Twitter stance retrieval → WildChat error‑dialogue retrieval. Although training and test distributions differed, the underlying aggregation‑search‑filter pipeline remained similar.

RLM consistently improved on the new domains when it improved on the training tasks, indicating that the learned decomposition strategy transferred across domains, whereas the transformer baseline often achieved higher training reward but limited cross‑domain gains.

Analysis of Generalization Mechanisms

The authors argue that the harness provides a high‑level inductive bias that encourages the root model to learn reusable decomposition strategies rather than shortcuts that only work for short tasks. Decomposition prompts further help RLM avoid short‑task‑specific tricks. However, the LID property is not guaranteed in every call; occasional leakage of task‑specific information back into the main context can occur.

Similarity metrics (token‑level and n‑gram) show that RLM‑generated trajectories for unseen tasks are closer to trajectories seen during training, though these metrics only capture surface similarity and cannot prove true equivalence of the underlying decomposition.

Conclusion

Embedding generalization into the harness and training on short tasks enables a recursive language model to extrapolate up to 32× longer sequences, achieve better cross‑domain transfer, and outperform transformer baselines despite higher compute cost. Expanding training data remains the primary driver of capability growth, while the harness determines how much of that capability can be transferred to longer or novel tasks.

Reference: [1] Alex Zhang, Omar Khattab. “Language model harnesses are compositional generalizers.” 2026‑07. https://alexzhang13.github.io/blog/2026/harness/