Why One Extra Loop Is All a 7B Model Needs – LoopCoder‑v2’s Surprising Sweet Spot

Model and Training

LoopCoder‑v2 is a 7‑billion‑parameter dense transformer trained on 18 T tokens with a 1:1 text‑code mix covering over 100 programming languages. The only inference‑time hyper‑parameter is the number of loops.

Parallel Loop Transformer (PLT) Mechanism

PLT removes serial dependence between loops using Cross‑Loop Position (CLP) offsets, allowing multiple passes to be computed in parallel. It shares KV caches across loops with a gated sliding‑window attention (G‑SWA), keeping memory growth flat. CLP introduces a fixed “position‑mismatch tax” Ω that remains constant across loops.

Benchmark Results

Evaluation on four code‑related benchmarks shows a peak at two total passes (one extra loop):

SWE‑bench Verified: 43.0 → 64.4

SWE‑bench Multilingual: 14.0 → 31.0

LiveCode‑Bench: 27.4 → 35.4

Average of ten tasks: 38.0 → 46.5

Adding a third or fourth loop degrades performance (e.g., SWE‑bench Verified drops to 27.6 and 22.4), falling below the no‑loop baseline.

Internal Diagnostic Probes

After each loop the authors measured:

Evolution of hidden states

Attention routing patterns

Changes in output distribution

All three indicators show that the second loop achieves most of the useful refinement: hidden states converge steadily, attention reallocates effectively, and output quality improves markedly. Subsequent loops produce diminishing updates, with attention routes freezing and hidden‑state changes oscillating, indicating near‑zero marginal gain.

Cost‑Benefit Analysis

The benefit curve collapses after the second pass while the position‑mismatch tax Ω stays constant, so any additional loop incurs net loss.

Comparison with Larger Models

LoopCoder‑v2 (2 passes) reaches 64.4 on SWE‑bench Verified, surpassing the 235‑billion‑parameter Qwen3‑235B (45.2) and approaching flagship open‑source models Kimi‑K2 (69.2) and Qwen3‑Coder‑480B (67.0). On agentic benchmarks Terminal‑Bench (26.3 → 34.2 and 11.2 → 21.0) and tool‑use BFCL (32.2 → 40.1) the gains are similarly large.

Future Directions

The authors suggest exploring adaptive position offsets, task‑dependent dynamic loop counts, and the relationship between internal looping and explicit chain‑of‑thought prompting.

Code example

huggingface.co/Multilingual-Multimodal-NLP/LoopCoder-V2