Lev: Looped Decision Models Adapt Compute Depth Per Query

Background: Decision Models Need Adaptive Compute

Decision models (e.g., Kev, Jev) take a context and candidate options, then output a probability distribution over options in a single forward pass. They are fast (tens of milliseconds) but their compute is fixed at training time: easy and hard questions consume the same resources, and they cannot "think harder" on difficult inputs. Existing ways to add compute—cascading to a larger model, generating chain-of-thought text, or retrofitting loops onto a pretrained model—all have drawbacks: cascading re-reads the input and needs multiple deployments; chain-of-thought forces every question to pay for reasoning tokens; adding loops to a pretrained Kev-0.8B (Qwen3.5-0.8B) failed because the gating stayed near zero, leaving the loop inactive.

Lev Architecture: Native Looped Backbone + Shared Pointer Head

Lev replaces the backbone with a native looped language model (Ouro or Nanbeige) that was pretrained with repeated layer passes. The same decoder layers run R times; after each loop a shared pointer head reads the hidden states at option end tokens and a <decide> position, producing a full probability distribution. Because the pointer head and LoRA adapters are shared across loops and the loss averages over all loops, every depth becomes a usable decision model , not just the final one.

Two immediate benefits: (1) serving k loops costs ~ k/R of full depth; (2) continuing to the next loop resumes from the previous hidden state, so the input is not re-encoded—unlike cascading.

Halting Mechanism: Predict Consistency with Deepest Answer

Standard early-exit uses confidence thresholds, but the authors found two failures on decision models: (a) shallow loops are overconfident on new data (94.8% calibrated accuracy on calibration set vs. 78.9% on Kev-Transfer at same confidence), and (b) confidence cannot distinguish errors that extra loops would fix (AUROC 0.46–0.61). Lev's halter instead predicts whether the current answer already matches the deepest loop's answer . It is a per-depth logistic regression using features from current and previous loop readouts: max probability, top-2 gap, normalized entropy, answer change flag, KL divergence. The target label is the model's own deepest prediction, so no human labels are needed ; thresholds are chosen on unlabeled target traffic to keep ≥98% agreement with the deepest answer.

Experimental Setup

Models: Lev-1.4B (Ouro-1.4B, 4 loops), Lev-2.6B (Ouro-2.6B, 4 loops), Lev-3B (Nanbeige4.2-3B, 2 loops; Nanbeige only supervised on final loop during pretraining).

Baselines: Qwen3-1.7B, Qwen3.5-2B, Qwen3.5-4B (dense), trained on same 31,329 examples with LoRA rank 16 for one epoch. No distillation from baselines.

Evaluation: Three tuning sets for recipe/halter design, then two held-out test sets: Kev-Breadth (5 domains, 14 unseen datasets) and LevBench (frozen before any model access, difficulty from in-distribution to completely unseen). Also JevBench 231 public questions.

Key Findings

1. Accuracy Improves with Depth, Diminishing Returns

Across all backbones and test sets, accuracy rises with loops. JevBench: Lev-2.6B goes from 63.2% (1 loop) to 76.2% (4 loops), +13.0 pp (95% CI 7.4–19.0). Most gain from the first extra loop: held-out average 51.3% → 62.0%; fourth loop adds ≤1 pp. Gains are larger on out-of-distribution data: +7 pp on training source, +9.0–19.5 pp on four farther test sets.

2. Extra Compute Focuses on Hard Questions

On JevBench, 1→4 loops: standard questions +10 correct, hard questions +19 correct; easy questions near 100% at all depths. Hard questions: 29 fixed, 10 broken (mostly corrections). On LevBench, first extra loop fixes 20.4% but breaks 8.6%; 12.6% are correct at a shallow loop but wrong at deepest—halter targeting deepest agreement cannot recover these. Gains vary by task: language understanding +19.0 pp on Kev-Breadth, zero on humor/color-preference "taste" tasks, suggesting extra loops provide serial composition of facts, not subjective judgment.

3. Adaptive Depth Saves 15–24% Compute with <1 pp Accuracy Loss

On held-out sets, halter uses 76–85% of max loops, accuracy within 0.3–0.8 pp of full depth. Halter automatically allocates more loops to hard questions (JevBench: easy 1.2, standard 2.4, hard 3.0 loops avg). Compared to fixed-depth frontier, adaptive depth advantage is modest (Lev-2.6B +0.2 pp, Lev-3B +1.0 pp) because Ouro's frontier is concave—first extra loop captures most gain. On locked test (recipe+halter fixed), adaptive depth loses ≤1.0 pp vs. deepest; Lev-1.4B and Lev-3B beat fixed frontier, Lev-2.6B matches it. Note: compute measured in loop counts; real latency gains need a serving system that batches variable-depth requests.

4. Calibration on Target Traffic Matters More Than Halter Features

On Ouro models, confidence/diff-gap/entropy/patience early-exit rules on labeled data all fall below fixed-depth frontier. But a simple max-probability threshold calibrated on unlabeled target requests with the "match deepest" criterion reaches within 0.2 pp of the learned halter. Conversely, fitting on labeled data with "is correct" target drops below frontier. Only 500 unlabeled questions per fold suffice for Lev-2.6B to match full-data calibration within 0.2 pp.

5. Lev Beats Dense and Cascade at Low Latency (≤50 ms)

At 39–46 ms (single H200, bf16, no fusion/graph capture/state cache), adaptive Lev-1.4B and Lev-3B outperform best dense by 5.4–7.5 pp and best cascade by 7.0–8.8 pp on LevBench locked test. Caveats: (a) dense gap partly due to no dense model in 22–79 ms range; (b) FLOPs per Lev model equal or exceed dense because each loop runs full backbone—advantage is latency, not compute; (c) Lev-2.6B at ~150 ms does not beat Kev-4B. Lev also matches larger models with fewer parameters: Lev-3B ≈ Qwen3.5-4B on JevBench, Lev-2.6B ≈ public Kev-4B (76.2% vs 75.8%, not significant). Fewer parameters ≠ fewer FLOPs.

Training Observations

Joint training across depths hardly hurts: vs. single-depth Ouro-2.6B, Lev-2.6B matches at 2 and 4 loops, ≤1.3 pp lower at 1 loop. Training only the last loop makes shallow readouts unusable.

More data mainly strengthens shallow layers: 31k → 120k examples: 1-loop held-out 50.5% → 62.9%, deepest 65.3% → 73.0%, adaptive cost 0.76 → 0.63. Partly from in-distribution data.

LoRA beats full fine-tuning: Full FT on Ouro drops deepest loop 1.3–1.9 pp and loses frontier advantage.

Limitations

Looped backbones only 1.4B–3B (largest publicly available); scaling effects unknown.

JevBench results based on 231 public questions only.

Adaptive compute measured in loop counts; real throughput/latency gains require a serving system that batches variable-depth requests.

Latency measured without any serving optimizations (fusion, graph capture, KV cache) for either looped or dense models.

Conclusion & Future Work

Lev turns loop depth into a serve-time knob : one decision-model checkpoint trades latency for accuracy per question without deploying multiple models. Depth gains concentrate on hard questions; a halter calibrated on unlabeled target traffic with the "match deepest" objective automatically directs compute to hard questions. Future directions: (1) halting objectives that recover the 12.6% of questions where shallow is right but deepest is wrong, potentially surpassing deepest accuracy; (2) test on larger looped backbones to see if hard-question gains scale.

Paper: https://arxiv.org/abs/2610.20951 Code: https://github.com/OpenLAIR/lev Models and data: https://huggingface.co/collections/songdj/lev Project homepage: https://openlair.github.io/lev/