How CoLT Accelerates Multimodal Reasoning Over 20× with a 3‑Step Latent Thought Chain

Motivation

Multimodal large language models (e.g., Qwen3‑VL, InternVL3, Step3‑VL) achieve strong visual QA, chart reasoning, and scientific reasoning performance by using Chain‑of‑Thought (CoT) prompting, which generates step‑by‑step natural‑language reasoning before the final answer. Text‑based CoT suffers from three structural bottlenecks:

Inference efficiency : each reasoning step requires autoregressive token generation, often consuming hundreds to thousands of tokens.

Expressive limitation : collapsing a continuous reasoning process into a single textual statement prevents maintaining multiple solution paths.

Error propagation : early mistakes are amplified along long chains.

Core Idea: Chain of Latent Thoughts (CoLT)

CoLT replaces the explicit textual chain with a fixed number of continuous latent vectors h_1,…,h_K (default K=3). Each vector corresponds to the language model’s hidden state at a designated reasoning position and is fed back as the next step’s input embedding, eliminating extra vocabulary tokens or special markers.

This yields a magnitude‑level efficiency gain: Text CoT averages 142 tokens per inference, while CoLT uses only 3 latent vectors. Generation time drops from ~ 7.24 s to 0.32 s on the MMStar benchmark (single‑card H200), a 22.6× speedup, and end‑to‑end inference improves 10.1× without sacrificing accuracy.

Step‑Level Supervision

CoLT introduces a lightweight external decoder D_\phi (default Qwen3‑0.6B, sharing the vocabulary with Qwen3‑VL‑8B) that provides two complementary supervision signals during training:

Forward decoding : conditioned on the current latent vector h_k, the decoder autoregressively generates the next textual reasoning step r_{k+1}, ensuring each latent vector encodes enough information to reconstruct explicit reasoning.

Backward decoding : using the previous textual step r_{k-1} as input, the decoder’s final hidden state is aligned (cosine similarity) with the latent vector h_k. A stop‑gradient operation prevents the latent vector from collapsing to a trivial match.

An internal step‑level supervision predicts the next latent representation h_{k+1} from h_k via a two‑layer MLP head f_\theta. Cosine‑similarity loss enforces structural coherence between consecutive steps. The total training objective combines the three losses with equal weights ( \alpha=\beta=\gamma=0.2).

Training and Zero‑Cost Inference

Training consists of a supervised‑fine‑tuning stage on the OneThinker image subset. Textual CoT annotations are segmented into K parts to align with latent steps. During inference, both the external decoder and the projection head are discarded; only the three latent vectors are generated, incurring zero additional computational overhead.

Experimental Results

CoLT was evaluated on eight multimodal benchmarks (visual understanding, chart reasoning, text‑rich VQA, scientific QA, composite tasks) using Qwen3‑VL‑8B‑Instruct as the backbone. It achieved an average accuracy of 79.1 % , surpassing Text CoT (75.7 %, +3.4 %) and leading existing latent‑reasoning baselines. The largest gains appear on ChartQA (+9.6 %) and TextVQA (+6.1 %). Compared with closed‑source models, CoLT reaches 84.6 % on MMBench (close to GPT‑5‑high at 83.8 %) and 74.7 % on ChartQA, far ahead of Gemini‑2.5‑Pro (59.7 %).

Ablation Studies

Four‑component ablations on representative benchmarks show that each supervision signal contributes positively. Individually, backward decoding (67.2 %) outperforms forward decoding (65.8 %) and internal supervision (63.6 %). Combining forward and backward yields the strongest pairwise result (69.8 %). Using all three together reaches 72.6 % average, with the backward decoder providing the largest marginal gain (+4.5 %).

Decoder size ablation demonstrates that scaling the external decoder from Qwen3‑0.6B to Qwen3‑8B improves average accuracy by only 0.6 %, confirming the “small‑but‑effective” nature of the decoder.

K‑step sensitivity analysis indicates optimal performance at K=3 (72.6 %). Fewer steps limit reasoning capacity, while more steps introduce redundancy. Training with K=3 generalizes well to K=6 with only a 0.7 % drop.

Interpretability

The forward decoder can project each latent vector h_k back to natural language, revealing the encoded reasoning fragment. Visualizations show that each latent step corresponds to distinct logical sub‑tasks (e.g., visual scanning, counting, pattern inference), demonstrating clear semantic boundaries.

Efficiency Measurements

On the MMStar benchmark, encoding time is ~0.45 s for all methods. Generation time for Text CoT is 7.24 s (142.1 tokens); CoLT generates 3 latent vectors in 0.32 s, a 22.6× generation speedup. End‑to‑end time improves 10.1×, and accuracy rises from 67.1 % (Text CoT) to 68.9 % (CoLT). Similar gains are observed on MMT‑Bench (generation 22.4× faster, end‑to‑end 10.1× faster, accuracy 67.4 % vs 63.3 %).

Resources

Paper: https://arxiv.org/abs/2606.31986 (ECCV 2026)

Code: https://github.com/hulianyuyy/CoLT