Turning Chain‑of‑Thought into Images: The Render‑of‑Thought Breakthrough

Background and Motivation

Large language models (LLMs) rely on chain‑of‑thought (CoT) prompting to solve complex reasoning tasks, but explicit CoT generates long token sequences that dramatically increase computation and memory usage. Implicit CoT removes the textual intermediate steps, yet it becomes a black‑box with no observable reasoning process.

Render‑of‑Thought (RoT) Concept

RoT bridges the gap by rendering each reasoning step as an image using the visual encoder of a multimodal model (VLM) as a semantic anchor. The textual CoT is transformed into a compact visual embedding, enabling dense latent reasoning while preserving traceability.

Architecture

Stage 1 – Visual Alignment

The LLM and the frozen visual encoder are kept fixed while a lightweight visual projection head is trained to map the LLM’s hidden state at each reasoning step h_t to the visual embedding of the rendered CoT image v_t. The alignment loss is a mean‑squared error between h_t (projected) and v_t. A special token <|img_end|> is also supervised with cross‑entropy against the ground‑truth answer token y.

Stage 2 – Latent Supervised Fine‑Tuning

After alignment, the projection head is frozen. LoRA is applied to the LLM to fine‑tune it for generating a sequence of latent visual tokens that mimic the visual encoder’s output. These tokens are autoregressively produced and later decoded into the final textual answer.

Inference and Decoding Strategies

Two termination strategies are explored:

Dynamic termination – the model stops when the probability of a special termination token peaks.

Static token budget – a fixed number of latent tokens is generated, after which <|img_end|> forces the switch to textual decoding.

Empirically, the static budget yields higher accuracy because the dynamic strategy suffers from instability in the latent space.

Experiments and Results

RoT is evaluated on GSM8K, MATH, SVAMP, MultiArith, and other reasoning benchmarks using Qwen‑VL and LLaVA backbones. Key findings:

Token compression of 3–4× compared with explicit CoT, leading to 3–4× faster inference.

Static token budgets of 32 (GSM8K‑Aug) and 64 (MATH) achieve the best accuracy, reflecting dataset difficulty.

RoT outperforms recent implicit methods such as Coconut and CoLaR, e.g., 97.2% accuracy on MultiArith with Qwen3‑VL‑4B.

Heat‑map visualizations of latent token similarity reveal structured, stage‑wise reasoning, confirming interpretability.

Ablation Studies

Removing Stage 1 drops MATH accuracy from 33.2% to 22.2%, showing visual alignment is crucial for building a stable latent space. Omitting Stage 2 also degrades performance, confirming the necessity of latent‑space fine‑tuning.

Analysis of Rendering Choices

Single‑row dynamic‑width images outperform fixed‑size multi‑row renders because they preserve left‑to‑right token order and avoid unnecessary spatial jumps in the visual domain.

Conclusion and Outlook

RoT introduces a promising "visual‑latent" reasoning paradigm that compresses CoT into dense image embeddings, delivering substantial speed gains and opening a new window into the opaque internal states of LLMs. Its plug‑and‑play nature eliminates extra pre‑training costs, making it attractive for deployment on resource‑constrained devices. Future work may explore broader multimodal encoders, larger token budgets, and richer visual rendering techniques.