Understanding Reasoning LLMs: DeepSeek R1 Variants, Inference‑Time Scaling, and Training Strategies

Reasoning LLMs are defined as models that answer complex, multi‑step questions by generating intermediate reasoning steps, unlike simple factual queries.

Key characteristics of reasoning models include:

Purpose: They excel at tasks like puzzles, advanced mathematics, and challenging coding problems, but are not required for simple tasks such as summarization or translation.

Trade‑offs: Higher inference cost, longer outputs, and a tendency to over‑think can lead to more errors, making them less efficient for straightforward use‑cases.

DeepSeek R1 family: Three variants exist – DeepSeek‑R1‑Zero (cold‑start RL only), DeepSeek‑R1 (adds SFT and further RL), and DeepSeek‑R1‑Distill (smaller distilled models trained on SFT data from larger checkpoints).

Inference‑time scaling: Improves answer quality by allocating more compute during generation (e.g., Chain‑of‑Thought prompting, voting, beam search), but increases latency and cost.

Training pipelines: RL‑only (DeepSeek‑R1‑Zero) uses accuracy and format rewards without any SFT; SFT + RL (DeepSeek‑R1) adds a supervised fine‑tuning stage followed by RL with additional consistency rewards; distillation creates smaller models from the large checkpoints using the same SFT data.

Additional insights:

Pure RL can produce viable reasoning abilities even in relatively small models (e.g., TinyZero with ~3B parameters).

Pure SFT can achieve strong performance when large, high‑quality instruction data is available (e.g., Sky‑T1 trained on 17K samples for $450).

Distilled models offer lower inference cost and run on modest hardware, serving as benchmarks for the limits of SFT‑only training.

Practical considerations for enterprises include evaluating when the higher cost of reasoning models is justified, such as in search‑augmented workflows that benefit from deep thinking, while being aware of hallucination risks.

Images from the original article illustrate model architectures, scaling concepts, and performance comparisons: