Unlocking Reasoning LLMs: Methods, DeepSeek R1 Insights, and Cost‑Effective Strategies

Reasoning models

A reasoning model is an LLM that must produce intermediate steps before arriving at a final answer. Typical examples are arithmetic problems (e.g., distance = speed × time) or multi‑step puzzles where the model must explicitly relate variables.

When to use a reasoning model

Reasoning models excel on complex tasks such as puzzles, advanced mathematics, and challenging programming problems. For simple tasks—summarization, translation, factual Q&A—standard LLMs are more efficient and cheaper.

Advantages and limitations

Advantages : higher accuracy on multi‑step problems, transparent reasoning traces.

Limitations : increased inference cost, longer responses, higher chance of “over‑thinking” errors.

DeepSeek‑R1 training pipeline

DeepSeek released three variants built on the 671 B DeepSeek‑V3 base model (Dec 2024):

DeepSeek‑R1‑Zero : pure reinforcement learning (RL) without any supervised fine‑tuning (SFT). Rewards are (i) accuracy (e.g., LeetCode compiler verification) and (ii) format (a language‑model reviewer enforces answer structure).

DeepSeek‑R1 : adds an SFT stage and a second RL phase on top of R1‑Zero, yielding a stronger flagship model.

DeepSeek‑R1‑Distill : uses the SFT data generated by the previous steps to fine‑tune smaller models (e.g., Llama 8B/70B, Qwen 1.5B‑30B), effectively distilling the large teacher.

Four main methods to build or improve reasoning models

Inference‑time scaling : increase compute at generation time (e.g., Chain‑of‑Thought prompting, voting, beam search) to obtain higher‑quality answers at the cost of more tokens.

Pure reinforcement learning (RL) : train directly with RL rewards (accuracy, format) without any SFT. DeepSeek‑R1‑Zero demonstrates that pure RL can induce reasoning behavior.

SFT + RL : first generate “cold‑start” SFT data with a base model, then perform instruction fine‑tuning followed by RL. This pipeline is used for DeepSeek‑R1 and is likely similar to OpenAI’s o1.

Distillation (SFT‑only) : generate high‑quality SFT data with a large teacher and fine‑tune a smaller student model. It is cheaper but does not push the frontier of reasoning capability.

Inference‑time scaling details

Chain‑of‑Thought (CoT) prompting adds a phrase such as “Step‑by‑step” to encourage the model to emit intermediate reasoning steps. Voting (multiple answer generation with majority selection) and beam search are additional techniques that improve answer quality while increasing token usage.

Pure RL findings

DeepSeek‑R1‑Zero uses two reward types:

Accuracy reward : deterministic evaluation (e.g., LeetCode compiler for code, exact match for math).

Format reward : a language‑model reviewer checks that the answer follows a prescribed structure (e.g., reasoning steps enclosed in tags).

Despite the lack of SFT, the model spontaneously develops an “aha moment” and begins to output reasoning traces.

SFT + RL pipeline for DeepSeek‑R1

1. Generate “cold‑start” SFT data with R1‑Zero. 2. Perform instruction fine‑tuning. 3. Run an RL phase that adds a consistency reward to avoid language‑mixing. 4. Collect 600 k CoT examples and 200 k knowledge‑based examples for a second RL iteration.

Distillation

Distillation fine‑tunes smaller models (e.g., 8 B‑70 B) on the SFT data generated by the large teacher. The resulting models are far more efficient while remaining competitive with some proprietary systems.

Budget‑friendly strategies

Distillation to obtain smaller, cheaper models.

Pure RL on tiny models (e.g., TinyZero, a 3 B model reproducing the R1‑Zero pipeline for under $30).

Journey learning

Journey learning augments SFT by deliberately including incorrect solution paths, allowing the model to learn from its mistakes. This resembles the self‑verification observed in TinyZero and can improve robustness.

Cost considerations

Training a model comparable to DeepSeek‑R1 is estimated to cost several million dollars (GPU‑hour rates ≈ $2/h). Exact figures are not disclosed, so estimates remain speculative. Distillation and tiny‑model RL provide orders‑of‑magnitude cheaper alternatives.

References

[1]

DeepSeek‑R1 technical report: https://arxiv.org/abs/2501.12948 [2] 2024 AI research papers (part 2): https://magazine.sebastianraschka.com/p/ai-research-papers-2024-part-2 [3] Scaling LLM test‑time compute: https://arxiv.org/abs/2408.03314 [4] LLM training: RLHF and alternatives: https://magazine.sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives [5] TinyZero repository: https://github.com/Jiayi-Pan/TinyZero/ [6] O1 Replication Journey (part 1): https://arxiv.org/abs/2410.18982