How DeepSeek R1 Uses Large‑Scale Reinforcement Learning to Replicate OpenAI o1

From ChatGPT to the latest AI breakthroughs, DeepSeek has become the newest focus in the field. In this article, a Tsinghua University associate professor analyzes the large‑scale reinforcement learning techniques behind DeepSeek R1, its core principles, and the future direction of large‑model technology.

1. Understanding DeepSeek R1 and the Trend of Large‑Model Technology

DeepSeek R1 is notable for faithfully reproducing the deep‑reasoning capabilities of OpenAI o1, which was released without any implementation details. By applying pure reinforcement learning at scale, DeepSeek appears to be the first team to achieve this replication and has openly shared a relatively detailed description.

The training process has two major highlights. First, DeepSeek R1 builds on the DeepSeek V3 base model and, through massive reinforcement learning, creates a purely RL‑enhanced strong‑reasoning model called DeepSeek‑R1‑Zero. This is significant because few teams have successfully applied reinforcement learning to large‑scale models and achieved large‑scale training.

Second, the reinforcement‑learning technique is not limited to domains with obvious reward signals (such as mathematics or code). It generalizes strong reasoning ability to other tasks, enabling users to experience deep‑thinking capabilities during writing and other applications.

The generalization is achieved in two stages. First, using the DeepSeek V3 base, a supervised‑fine‑tuning (SFT) dataset is generated that enhances the readability of the reasoning process, combining deep‑reasoning data with conventional SFT data. Second, reinforcement learning further fine‑tunes the model, producing a strong‑reasoning model with broad task generalization.

Thus, DeepSeek R1’s contributions are twofold: (1) a rule‑driven method that makes large‑scale reinforcement learning feasible, and (2) a mixed‑fine‑tuning approach that blends deep‑reasoning SFT data with general SFT data to achieve cross‑task reasoning, successfully matching OpenAI o1’s inference level.

DeepSeek R1’s open‑source release allows the global community to experience deep‑thinking capabilities similar to the impact of ChatGPT in early 2023, pushing the AI field forward by a significant step. However, the article also cautions that the model’s importance should be evaluated realistically, noting that OpenAI’s closed, costly approach with o1 limited its widespread adoption.

2. The Concept of Ability Density

The author introduces the “ability density” law, analogous to Moore’s law, stating that a model’s capability per parameter roughly doubles every 100 days. This rapid increase is driven by higher data quality, sparse‑activation architectures (e.g., MoE), and advanced scaling‑prediction methods that optimize training hyper‑parameters before large‑scale runs.

These factors collectively enable models to achieve the same performance with half the parameters or compute, reducing both training and inference costs. The article argues that future AI development will continue along this efficiency trajectory, making powerful models more affordable and widely accessible.

3. Future Directions and Open Questions

The discussion highlights several research avenues: exploring more efficient model architectures such as MoE, improving data quality, and developing modular, sparsely activated systems. While MoE is promising, the author stresses that no single architecture can be deemed the ultimate solution for AGI.

Finally, the article emphasizes the importance of open, collaborative research, low‑cost innovation, and long‑term commitment to advancing AI. It calls for supporting idealistic teams like DeepSeek, fostering talent, and encouraging diverse approaches to achieve a truly inclusive intelligent‑revolution.