Decoding OpenAI o1: How RL and LLM Fuse to Power Hidden Chain‑of‑Thought

The author treats OpenAI o1 as a breakthrough in large‑model technology, focusing on its reinforcement‑learning (RL) driven hidden chain‑of‑thought (Hidden COT) generation. By examining public hints, system cards, pricing differences, and analogies to AlphaZero, the article builds a speculative but technically grounded model called Reverse‑o1 to explain o1’s possible mechanisms.

Key Significance of o1

Self‑reflection and error correction: Unlike GPT‑4, o1 can detect and fix its own token‑level mistakes during hidden reasoning, greatly extending reliable long‑chain reasoning.

New RL scaling law: The model likely incorporates tree‑search (e.g., Monte‑Carlo Tree Search) or Best‑of‑N sampling, enabling flexible scaling of compute during inference.

Enabling strong small models: By offloading logical reasoning to a powerful RL‑augmented core, smaller models can achieve comparable performance when combined with external knowledge retrieval.

Safety alignment paradigm shift: Enhanced logical ability allows the model to follow explicit safety rules (akin to Anthropic’s AI Constitution) without heavy RLHF safety training.

Generalization beyond STEM: The author hypothesizes that reward definitions can be crafted for fuzzy‑reward domains (e.g., essay grading) by encoding textual criteria as reward signals.

Speculated Training Process

Training likely diverges from GPT‑4 in two major ways:

Pre‑training: A fresh pre‑training run with a heavy bias toward logical‑heavy data (STEM, code, research papers) to boost reasoning capacity.

Post‑training (RLHF): Includes a standard RLHF stage for instruction following, but separates safety fine‑tuning to a later stage and heavily emphasizes process‑reward models (PRM) alongside output‑reward models (ORM) to provide dense feedback during hidden reasoning.

The inference pipeline adds a "Think" stage where the model first generates a hidden COT, optionally summarizes it, and finally produces the answer.

Model Composition

Evidence from pricing and system cards suggests o1 consists of at least three components:

A main LLM backbone.

A hidden‑COT summarizer model.

A configurable pool of auxiliary models (likely tree‑search agents) whose count scales with model tier (e.g., o1 mini vs. o1 Preview).

Reverse‑o1 Design

The proposed Reverse‑o1 merges a Transformer LLM with an RL head inspired by AlphaZero. The RL head outputs:

A policy distribution over discrete "thought‑factors" (e.g., DecomposeProblem, ProposeHypothesis, CheckResult).

A value estimate of the current hidden‑COT state’s likelihood of reaching a correct answer.

During generation, the policy selects a thought‑factor, the LLM then emits the corresponding token segment, and the new tokens are fed back as the next state. Tree‑search (MCTS) can be applied over thought‑factors, with Best‑of‑N sampling and a PRM evaluator selecting the highest‑quality token continuation.

Reward Modeling

Two reward types are discussed:

Output Reward Model (ORM): Binary reward based on final answer correctness.

Process Reward Model (PRM): Step‑wise feedback derived from human‑annotated COT or automatically generated via MCTS‑based self‑play, providing richer learning signals.

The author believes o1 employs both, using PRM for data‑efficient learning of the hidden reasoning process.

Scaling Insights

Inference‑time compute scaling (e.g., increasing Best‑of‑N or tree depth) helps on easy‑to‑moderate problems but hits a ceiling on hard logical tasks, where pre‑training logical capacity must be improved. This aligns with recent papers showing test‑time compute can sometimes outweigh model size scaling.

Practical Takeaways

Reproducing o1‑like performance without massive resources may involve selecting a strong base LLM, adding a Think stage with simple tree‑search or Best‑of‑N sampling, and using lightweight PRM‑style evaluators.

Future AI safety pipelines might first boost reasoning then apply rule‑based safety prompts, simplifying alignment.

Small‑model ecosystems could adopt a "divide‑and‑conquer of ability" (DCA) strategy: language via a compact model, reasoning via an o1‑style RL‑augmented core, and world knowledge via external retrieval.