Building a From‑Scratch LLM Training Framework: Full GRPO vs PPO vs DPO Comparison on GSM8K

Overview

Project grpo‑llm implements three reinforcement‑learning‑based fine‑tuning algorithms—GRPO, PPO and DPO—on the GSM8K mathematical reasoning dataset. It includes an asynchronous Trio rollout worker, FSDP multi‑GPU support, sandboxed code execution, and a C++ pybind11 reward extension.

Algorithms

GRPO (Group Relative Policy Optimization)

GRPO replaces a learned value network with group‑wise normalization. For each prompt G completions are sampled, scored, and the advantage is computed by subtracting the group mean and dividing by the group standard deviation:

def compute_advantages(self, rewards: torch.Tensor) -> torch.Tensor:
    rewards_grouped = rewards.view(-1, self.config.group_size)  # (batch, G)
    mean = rewards_grouped.mean(dim=-1, keepdim=True)
    std  = rewards_grouped.std(dim=-1, keepdim=True)
    return ((rewards_grouped - mean) / (std + 1e-8)).view(-1)

Only a single model needs to reside in memory, making GRPO cheaper than PPO.

PPO (Proximal Policy Optimization)

PPO adds a clipped surrogate objective to limit policy updates:

ratio = torch.exp(log_probs - old_log_probs)  # importance weight
obj_unclipped = ratio * advantages
obj_clipped = torch.clamp(ratio, 1 - epsilon, 1 + epsilon) * advantages
loss = -torch.min(obj_unclipped, obj_clipped).mean()

The same clipping objective is reused for GRPO; the only difference is the baseline (group mean vs learned value).

DPO (Direct Preference Optimization)

DPO optimizes a preference pair without rollouts or a KL penalty:

logits = beta * (
    (lp_chosen.sum() - ref_chosen.sum()) -
    (lp_rejected.sum() - ref_rejected.sum())
)
loss = -F.logsigmoid(logits)

Preference pairs are generated on‑the‑fly by scoring two completions with a binary math‑reward (1 for correct, 0 for incorrect).

Experimental Setup

Model: Qwen2.5‑0.5B‑Instruct . Dataset: GSM8K. Training: 300 iterations, evaluated on a fixed 200‑question hold‑out set. Hyper‑parameters: learning rate 1e‑6, KL coefficient β=0.01, identical across algorithms. Hardware: AWS g4dn.xlarge (Tesla T4 GPU). The only variable is the algorithm.

Results

DPO : 29.5% final accuracy, 120 minutes training time.

GRPO : 28.5% accuracy, 188 minutes.

PPO : 25.0% accuracy, 191 minutes.

DPO finishes in roughly half the time of the other methods and achieves the highest accuracy.

Unexpected Findings

GRPO was expected to excel because it avoids a value network, but with a 0.5 B model most rollouts produce a reward of 0, causing group‑wise advantages to collapse to zero. Consequently many training steps generate zero gradients, wasting compute. PPO suffers similarly and adds extra clipping overhead.

DPO sidesteps this issue: even when both completions are wrong, the relative preference signal remains non‑zero, ensuring continual gradient updates. For small models and sparse binary rewards, the density of the preference signal matters more than the algorithmic sophistication of online RL.

Implementation Details

Trio rollout worker : Uses Trio’s nursery for structured concurrency; exceptions in workers propagate automatically, unlike asyncio which requires manual handling.

Sandboxed code execution : Module envs/code_exec.py runs each generated snippet in an isolated subprocess with memory limits, timeouts and output truncation, making it easy to replace the sandbox later.

C++ pybind11 reward extension : Function normalize_answer is called millions of times during training; the C++ implementation is 1.7× faster than pure Python, yielding noticeable overall speed gains.

All 25 unit tests for the reward and execution environment run on CPU only, allowing rapid verification before consuming GPU resources.

Future Work

Planned extensions include denser step‑wise reward models, larger models (>7 B parameters), longer training (≥2000 iterations), and multiple random seeds to achieve statistical significance.

Reproducibility

Code repository: https://github.com/Uttaprexa/grpo-llm

Configuration file: configs/grpo_gsm8k.yaml Experiment script: experiments/compare_algorithms.py Results file:

experiments/results.json