Tagged articles
PPO
54 articles · Page 1 of 1
This article breaks down six high-frequency GRPO interview questions from top Chinese tech companies, covering GRPO vs PPO trade-offs, group-relative advantage calculation with concrete numbers, handling all-correct/all-wrong sample groups, KL constraint mechanics, convergence monitoring priorities, and reward hacking detection via shadow evaluation.
Advantage EstimationGRPOInterview Preparation
0 likes · 19 min read

This article surveys the 2026 reinforcement learning landscape for large language models, detailing foundational algorithms (PPO, DPO, GRPO), advanced GRPO variants (DAPO, GSPO, GMPO, GFPO), emerging Agentic RL methods (ARPO, Tree-GRPO), and a practical scenario-based selection table.
Algorithm SelectionDPOGRPO
0 likes · 12 min read

The article explores how batch size scaling impacts LLM reinforcement learning training efficiency, introducing critical generation and training batch concepts, showing optimal batch size balances sample efficiency and system throughput, with experiments achieving 29% time reduction on fixed hardware.
Batch Size ScalingCritical BatchGRPO
0 likes · 36 min read

The article presents a from‑scratch LLM training framework called grpo‑llm, implements GRPO with Trio rollout, FSDP and a C++ reward extension, and conducts a controlled experiment comparing GRPO, PPO and DPO on the GSM8K math‑reasoning benchmark, revealing why DPO outperforms the other two under sparse binary rewards.
DPOGRPOGSM8K
0 likes · 10 min read

The article recounts how Proximal Policy Optimization, initially dismissed by NeurIPS 2017 for limited novelty, later became a cornerstone of RLHF and large‑language‑model training, illustrating how academic evaluation can miss long‑term impact, with parallels to other once‑rejected breakthroughs such as LSTM, SIFT and Dropout.
Algorithm RejectionNeurIPSPPO
0 likes · 5 min read

GLM‑5.2 replaces the Group Relative Policy Optimization (GRPO) algorithm with a critic‑based PPO approach for long‑horizon tasks, arguing that GRPO’s group comparison breaks down on variable‑length trajectories, a shift that has sparked vigorous debate across the reinforcement‑learning community.
DeepSeekGLM-5.2GRPO
0 likes · 10 min read

This article surveys five years of large‑model reinforcement learning, detailing the evolution from PPO + RLHF to DPO and GRPO, comparing reward‑model‑based and verifiable‑reward approaches, discussing multi‑agent extensions, and evaluating open‑source frameworks for training LLM‑driven agents.
AI AlignmentDPOGRPO
0 likes · 34 min read

UnityMAS‑O introduces a general reinforcement‑learning framework that converts predefined LLM multi‑agent workflows into trainable tasks, enabling credit assignment across roles, supporting parameter‑sharing configurations, and demonstrating significant F1 and test‑pass improvements on QA and code‑generation benchmarks.
LLMMulti-Agent Reinforcement LearningPPO
0 likes · 12 min read

This article reviews the five‑year evolution of reinforcement‑learning techniques for large language models, comparing PPO, DPO, GRPO and emerging multi‑agent approaches, analyzing their reward signals, practical trade‑offs, and the open‑source frameworks that support them.
DPOGRPOLLM
0 likes · 34 min read

DiffusionOPD introduces a unified on‑policy distillation framework for diffusion models that decouples single‑task online policy exploration from multi‑task capability integration, training expert teachers per task and distilling their skills into a single student model, achieving faster convergence and higher performance across composition, OCR, and aesthetic tasks.
KL DivergenceOn-Policy DistillationPPO
0 likes · 8 min read

The article reviews five years of LLM‑centric reinforcement learning, tracing the evolution from early Q‑learning to PPO, then to Direct Preference Optimization, Group Relative Policy Optimization, and finally multi‑agent RL, detailing each method’s mechanics, strengths, failure modes, practical considerations, and emerging open‑source toolchains.
DPOGRPOLLM Alignment
0 likes · 33 min read

This article surveys the rapid evolution of reinforcement‑learning algorithms for large‑language‑model inference from early REINFORCE and PPO to newer approaches such as GRPO, RLOO, DAPO, CISPO, DPPO, ScaleRL and MaxRL, highlighting their design motivations, mathematical formulations, empirical trade‑offs and open research challenges.
GRPOLLMMaxRL
0 likes · 27 min read

The Task Tokens method introduced by an Israeli research team reduces the number of trainable parameters per task by up to 125‑fold and speeds up convergence by six times, while preserving the flexibility of Behavior Foundation Models and demonstrating strong performance, robustness, and compatibility across a suite of embodied control tasks.
Behavior Foundation ModelsEmbodied AIMulti-Modal Prompting
0 likes · 13 min read

DeepAries is a novel deep reinforcement‑learning framework that jointly learns when to rebalance a portfolio and how to allocate assets by combining a Transformer‑based state encoder with PPO, and extensive experiments on four major markets show it significantly outperforms fixed‑frequency baselines in risk‑adjusted return, transaction cost, and drawdown.
DeepAriesPPOPortfolio Management
0 likes · 15 min read

This article explains why interviewers use seemingly rote questions to probe a candidate's true grasp of concepts, contrasts memorization with genuine understanding using PPO vs GRPO, and provides a practical three‑question framework and dialogue examples to help candidates answer technical principle questions confidently.
Answering TechniquesGRPOInterview
0 likes · 12 min read

A new MIT paper reveals that pretrained large models already contain many hidden expert submodels, and that a simple one‑step Gaussian perturbation (RandOpt) can locate and ensemble these experts to achieve performance comparable to or better than traditional GRPO/PPO tuning, especially as model size grows.
GRPOModel ScalingPPO
0 likes · 9 min read

This article systematically reviews the main reinforcement‑learning algorithms—PPO, GRPO, DAPO, GSPO, and SAPO—used for fine‑tuning large language models, explaining why supervised fine‑tuning precedes RL, how each method improves training efficiency and stability, and what trade‑offs they entail.
GRPOPPORL
0 likes · 15 min read

The article introduces the TLM large‑model development platform, details its fine‑tuning options, explains reinforcement learning fundamentals and key algorithms such as PPO and the newer GRPO, describes the architecture of a reinforcement evaluator, and shows how to configure RL training on the platform.
AI PlatformGRPOLLMOps
0 likes · 10 min read

This article examines why regression models need uncertainty estimates, explains aleatoric and epistemic uncertainty, compares four neural‑network approaches (Mean + LogStd, Mean + LogVariance, MC Dropout, simplified PPO) on a concrete‑strength dataset, and analyzes their experimental performance and limitations.
Monte Carlo DropoutPPOregression
0 likes · 10 min read

This article examines how reinforcement‑learning techniques such as PPO, DPO, and GRPO are integrated into the Baixiaosheng QA system to improve answer stability, deepen domain knowledge understanding, and accelerate response generation, and it evaluates the impact of Reinforcement Fine‑Tuning (RFT) on real‑world performance.
AIDPOGRPO
0 likes · 16 min read

The article explains the fundamental principles of PPO and GRPO reinforcement‑learning algorithms, compares their architectures and training workflows, highlights why GRPO is gaining traction in large‑model fine‑tuning, discusses associated risks, and offers practical guidance on group size selection for engineers preparing for interviews.
GRPOPPORLHF
0 likes · 9 min read

This article explains the GRPO algorithm, an improvement over PPO for large language model training that eliminates the value network, uses group‑relative advantage estimation, and offers flexible supervision, resulting in higher efficiency, stability, and performance on tasks such as mathematical reasoning.
GRPOLLM trainingPPO
0 likes · 16 min read

This article provides a comprehensive technical analysis of PPO‑based reinforcement learning methods for large language models, detailing the evolution from the original PPO algorithm through GRPO, DAPO, and GSPO, and explaining their motivations, mathematical formulations, advantages, and practical challenges such as entropy collapse and importance‑sampling variance.
DAPOGRPOGSPO
0 likes · 30 min read

This article explains the motivation, mathematical foundations, implementation details, advantages, experimental results, and future directions of Group Relative Policy Optimization (GRPO), a novel reinforcement‑learning algorithm that replaces PPO’s value network with efficient group‑wise relative evaluation for large language models.
GRPOLLMPPO
0 likes · 17 min read

This article systematically adapts classic deep reinforcement‑learning techniques—such as multi‑step returns, TD(λ)/GAE, V‑trace corrections, uncertainty‑aware weighting, safety constraints, distribution‑robust optimization, and value‑guided decoding—to improve large language model training and inference, providing concrete formulas, implementation tips, and empirical results.
Deep RLGAELLM
0 likes · 17 min read

This article explains the RLHF (Reinforcement Learning from Human Feedback) pipeline that powers ChatGPT and other large language models, covering the limitations of traditional fine‑tuning, the creation of human‑feedback datasets, reward‑model training, loss design, and the final PPO‑based fine‑tuning step.
ChatGPTHuman FeedbackPPO
0 likes · 8 min read

The article explains GRPO, a reinforcement‑learning algorithm that extends PPO with group sampling, no value network, dual penalties and KL regularisation, showing how it improves efficiency and stability when fine‑tuning large language models such as DeepSeek‑Math and DeepSeek‑R1.
DeepSeekGRPOPPO
0 likes · 6 min read

This comprehensive guide examines large‑scale deep reinforcement learning, detailing policy‑gradient fundamentals, the mathematics of PPO and GAE, practical implementation tricks, reward and observation normalization, network initialization, and the newer Phasic Policy Gradient method, all supported by code snippets and key research references.
Deep RLGAEPPG
0 likes · 19 min read

DeepSeek‑R1 replaces the traditional PPO‑based RLHF approach with GRPO, reducing reliance on human‑labeled data by using pure reinforcement learning environments and carefully designed reward mechanisms; the article explains reinforcement learning fundamentals, compares PPO, DPO and GRPO, and offers practical application recommendations.
AI AlignmentDPOGRPO
0 likes · 14 min read

The article reviews reinforcement-learning fundamentals and the progression from policy-gradient to PPO, then introduces Group Relative Policy Optimization (GRPO)—a critic-free method that normalizes rewards across multiple sampled outputs to compute group-relative advantages—and shows how DeepSeek-R1 leverages GRPO with rule-based rewards to achieve strong reasoning performance.
GRPOPPOPolicy Optimization
0 likes · 16 min read


DevOpsFeb 23, 2025 · Artificial Intelligence
This article explains how DeepSeek‑R1‑Zero uses group‑relative policy optimization (GRPO) to enhance inference without labeled data, introduces reinforcement learning with human feedback (RLHF) and its components, and compares the PPO and GRPO algorithms, highlighting their suitable engineering scenarios and practical implications for AI applications.
AI model trainingGRPOPPO
0 likes · 15 min read

This article reviews a new risk‑sensitive trading agent that combines reinforcement learning with large language models to extract stock recommendations and news‑based risk scores, describes the extended CVaR‑PPO algorithm, presents extensive experiments on the FNSPID dataset, and discusses the resulting performance gains and future work.
CVaRDeepSeekFinRL
0 likes · 10 min read

Xiaohongshu’s team unveiled a self‑developed RLHF system that trains multimodal large language models using heterogeneous and homogeneous network architectures, extensive PPO optimizations, and Medusa speculative sampling, achieving over 50% throughput gains, reduced hardware needs, and 5‑20% performance improvements on zero‑shot benchmarks.
PPOPRMRLHF
0 likes · 21 min read

This article provides a thorough, step‑by‑step explanation of reinforcement‑learning theory—covering policy‑based objectives, value‑function definitions, the derivation of policy gradients, actor‑critic architecture, advantage estimation, importance sampling, GAE, and the PPO algorithm—aimed at readers with little prior RL knowledge.
Advantage EstimationPPOactor-critic
0 likes · 31 min read

This article analytically explores the implicit assumptions behind the RLHF optimization objective, examines how they limit DPO and PPO methods, and proposes practical improvements such as rejection sampling and online on‑policy data selection to narrow the gap between theory and practice.
AI AlignmentDPOPPO
0 likes · 22 min read

This article provides a thorough, four‑part overview of RLHF for large language models, covering preference‑optimization algorithms (PPO‑based and offline RL approaches), reward‑model training techniques, inference‑time exploration strategies, and practical implementation details including the OpenRLHF framework and resource‑allocation tricks.
DPOLLM OptimizationOpenRLHF
0 likes · 27 min read


NewBeeNLPSep 23, 2024 · Artificial Intelligence
This article analyzes recent post‑training trends in large language models, comparing DPO and PPO, examining the scarcity of open‑source preference data, the iterative training process, the rise of synthetic data pipelines, and emerging methods for improving math and reasoning capabilities.
DPOLLMPPO
0 likes · 12 min read

This tutorial explains how to set up a multi‑agent reinforcement learning (MARL) environment with VMAS, install required dependencies, configure PPO hyper‑parameters, build policy and critic networks, collect data with TorchRL, and run a training loop to train agents for coordinated navigation tasks.
Multi-Agent Reinforcement LearningPPOTorchRL
0 likes · 10 min read

This article surveys the current RLHF ecosystem, comparing on‑policy methods like PPO with off‑policy approaches such as DPO, and examines recent variants—including ReMax, GRPO, DPOP, TDPO, and ORPO—highlighting their algorithmic differences, resource trade‑offs, and practical performance insights.
DPOLLMPPO
0 likes · 23 min read


NewBeeNLPApr 1, 2024 · Artificial Intelligence
This article provides a detailed technical walkthrough of Llama 2's Reinforcement Learning with Human Feedback pipeline, covering human preference data collection, reward‑model design and training, iterative fine‑tuning with PPO and rejection sampling, the Ghost Attention technique for multi‑turn consistency, and the resulting experimental evaluations.
Ghost AttentionLlama 2PPO
0 likes · 18 min read

This article explains how a deep reinforcement learning model, built on a graph‑based representation of urban elements and trained with PPO, can automate land‑use and road planning to achieve Service, Ecology, and Traffic objectives for 15‑minute city neighborhoods.
15-minute cityGraph Neural NetworkPPO
0 likes · 9 min read

This article explains why Reinforcement Learning from Human Feedback (RLHF) is crucial for LLM intelligence, outlines the three-stage training pipeline, details InstructGPT's reward model and PPO optimization, and provides a practical guide to implementing RLHF with deep‑learning frameworks.
PPORLHFReward Modeling
0 likes · 17 min read

This article provides a comprehensive overview of reinforcement learning, covering fundamental concepts, differences from supervised learning, algorithm families, policy gradient methods, practical tricks like baselines and reward‑to‑go, and detailed explanations of TRPO and PPO variants with illustrative diagrams.
PPOactor-criticmachine learning
0 likes · 19 min read

The article examines why rising RM scores in large‑model training don't ensure superior LLM performance and presents seven practical RLHF tricks—ranging from KL‑penalty to global gradient clipping—that improve PPO stability and reduce resource overhead.
LLM trainingPPORLHF
0 likes · 7 min read

This article demonstrates how to use Reinforcement Learning from Human Feedback (RLHF) with a PPO algorithm and a sentiment‑analysis model to train a language model that generates positive product reviews, covering task definition, data sampling, reward evaluation, model optimization, and experimental results.
GPTLanguage ModelPPO
0 likes · 11 min read


ArchitectFeb 19, 2023 · Artificial Intelligence
This article demonstrates how to apply Reinforcement Learning from Human Feedback (RLHF) using a sentiment‑analysis model as a reward function and Proximal Policy Optimization (PPO) to fine‑tune a language model that generates positive product reviews, complete with code snippets and experimental results.
Language ModelPPORLHF
0 likes · 10 min read

ChatGPT, OpenAI’s latest language model, builds on GPT‑3 but uses supervised fine‑tuning and Reinforcement Learning from Human Feedback (RLHF) to improve alignment, yet its training methods still cause consistency issues such as invalid help, hallucinations, bias, and limited explainability.
ChatGPTPPORLHF
0 likes · 17 min read

ChatGPT, OpenAI’s latest language model, builds on GPT‑3 using supervised fine‑tuning and Reinforcement Learning from Human Feedback (RLHF) with PPO, addressing consistency issues by aligning model outputs with human preferences, while discussing training methods, limitations, and evaluation metrics.
AI AlignmentChatGPTPPO
0 likes · 15 min read

This article explains how ChatGPT builds on GPT‑3 with improved accuracy and coherence, details its training pipeline that combines supervised fine‑tuning and Reinforcement Learning from Human Feedback (RLHF), discusses consistency challenges, evaluation metrics, and the limitations of the RLHF approach.
AI AlignmentChatGPTPPO
0 likes · 15 min read


ArchitectFeb 6, 2023 · Artificial Intelligence
This article explains the underlying mechanisms of ChatGPT, including its GPT‑3 foundation, the role of supervised fine‑tuning, human‑feedback reinforcement learning (RLHF), PPO optimization, consistency issues, evaluation metrics, and the limitations of these training strategies, with references to key research papers.
AI AlignmentChatGPTPPO
0 likes · 16 min read

The article outlines ChatGPT’s conversational capabilities, its InstructGPT‑based architecture, a three‑stage RLHF training pipeline involving supervised fine‑tuning, human‑ranked response generation, and PPO optimization, and discusses its strengths, limitations, diverse applications, and future directions for multimodal, up‑to‑date assistants.
AI applicationsChatGPTPPO
0 likes · 18 min read

This tutorial demonstrates how to use Huawei Cloud ModelArts and the Proximal Policy Optimization (PPO) reinforcement‑learning algorithm to train an AI agent that can clear most Super Mario levels within about 1500 episodes, even for users with no coding experience.
AIModelArtsPPO
0 likes · 6 min read

This article introduces model‑free reinforcement learning fundamentals, reviews mainstream solution methods such as Monte‑Carlo, Temporal‑Difference, and n‑step TD with eligibility traces, discusses their application in online advertising and presents Tencent's game advertising practice, including algorithm choices, reward design, and experimental results.
A3CAdvertisingPPO
0 likes · 17 min read

This article presents a comprehensive study of using reinforcement learning to improve e‑commerce product ranking, covering the limitations of traditional scoring, the design of context‑aware models, a pointer‑network based sequence generator, various RL algorithms, extensive offline evaluations, and successful online deployment with future research directions.
PPOdeep learninge-commerce
0 likes · 28 min read
