BestHub
Sign in
Tagged articles

PPO

54 articles · Page 1 of 1
Wu Shixiong's Large Model Academy
Wu Shixiong's Large Model Academy
Sep 29, 2026 · Interview Experience

GRPO Interview Mastery: From Critic-Free Design to Collapse Detection & Reward Hacking

This article breaks down six high-frequency GRPO interview questions from top Chinese tech companies, covering GRPO vs PPO trade-offs, group-relative advantage calculation with concrete numbers, handling all-correct/all-wrong sample groups, KL constraint mechanics, convergence monitoring priorities, and reward hacking detection via shadow evaluation.

Advantage EstimationGRPOInterview Preparation
0 likes · 19 min read
GRPO Interview Mastery: From Critic-Free Design to Collapse Detection & Reward Hacking
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Sep 22, 2026 · Artificial Intelligence

Scaling LLM RL: How Batch Size Affects Training Speed and Efficiency

The article explores how batch size scaling impacts LLM reinforcement learning training efficiency, introducing critical generation and training batch concepts, showing optimal batch size balances sample efficiency and system throughput, with experiments achieving 29% time reduction on fixed hardware.

Batch Size ScalingCritical BatchGRPO
0 likes · 36 min read
Scaling LLM RL: How Batch Size Affects Training Speed and Efficiency
DeepHub IMBA
DeepHub IMBA
Aug 2, 2026 · Artificial Intelligence

Building a From‑Scratch LLM Training Framework: Full GRPO vs PPO vs DPO Comparison on GSM8K

The article presents a from‑scratch LLM training framework called grpo‑llm, implements GRPO with Trio rollout, FSDP and a C++ reward extension, and conducts a controlled experiment comparing GRPO, PPO and DPO on the GSM8K math‑reasoning benchmark, revealing why DPO outperforms the other two under sparse binary rewards.

DPOGRPOGSM8K
0 likes · 10 min read
Building a From‑Scratch LLM Training Framework: Full GRPO vs PPO vs DPO Comparison on GSM8K
Machine Heart
Machine Heart
Jun 21, 2026 · Artificial Intelligence

Why the Once‑Rejected PPO Algorithm Became a Pillar of Modern LLM Training

The article recounts how Proximal Policy Optimization, initially dismissed by NeurIPS 2017 for limited novelty, later became a cornerstone of RLHF and large‑language‑model training, illustrating how academic evaluation can miss long‑term impact, with parallels to other once‑rejected breakthroughs such as LSTM, SIFT and Dropout.

Algorithm RejectionNeurIPSPPO
0 likes · 5 min read
Why the Once‑Rejected PPO Algorithm Became a Pillar of Modern LLM Training
Machine Heart
Machine Heart
Jun 21, 2026 · Artificial Intelligence

Is GRPO Obsolete? Why GLM‑5.2 Dropped It and What It Means for RL

GLM‑5.2 replaces the Group Relative Policy Optimization (GRPO) algorithm with a critic‑based PPO approach for long‑horizon tasks, arguing that GRPO’s group comparison breaks down on variable‑length trajectories, a shift that has sparked vigorous debate across the reinforcement‑learning community.

DeepSeekGLM-5.2GRPO
0 likes · 10 min read
Is GRPO Obsolete? Why GLM‑5.2 Dropped It and What It Means for RL
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jun 10, 2026 · Artificial Intelligence

Beyond Orchestrating Workflows: How UnityMAS-O Trains LLM-Based Multi‑Agent Systems

UnityMAS‑O introduces a general reinforcement‑learning framework that converts predefined LLM multi‑agent workflows into trainable tasks, enabling credit assignment across roles, supporting parameter‑sharing configurations, and demonstrating significant F1 and test‑pass improvements on QA and code‑generation benchmarks.

LLMMulti-Agent Reinforcement LearningPPO
0 likes · 12 min read
Beyond Orchestrating Workflows: How UnityMAS-O Trains LLM-Based Multi‑Agent Systems
Machine Heart
Machine Heart
May 29, 2026 · Artificial Intelligence

DiffusionOPD: A New Online Policy Distillation Paradigm for Multi‑Task Diffusion Models

DiffusionOPD introduces a unified on‑policy distillation framework for diffusion models that decouples single‑task online policy exploration from multi‑task capability integration, training expert teachers per task and distilling their skills into a single student model, achieving faster convergence and higher performance across composition, OCR, and aesthetic tasks.

KL DivergenceOn-Policy DistillationPPO
0 likes · 8 min read
DiffusionOPD: A New Online Policy Distillation Paradigm for Multi‑Task Diffusion Models
DeepHub IMBA
DeepHub IMBA
May 19, 2026 · Artificial Intelligence

A 2026 Survey of LLM‑Focused RL: From PPO to DPO, GRPO, and Multi‑Agent RL

The article reviews five years of LLM‑centric reinforcement learning, tracing the evolution from early Q‑learning to PPO, then to Direct Preference Optimization, Group Relative Policy Optimization, and finally multi‑agent RL, detailing each method’s mechanics, strengths, failure modes, practical considerations, and emerging open‑source toolchains.

DPOGRPOLLM Alignment
0 likes · 33 min read
A 2026 Survey of LLM‑Focused RL: From PPO to DPO, GRPO, and Multi‑Agent RL
Machine Heart
Machine Heart
May 1, 2026 · Artificial Intelligence

From PPO to MaxRL: The Evolution of Reinforcement Learning for LLM Inference

This article surveys the rapid evolution of reinforcement‑learning algorithms for large‑language‑model inference from early REINFORCE and PPO to newer approaches such as GRPO, RLOO, DAPO, CISPO, DPPO, ScaleRL and MaxRL, highlighting their design motivations, mathematical formulations, empirical trade‑offs and open research challenges.

GRPOLLMMaxRL
0 likes · 27 min read
From PPO to MaxRL: The Evolution of Reinforcement Learning for LLM Inference
HyperAI Super Neural
HyperAI Super Neural
Apr 23, 2026 · Artificial Intelligence

Task Tokens Cut Per-Task Trainable Parameters 125× and Boost Convergence 6× for Embodied AI

The Task Tokens method introduced by an Israeli research team reduces the number of trainable parameters per task by up to 125‑fold and speeds up convergence by six times, while preserving the flexibility of Behavior Foundation Models and demonstrating strong performance, robustness, and compatibility across a suite of embodied control tasks.

Behavior Foundation ModelsEmbodied AIMulti-Modal Prompting
0 likes · 13 min read
Task Tokens Cut Per-Task Trainable Parameters 125× and Boost Convergence 6× for Embodied AI
Bighead's Algorithm Notes
Bighead's Algorithm Notes
Apr 22, 2026 · Artificial Intelligence

How DeepAries’s Adaptive Rebalancing Timing Boosts Portfolio Returns

DeepAries is a novel deep reinforcement‑learning framework that jointly learns when to rebalance a portfolio and how to allocate assets by combining a Transformer‑based state encoder with PPO, and extensive experiments on four major markets show it significantly outperforms fixed‑frequency baselines in risk‑adjusted return, transaction cost, and drawdown.

DeepAriesPPOReinforcement Learning
0 likes · 15 min read
How DeepAries’s Adaptive Rebalancing Timing Boosts Portfolio Returns
Wu Shixiong's Large Model Academy
Wu Shixiong's Large Model Academy
Apr 16, 2026 · Interview Experience

Turn Memorized Answers into Deep Understanding for Tech Interviews

This article explains why interviewers use seemingly rote questions to probe a candidate's true grasp of concepts, contrasts memorization with genuine understanding using PPO vs GRPO, and provides a practical three‑question framework and dialogue examples to help candidates answer technical principle questions confidently.

Answering TechniquesGRPOPPO
0 likes · 12 min read
Turn Memorized Answers into Deep Understanding for Tech Interviews
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Mar 17, 2026 · Artificial Intelligence

MIT Study Shows Adding Noise to Large Models Can Replace GRPO/PPO Tuning

A new MIT paper reveals that pretrained large models already contain many hidden expert submodels, and that a simple one‑step Gaussian perturbation (RandOpt) can locate and ensemble these experts to achieve performance comparable to or better than traditional GRPO/PPO tuning, especially as model size grows.

GRPOPPORandOpt
0 likes · 9 min read
MIT Study Shows Adding Noise to Large Models Can Replace GRPO/PPO Tuning
360 Smart Cloud
360 Smart Cloud
Nov 14, 2025 · Artificial Intelligence

How TLM Platform Powers LLM Ops with PPO, GRPO and Reinforcement Evaluators

The article introduces the TLM large‑model development platform, details its fine‑tuning options, explains reinforcement learning fundamentals and key algorithms such as PPO and the newer GRPO, describes the architecture of a reinforcement evaluator, and shows how to configure RL training on the platform.

AI platformGRPOLLMOps
0 likes · 10 min read
How TLM Platform Powers LLM Ops with PPO, GRPO and Reinforcement Evaluators
Data Party THU
Data Party THU
Nov 10, 2025 · Artificial Intelligence

Which Neural Network Method Best Estimates Uncertainty in Regression? A Comparative Study

This article examines why regression models need uncertainty estimates, explains aleatoric and epistemic uncertainty, compares four neural‑network approaches (Mean + LogStd, Mean + LogVariance, MC Dropout, simplified PPO) on a concrete‑strength dataset, and analyzes their experimental performance and limitations.

Monte Carlo DropoutPPOUncertainty Estimation
0 likes · 10 min read
Which Neural Network Method Best Estimates Uncertainty in Regression? A Comparative Study
Zhuanzhuan Tech
Zhuanzhuan Tech
Oct 29, 2025 · Artificial Intelligence

How Reinforcement Learning Boosts Stability and Speed in LLM QA Systems

This article examines how reinforcement‑learning techniques such as PPO, DPO, and GRPO are integrated into the Baixiaosheng QA system to improve answer stability, deepen domain knowledge understanding, and accelerate response generation, and it evaluates the impact of Reinforcement Fine‑Tuning (RFT) on real‑world performance.

AIDPOGRPO
0 likes · 16 min read
How Reinforcement Learning Boosts Stability and Speed in LLM QA Systems
Fun with Large Models
Fun with Large Models
Sep 24, 2025 · Artificial Intelligence

Interview Guide: Core Differences Between PPO and GRPO Algorithms for Large Model Fine‑Tuning

The article explains the fundamental principles of PPO and GRPO reinforcement‑learning algorithms, compares their architectures and training workflows, highlights why GRPO is gaining traction in large‑model fine‑tuning, discusses associated risks, and offers practical guidance on group size selection for engineers preparing for interviews.

GRPOPPORLHF
0 likes · 9 min read
Interview Guide: Core Differences Between PPO and GRPO Algorithms for Large Model Fine‑Tuning
Data Party THU
Data Party THU
Sep 4, 2025 · Artificial Intelligence

Unraveling PPO Variants: From GRPO to DAPO and GSPO – A Deep Dive

This article provides a comprehensive technical analysis of PPO‑based reinforcement learning methods for large language models, detailing the evolution from the original PPO algorithm through GRPO, DAPO, and GSPO, and explaining their motivations, mathematical formulations, advantages, and practical challenges such as entropy collapse and importance‑sampling variance.

DAPOGRPOGSPO
0 likes · 30 min read
Unraveling PPO Variants: From GRPO to DAPO and GSPO – A Deep Dive
Sohu Tech Products
Sohu Tech Products
Sep 3, 2025 · Artificial Intelligence

How GRPO Revolutionizes RLHF for Large Language Models

This article explains the motivation, mathematical foundations, implementation details, advantages, experimental results, and future directions of Group Relative Policy Optimization (GRPO), a novel reinforcement‑learning algorithm that replaces PPO’s value network with efficient group‑wise relative evaluation for large language models.

Artificial IntelligenceGRPOLLM
0 likes · 17 min read
How GRPO Revolutionizes RLHF for Large Language Models
Baobao Algorithm Notes
Baobao Algorithm Notes
Aug 15, 2025 · Artificial Intelligence

Unlocking LLM Performance: Classic Deep RL Tricks Reimagined for Modern Training

This article systematically adapts classic deep reinforcement‑learning techniques—such as multi‑step returns, TD(λ)/GAE, V‑trace corrections, uncertainty‑aware weighting, safety constraints, distribution‑robust optimization, and value‑guided decoding—to improve large language model training and inference, providing concrete formulas, implementation tips, and empirical results.

Deep RLGAELLM
0 likes · 17 min read
Unlocking LLM Performance: Classic Deep RL Tricks Reimagined for Modern Training
AI Algorithm Path
AI Algorithm Path
Jul 27, 2025 · Artificial Intelligence

Understanding RLHF: How Human Feedback Trains Modern LLMs

This article explains the RLHF (Reinforcement Learning from Human Feedback) pipeline that powers ChatGPT and other large language models, covering the limitations of traditional fine‑tuning, the creation of human‑feedback datasets, reward‑model training, loss design, and the final PPO‑based fine‑tuning step.

ChatGPTHuman FeedbackPPO
0 likes · 8 min read
Understanding RLHF: How Human Feedback Trains Modern LLMs
AI Algorithm Path
AI Algorithm Path
Apr 13, 2025 · Artificial Intelligence

Understanding GRPO: Group Relative Policy Optimization for LLM Training

The article explains GRPO, a reinforcement‑learning algorithm that extends PPO with group sampling, no value network, dual penalties and KL regularisation, showing how it improves efficiency and stability when fine‑tuning large language models such as DeepSeek‑Math and DeepSeek‑R1.

DeepSeekGRPOPPO
0 likes · 6 min read
Understanding GRPO: Group Relative Policy Optimization for LLM Training
Baobao Algorithm Notes
Baobao Algorithm Notes
Mar 20, 2025 · Artificial Intelligence

Unlocking Large‑Scale Deep Reinforcement Learning: PPO, GAE, and PPG Deep Dive

This comprehensive guide examines large‑scale deep reinforcement learning, detailing policy‑gradient fundamentals, the mathematics of PPO and GAE, practical implementation tricks, reward and observation normalization, network initialization, and the newer Phasic Policy Gradient method, all supported by code snippets and key research references.

Deep RLGAEPPG
0 likes · 19 min read
Unlocking Large‑Scale Deep Reinforcement Learning: PPO, GAE, and PPG Deep Dive
Data Thinking Notes
Data Thinking Notes
Mar 16, 2025 · Artificial Intelligence

Why DeepSeek R1 Swaps PPO for GRPO: A Deep Dive into RLHF Alternatives

DeepSeek‑R1 replaces the traditional PPO‑based RLHF approach with GRPO, reducing reliance on human‑labeled data by using pure reinforcement learning environments and carefully designed reward mechanisms; the article explains reinforcement learning fundamentals, compares PPO, DPO and GRPO, and offers practical application recommendations.

AI AlignmentDPOGRPO
0 likes · 14 min read
Why DeepSeek R1 Swaps PPO for GRPO: A Deep Dive into RLHF Alternatives
Tencent Technical Engineering
Tencent Technical Engineering
Feb 24, 2025 · Artificial Intelligence

Understanding GRPO: Group Relative Policy Optimization in Reinforcement Learning and Large Language Models

The article reviews reinforcement-learning fundamentals and the progression from policy-gradient to PPO, then introduces Group Relative Policy Optimization (GRPO)—a critic-free method that normalizes rewards across multiple sampled outputs to compute group-relative advantages—and shows how DeepSeek-R1 leverages GRPO with rule-based rewards to achieve strong reasoning performance.

GRPOPPORLHF
0 likes · 16 min read
Understanding GRPO: Group Relative Policy Optimization in Reinforcement Learning and Large Language Models
DevOps
DevOps
Feb 23, 2025 · Artificial Intelligence

Understanding Reinforcement Learning, RLHF, PPO and GRPO for AI Applications

This article explains how DeepSeek‑R1‑Zero uses group‑relative policy optimization (GRPO) to enhance inference without labeled data, introduces reinforcement learning with human feedback (RLHF) and its components, and compares the PPO and GRPO algorithms, highlighting their suitable engineering scenarios and practical implications for AI applications.

AI model trainingGRPOPPO
0 likes · 15 min read
Understanding Reinforcement Learning, RLHF, PPO and GRPO for AI Applications
Bighead's Algorithm Notes
Bighead's Algorithm Notes
Feb 15, 2025 · Artificial Intelligence

FinRL‑DeepSeek: How Integrating DeepSeek with RL Improves Portfolio Returns (Code Open‑Source)

This article reviews a new risk‑sensitive trading agent that combines reinforcement learning with large language models to extract stock recommendations and news‑based risk scores, describes the extended CVaR‑PPO algorithm, presents extensive experiments on the FNSPID dataset, and discusses the resulting performance gains and future work.

Algorithmic TradingCVaRDeepSeek
0 likes · 10 min read
FinRL‑DeepSeek: How Integrating DeepSeek with RL Improves Portfolio Returns (Code Open‑Source)
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Jan 2, 2025 · Artificial Intelligence

Xiaohongshu's Self-developed RLHF System for Multimodal Large Language Models: Design, Optimization, and Performance

Xiaohongshu’s team unveiled a self‑developed RLHF system that trains multimodal large language models using heterogeneous and homogeneous network architectures, extensive PPO optimizations, and Medusa speculative sampling, achieving over 50% throughput gains, reduced hardware needs, and 5‑20% performance improvements on zero‑shot benchmarks.

Distributed TrainingPPOPRM
0 likes · 21 min read
Xiaohongshu's Self-developed RLHF System for Multimodal Large Language Models: Design, Optimization, and Performance
Baobao Algorithm Notes
Baobao Algorithm Notes
Nov 18, 2024 · Artificial Intelligence

Demystifying Actor‑Critic and PPO: From Policy Gradients to Practical RL

This article provides a thorough, step‑by‑step explanation of reinforcement‑learning theory—covering policy‑based objectives, value‑function definitions, the derivation of policy gradients, actor‑critic architecture, advantage estimation, importance sampling, GAE, and the PPO algorithm—aimed at readers with little prior RL knowledge.

Advantage EstimationPPOactor-critic
0 likes · 31 min read
Demystifying Actor‑Critic and PPO: From Policy Gradients to Practical RL
Baobao Algorithm Notes
Baobao Algorithm Notes
Oct 22, 2024 · Artificial Intelligence

Uncovering Hidden Assumptions in RLHF: Theory, DPO & PPO Pitfalls

This article analytically explores the implicit assumptions behind the RLHF optimization objective, examines how they limit DPO and PPO methods, and proposes practical improvements such as rejection sampling and online on‑policy data selection to narrow the gap between theory and practice.

AI AlignmentDPOPPO
0 likes · 22 min read
Uncovering Hidden Assumptions in RLHF: Theory, DPO & PPO Pitfalls
Baobao Algorithm Notes
Baobao Algorithm Notes
Oct 21, 2024 · Artificial Intelligence

Unraveling RLHF: From PPO to DPO and Beyond – A Comprehensive Guide

This article provides a thorough, four‑part overview of RLHF for large language models, covering preference‑optimization algorithms (PPO‑based and offline RL approaches), reward‑model training techniques, inference‑time exploration strategies, and practical implementation details including the OpenRLHF framework and resource‑allocation tricks.

DPOLLM OptimizationOpenRLHF
0 likes · 27 min read
Unraveling RLHF: From PPO to DPO and Beyond – A Comprehensive Guide
Python Programming Learning Circle
Python Programming Learning Circle
Sep 10, 2024 · Artificial Intelligence

Using TorchRL to Implement Multi‑Agent PPO for MARL

This tutorial explains how to set up a multi‑agent reinforcement learning (MARL) environment with VMAS, install required dependencies, configure PPO hyper‑parameters, build policy and critic networks, collect data with TorchRL, and run a training loop to train agents for coordinated navigation tasks.

Multi-Agent Reinforcement LearningPPOTorchRL
0 likes · 10 min read
Using TorchRL to Implement Multi‑Agent PPO for MARL
Baobao Algorithm Notes
Baobao Algorithm Notes
May 30, 2024 · Artificial Intelligence

What’s the Latest RLHF Landscape? From PPO to ORPO Explained

This article surveys the current RLHF ecosystem, comparing on‑policy methods like PPO with off‑policy approaches such as DPO, and examines recent variants—including ReMax, GRPO, DPOP, TDPO, and ORPO—highlighting their algorithmic differences, resource trade‑offs, and practical performance insights.

DPOLLMPPO
0 likes · 23 min read
What’s the Latest RLHF Landscape? From PPO to ORPO Explained
NewBeeNLP
NewBeeNLP
Apr 1, 2024 · Artificial Intelligence

How Llama 2 Uses RLHF, PPO, Rejection Sampling, and Ghost Attention

This article provides a detailed technical walkthrough of Llama 2's Reinforcement Learning with Human Feedback pipeline, covering human preference data collection, reward‑model design and training, iterative fine‑tuning with PPO and rejection sampling, the Ghost Attention technique for multi‑turn consistency, and the resulting experimental evaluations.

Ghost AttentionLlama 2PPO
0 likes · 18 min read
How Llama 2 Uses RLHF, PPO, Rejection Sampling, and Ghost Attention
Baobao Algorithm Notes
Baobao Algorithm Notes
Oct 9, 2023 · Artificial Intelligence

Demystifying RLHF and PPO for Large Language Models: Theory and Practice

This article explains why Reinforcement Learning from Human Feedback (RLHF) is crucial for LLM intelligence, outlines the three-stage training pipeline, details InstructGPT's reward model and PPO optimization, and provides a practical guide to implementing RLHF with deep‑learning frameworks.

Artificial IntelligencePPORLHF
0 likes · 17 min read
Demystifying RLHF and PPO for Large Language Models: Theory and Practice
Baidu Geek Talk
Baidu Geek Talk
Aug 16, 2023 · Artificial Intelligence

Understanding Reinforcement Learning: From Basics to PPO and Policy Gradient

This article provides a comprehensive overview of reinforcement learning, covering fundamental concepts, differences from supervised learning, algorithm families, policy gradient methods, practical tricks like baselines and reward‑to‑go, and detailed explanations of TRPO and PPO variants with illustrative diagrams.

PPOReinforcement Learningactor-critic
0 likes · 19 min read
Understanding Reinforcement Learning: From Basics to PPO and Policy Gradient
IT Architects Alliance
IT Architects Alliance
Feb 23, 2023 · Artificial Intelligence

Training a Positive Review Generator with RLHF and PPO

This article demonstrates how to use Reinforcement Learning from Human Feedback (RLHF) with a PPO algorithm and a sentiment‑analysis model to train a language model that generates positive product reviews, covering task definition, data sampling, reward evaluation, model optimization, and experimental results.

GPTPPORLHF
0 likes · 11 min read
Training a Positive Review Generator with RLHF and PPO
Architect
Architect
Feb 19, 2023 · Artificial Intelligence

Training a Positive Review Generator with RLHF and PPO

This article demonstrates how to apply Reinforcement Learning from Human Feedback (RLHF) using a sentiment‑analysis model as a reward function and Proximal Policy Optimization (PPO) to fine‑tune a language model that generates positive product reviews, complete with code snippets and experimental results.

PPORLHFReinforcement Learning
0 likes · 10 min read
Training a Positive Review Generator with RLHF and PPO
dbaplus Community
dbaplus Community
Feb 18, 2023 · Artificial Intelligence

Why ChatGPT Still Gets It Wrong: Inside RLHF and Model Consistency

ChatGPT, OpenAI’s latest language model, builds on GPT‑3 but uses supervised fine‑tuning and Reinforcement Learning from Human Feedback (RLHF) to improve alignment, yet its training methods still cause consistency issues such as invalid help, hallucinations, bias, and limited explainability.

ChatGPTPPORLHF
0 likes · 17 min read
Why ChatGPT Still Gets It Wrong: Inside RLHF and Model Consistency
IT Architects Alliance
IT Architects Alliance
Feb 9, 2023 · Artificial Intelligence

How ChatGPT Works: Model Architecture, Training Strategies, and RLHF

ChatGPT, OpenAI’s latest language model, builds on GPT‑3 using supervised fine‑tuning and Reinforcement Learning from Human Feedback (RLHF) with PPO, addressing consistency issues by aligning model outputs with human preferences, while discussing training methods, limitations, and evaluation metrics.

AI AlignmentChatGPTPPO
0 likes · 15 min read
How ChatGPT Works: Model Architecture, Training Strategies, and RLHF
Architects' Tech Alliance
Architects' Tech Alliance
Feb 7, 2023 · Artificial Intelligence

ChatGPT: Technical Principles, Architecture, and the Role of Human‑Feedback Reinforcement Learning

This article explains how ChatGPT builds on GPT‑3 with improved accuracy and coherence, details its training pipeline that combines supervised fine‑tuning and Reinforcement Learning from Human Feedback (RLHF), discusses consistency challenges, evaluation metrics, and the limitations of the RLHF approach.

AI AlignmentChatGPTPPO
0 likes · 15 min read
ChatGPT: Technical Principles, Architecture, and the Role of Human‑Feedback Reinforcement Learning
Architect
Architect
Feb 6, 2023 · Artificial Intelligence

Understanding How ChatGPT Works: RLHF, PPO, and Consistency Challenges

This article explains the underlying mechanisms of ChatGPT, including its GPT‑3 foundation, the role of supervised fine‑tuning, human‑feedback reinforcement learning (RLHF), PPO optimization, consistency issues, evaluation metrics, and the limitations of these training strategies, with references to key research papers.

AI AlignmentChatGPTPPO
0 likes · 16 min read
Understanding How ChatGPT Works: RLHF, PPO, and Consistency Challenges
Tencent Cloud Developer
Tencent Cloud Developer
Dec 9, 2022 · Artificial Intelligence

An Overview of ChatGPT: Technology, Training Process, and Applications

The article outlines ChatGPT’s conversational capabilities, its InstructGPT‑based architecture, a three‑stage RLHF training pipeline involving supervised fine‑tuning, human‑ranked response generation, and PPO optimization, and discusses its strengths, limitations, diverse applications, and future directions for multimodal, up‑to‑date assistants.

AI applicationsChatGPTPPO
0 likes · 18 min read
An Overview of ChatGPT: Technology, Training Process, and Applications
Huawei Cloud Developer Alliance
Huawei Cloud Developer Alliance
Jun 1, 2022 · Artificial Intelligence

How AI Beats Super Mario with PPO in 5 Minutes

This tutorial demonstrates how to use Huawei Cloud ModelArts and the Proximal Policy Optimization (PPO) reinforcement‑learning algorithm to train an AI agent that can clear most Super Mario levels within about 1500 episodes, even for users with no coding experience.

AIModelArtsPPO
0 likes · 6 min read
How AI Beats Super Mario with PPO in 5 Minutes
IEG Growth Platform Technology Team
IEG Growth Platform Technology Team
Dec 6, 2021 · Artificial Intelligence

Model-Free Reinforcement Learning for ROI Optimization: Methods, Advertising Applications, and Tencent Game Advertising Practice

This article introduces model‑free reinforcement learning fundamentals, reviews mainstream solution methods such as Monte‑Carlo, Temporal‑Difference, and n‑step TD with eligibility traces, discusses their application in online advertising and presents Tencent's game advertising practice, including algorithm choices, reward design, and experimental results.

A3CAdvertisingPPO
0 likes · 17 min read
Model-Free Reinforcement Learning for ROI Optimization: Methods, Advertising Applications, and Tencent Game Advertising Practice
DataFunTalk
DataFunTalk
Oct 4, 2020 · Artificial Intelligence

Reinforcement Learning for Product Ranking: Model Design, Experiments, and Online Deployment

This article presents a comprehensive study of using reinforcement learning to improve e‑commerce product ranking, covering the limitations of traditional scoring, the design of context‑aware models, a pointer‑network based sequence generator, various RL algorithms, extensive offline evaluations, and successful online deployment with future research directions.

PPOReinforcement Learningdeep learning
0 likes · 28 min read
Reinforcement Learning for Product Ranking: Model Design, Experiments, and Online Deployment