RLHF Techniques and Challenges in Large Language Models and Multimodal Applications

The presentation begins with an overview of Reinforcement Learning (RL) and Reinforcement Learning from Human Feedback (RLHF), explaining core concepts such as states, actions, rewards, and the Markov Decision Process, and clarifying common misconceptions about RL versus supervised learning.

It then details RLHF pipelines used in recent large language models, describing InstructGPT and ChatGPT’s three‑stage training (SFT, reward‑model training, and PPO‑based RLHF), the role of KL‑divergence adjustments, and the performance gains achieved by these modifications.

Subsequent sections introduce Constitutional AI, which combines supervised self‑critique with a reward model, and compare RLHF with the newer RLAIF approach that replaces human‑annotated ratings with model‑generated scores, dramatically reducing annotation costs.

The article surveys RLHF implementations in LLaMA 2 (Reject Sampling + PPO/Reward Model) and LLaMA 3 (Reject Sampling + DPO/Reward Model), explaining the shift from PPO to the lighter DPO algorithm and the iterative sampling process used to generate high‑quality data.

It also covers the concept of Super Alignment, a strategy aiming to surpass human performance by leveraging strong pre‑training, weak‑to‑strong generalization, and iterative RLHF loops.

Recent multimodal advancements are discussed, including GPT‑4o’s full‑multimodal input/output capabilities, OpenAI’s o1 model that emphasizes inference‑heavy reasoning, and fast‑hand’s own alignment methods for both large language models and multimodal models such as LLaVA.

For multimodal alignment, the talk presents a self‑training framework (TSO) that iteratively refines preference data using scaled preference optimization, dual‑clip reward loss, and mini‑batch updates, and describes how these techniques improve diversity, adaptability, and validation metrics.

Evaluation strategies are proposed that incorporate vision‑based scoring to avoid over‑reliance on textual ground truth, and extensive experimental results on benchmarks (AlignBench, MT‑Bench, AlpacaEval‑v2, Arena‑Hard, and various VQA datasets) demonstrate consistent gains both in‑domain and out‑of‑domain.

Finally, the presentation summarizes ablation studies on model‑matrix preference generation, iterative DPO, and dual‑clip loss, confirming the robustness of the proposed methods across a range of tasks and model sizes.