What Is Reinforcement Fine-Tuning (RFT) and How Does It Supercharge LLMs?

Introduction

Reinforcement Fine‑Tuning (RFT), also called ReFT, augments supervised fine‑tuning (SFT) with reinforcement learning to improve a model’s reasoning on high‑quality task data and reference answers.

Core Idea

RFT uses a training set for SFT and a separate validation set for evaluation. After SFT, the model generates outputs that are scored by a grader comparing them to the correct answer, producing a reward in the range 0 to 1 that guides further optimization.

The grader returns 0 when the correct answer is absent, 1 when it appears in the first position, and intermediate values for partial matches.

Training Workflow

The pipeline consists of three stages:

Warm‑up (SFT) : Fine‑tune on (Question, Chain‑of‑Thought) pairs for 1–2 epochs to learn basic reasoning patterns.

Reinforcement Learning : Apply Proximal Policy Optimization (PPO). For each question the model samples multiple reasoning paths; the grader evaluates each path and provides a reward signal. This is analogous to self‑play in AlphaZero.

Evaluation : After each PPO iteration evaluate on the validation set to monitor generalization.

Data Format

Both training and validation data are stored as .jsonl files, one JSON object per line. In the gene‑prediction example each object contains three fields: a case report (patient information and symptoms), an instruction, and the correct answer (a list of candidate genes). During training the model only receives the case report and instruction; it must output a ranked list of genes.

Grader Configuration

The grader can be selected based on output format (list, free‑form text, etc.). Hyper‑parameters such as batch size, learning‑rate multiplier, and number of epochs are configurable.

Results

OpenAI’s internal experiments show that a smaller o1‑mini model fine‑tuned with RFT outperforms the larger baseline o1 on top‑1, top‑5, and top‑max metrics for gene‑prediction. On the GSM8K mathematical‑reasoning benchmark, ReFT improves accuracy of CodeLLAMA models by roughly 10 percentage points compared to pure SFT.

Related Work

The Reinforcement Learning Fine‑Tuning concept was introduced by ByteDance and presented at ACL 2024 under the name “ReFT: Reasoning with Reinforced Fine‑Tuning.” The paper (https://arxiv.org/pdf/2401.08967) describes a two‑stage pipeline (warm‑up SFT followed by PPO‑based RL) and reports consistent gains over SFT across multiple datasets.