Can Language Models Self‑Train Without Data? Inside the Language Self‑Play Framework

Background

Large language models (LLMs) have recently achieved near‑human or superhuman performance on instruction following and complex reasoning tasks, but their progress heavily relies on massive, high‑quality training data. Human‑generated data is ultimately limited, prompting the question of whether models can evolve without external data by "self‑practice" and "self‑challenge".

Language Self‑Play (LSP) Framework

Inspired by AlphaGo’s self‑play, Meta Superintelligence Labs and UC Berkeley propose a novel framework called Language Self‑Play (LSP). The framework lets a single LLM instantiate two roles: a Challenger that generates difficult queries and a Solver that answers them. The two agents engage in a competitive minimax game, driving continual improvement without any external data.

Roles

Challenger (π_Ch) : Generates a query q and aims to minimize the reward obtained by the Solver, i.e., to make the problem as hard as possible.

Solver (π_Sol) : Receives the query q and produces an answer a, aiming to maximize the reward R(q,a) from an external reward model or environment.

Advantage Computation

Solver advantage (A_Sol) : For each query q_i, compute the average reward of all answers as a baseline. The advantage of a specific answer is the difference between its reward and this baseline. Positive advantage encourages the model to increase the answer’s probability; negative advantage suppresses it.

Challenger advantage (A_Ch) : The difficulty of a query is measured by the Solver’s average score on that query. The baseline is the average score across all queries. A positive advantage means the query successfully drives the Solver’s score below the global average and is therefore rewarded; a negative advantage indicates a weak query.

Loss Functions and Model Update

The training uses a policy‑gradient objective with two components:

Policy term: (probability / stop‑gradient‑probability) * advantage. For the Solver, a positive advantage makes the term negative, causing gradient descent to increase the probability of the answer; the opposite holds for the Challenger.

KL regularization term: -β * log(current_probability / reference_probability), which penalizes divergence from a fixed reference model (the initial base). This term is crucial for the Challenger to prevent it from generating meaningless gibberish that merely lowers the Solver’s score.

Self‑Reward Extension (LSP)

The original zero‑sum version (LSP‑Zero) sometimes leads to degenerate behavior, such as the Solver always outputting code to trick the reward model. To mitigate this, the authors add a self‑reward mechanism: the model (or a reference model) evaluates each generated (query, answer) pair with a quality score from 0 to 7. This score is added to the Solver’s reward and, averaged, to the Challenger’s reward, turning the game into a cooperative improvement process.

Experimental Setup

Base model: Llama‑3.2‑3B‑Instruct . Baselines: GRPO (reinforcement learning with human data), LSP‑Zero (pure zero‑sum self‑play), and LSP (self‑play with self‑reward). Evaluation: AlpacaEval benchmark with GPT‑4o as the judge, measuring win rate over the base model. Two training scenarios were explored:

Training from the base model to test whether data‑free learning can reach the performance of data‑driven methods.

Continuing training from a GRPO‑fine‑tuned model to see if LSP can further improve an already trained model.

Results

Scenario 1: From Base Model

All methods significantly outperform the base model, confirming that reinforcement learning fine‑tuning is effective. LSP‑Zero achieves a win rate of 40.1% and LSP 40.6%, comparable to GRPO’s 40.9%, demonstrating that self‑play alone can match human‑data training. LSP slightly outperforms LSP‑Zero, showing the benefit of self‑reward.

Scenario 2: From GRPO Model

Applying LSP raises the overall win rate from 40.9% to 43.1%, with a notable jump on the Vicuna task (from 28.7% to 46.3%). LSP‑Zero, however, degrades performance to 40.0%, highlighting instability of a pure zero‑sum game in long‑term training. The authors note a drop on chat‑oriented tasks (e.g., Koala) because the Challenger learns to generate more structured, instruction‑like queries.

Conclusion

The paper demonstrates that large language models can achieve substantial performance gains without any external data by engaging in a self‑play game where the same model alternates between challenger and solver roles. The self‑reward extension further stabilizes training and yields improvements beyond data‑driven baselines, suggesting a promising direction toward autonomous, continual learning for AI systems.

Key Figures