Can Large Language Models Self‑Evolve Beyond Math and Code?

From RLVR to RLSVR: Enabling Self‑Verifiable Rewards for Open‑Ended LLMs

Reinforcement Learning with Verifiable Rewards (RLVR) improves large‑model reasoning on math and code because standard answers and compilers provide deterministic, low‑cost feedback. Open‑ended tasks such as creative writing and research analysis lack unique answers, making existing reward models or LLM judges biased and costly.

RLSVR – Task Transformation for Self‑Verifiable Rewards

The authors propose RLSVR (Reinforcement Learning with Self‑Verifiable Rewards) . Inspired by self‑supervised learning, RLSVR converts an open‑ended task into a proxy environment that injects a hidden variable (e.g., perturbed input, removed information) and records it. The model still solves the original task, ensuring relevance, while the hidden state enables deterministic verification of the model’s judgment, yielding a self‑verifiable reward generated automatically by the environment.

SpyRL: A Self‑Play Instance of RLSVR

To demonstrate RLSVR, the authors design SpyRL (Self‑Play Reinforcement Learning) , an asymmetric multi‑agent game. Each round has two phases:

Performing : The environment samples an input, randomly assigns one player as the “spy” who receives a degraded version of the input, while other “civilian” players see the full input. All agents generate the same target output (summary, story, or solution).

Detection : All outputs are revealed; agents must identify the spy based on quality differences. Because the spy’s identity is pre‑recorded, correct detection provides a deterministic reward. The number of suspicion votes also serves as a performance reward for the Performing phase.

The reward scheme uses a zero‑sum signal for the spy vs. civilians and a GRPO‑style relative advantage for detection.

Experimental Results

SpyRL was evaluated on Qwen3‑4B and Qwen3‑8B across three task families and compared with self‑evolution baselines R‑Zero and Absolute Zero.

Summarization : On GovReport, ROUGE‑L rose from 30.2 to 36.7 (4B) and 29.0 to 34.1 (8B). Average A/B win rates reached 73.9% / 75.4%.

Creative Writing : On WritingPrompts and WritingBench, SpyRL achieved overall win rates above 80% against baselines, with notable gains in novelty and emotional expression.

Math & Reasoning : On seven benchmarks (GSM8K, Math500, AIME 2024/2025, Minerva, MMLU‑Pro, GPQA‑Diamond), SpyRL improved average scores by 8.97 pp (4B) and 6.16 pp (8B); e.g., AIME 2025 from 6.7 → 20.0, GPQA‑Diamond from 26.3 → 41.3.

Human Evaluation : Ten PhD students blind‑rated 400 creative‑writing samples; SpyRL’s win rates were 80.0% / 78.5% / 74.0% against the three baselines on WritingPrompts, and 85.0% / 80.5% / 72.0% on WritingBench.

Cross‑Domain Transfer : Training on PubMed papers boosted ROUGE‑L on arXiv, PubMed, and BillSum from 33.2 to 38.1, with an average A/B win rate of 70.2%.

Insights and Future Directions

RLSVR bridges self‑supervised learning and reinforcement learning: self‑supervision creates labels from data; RLSVR creates rewards from a designed environment. This reframes verification from a fixed task property to a resource that can be engineered. The authors suggest that future research should explore diverse task‑transformation strategies as a new lever for large‑model self‑improvement, alongside data, model architecture, and optimization algorithms.