Can Models Keep Getting Stronger After Deployment? SERPO Enables Label‑Free Self‑Evolution on Open‑Ended Tasks

Background and Motivation

Test‑Time Reinforcement Learning (TTRL) proposes that a model should sample its own answers, construct its own reward, and update its parameters during inference. Existing TTRL methods rely on answer voting and pseudo‑labels, which work for tasks with a single correct answer (e.g., math problems) but struggle with open‑ended generation where consensus does not guarantee quality.

SERPO: Self‑Evolving Rubric Policy Optimization

SERPO replaces the voting step with a self‑evolving rubric. Instead of asking “which answer has the most supporters,” the model extracts quality criteria from the differences among its own answers and uses these criteria to evaluate each response. As the policy improves, both answers and rubrics continue to evolve.

Key components:

Answer pool : For each question, SERPO maintains three tiers of answers—Good, Normal, Bad. Keeping the Normal tier ensures the rubric can distinguish fine‑grained quality differences.

Rubric generator : Frozen at initialization, it reads prompts, existing rubrics, and the answer pool, then proposes new atomic criteria from good‑vs‑bad answer differences. New and old rubrics are merged, de‑duplicated, and filtered based on two signals: (1) the ability to widen score gaps between answers, and (2) the ability to correctly order Good > Normal > Bad.

Judge : Also frozen, it outputs a probabilistic verdict token rather than a hard Pass/Fail, allowing the reward to reflect confidence levels.

Actor : The only component that updates; it receives rewards derived from the surviving rubrics and learns via GRPO (Generalized Reward‑Weighted Policy Optimization).

Why TTRL Is Needed

Many domain‑specific tasks suffer from a gap between pre‑training data and real‑world inputs (e.g., patient symptom descriptions, research questions). Keeping model parameters frozen after deployment prevents adaptation to new input distributions. TTRL leverages real test‑time prompts as the adaptation signal, and RL—rather than supervised fine‑tuning—optimizes the model without requiring a reference answer.

Experimental Setup

Experiments were conducted on two base models: Qwen3‑4B‑Instruct‑2507 and Qwen3.5‑9B. The models were evaluated on in‑domain benchmarks (HealthBench, ResearchQA) and out‑of‑distribution (OOD) sets (MedQA, LLMEval‑Med, GPQA‑Diamond, RaR‑Science). GPT‑5.1 was used only for final evaluation and not for adaptation.

Main Results

On Qwen3‑4B, SERPO raised HealthBench from 32.30 to 49.83 and ResearchQA from 57.29 to 77.60, achieving an average score of 61.98—only 0.91 points below the privileged‑supervision baseline that uses external judges and official rubrics. On the 9B model, gains of 20.63 (HealthBench) and 14.56 (ResearchQA) points were observed. Across eight OOD settings, SERPO consistently outperformed the base model, with the highest improvement of 11.4%.

Ablation Study

Variants that removed one component at a time showed that the full closed‑loop (actor + rubric + probabilistic judge) yields the best performance. Removing actor evolution drops HealthBench to 32.06 (near base). Removing rubric evolution reduces gains despite longer answers (answer length 2.29× base but lower score). Freezing the judge or allowing it to train also harms stability, indicating that a stable evaluation signal is crucial.

Long‑Term and Cross‑Benchmark Evolution

Extending training from 30 to 45 epochs gave an additional 3.6% relative improvement, and the performance curve showed no clear plateau. Linear extrapolation suggests the curve could meet the privileged‑supervision line around epoch 99. In a sequential training experiment (30 epochs on HealthBench, then 30 on ResearchQA), SERPO maintained and even increased performance on the first benchmark while improving the second, demonstrating that learning new tasks does not erase previously acquired abilities.

Key Insight

The authors argue that self‑evolution requires the evaluation capability to exceed the generation capability; otherwise, the reward signal becomes ambiguous. SERPO’s design keeps the judge frozen while allowing the rubric to evolve with answer evidence, thereby preventing drift between actor and evaluator.

Future Directions

To achieve truly open‑ended lifelong learning, external feedback (e.g., tool results, environment outcomes, user behavior) must be incorporated so that the evaluator can detect gaps between its internal judgments and real task success. Enhancing the evaluator’s logical reasoning to pinpoint missing criteria or broken inference chains is identified as the next step.

Paper: SERPO: Self‑Evolving Rubric Policy Optimization for Open‑Ended Test‑Time Reinforcement Learning
Link: https://arxiv.org/abs/2607.26873
Code: https://github.com/chiefovoavicii/SERPO
Homepage: https://chiefovoavicii.github.io/SERPO/