Top AI Labs Converge: System Prompts as Reward Functions for RL Agents
This article traces the evolution from RLHF to RLVR and introduces RULER, a general reward function that uses LLM-as-judge to score agent trajectories against system prompts, enabling RL for non-verifiable tasks like RAG and tool use, with code examples from OpenPipe's ART framework.
Reinforcement Learning Basics
Reinforcement Learning (RL) centers on a simple loop: an agent observes a state (S), takes an action (A), receives a scalar reward (R) from the environment, and transitions to a new state (S'). This cycle repeats, producing a trajectory of (S, A, R, S') tuples. The agent updates its behavior over time to maximize cumulative reward.
Applying RL to LLMs: From RLHF to PPO
Early RL for LLMs used human preferences as the environment. OpenAI's InstructGPT (2022) introduced RLHF: humans rank model outputs, a reward model learns to predict those rankings, and PPO (Proximal Policy Optimization) fine-tunes the LLM using the reward model. ChatGPT was built this way. However, human labeling cannot scale to the hundreds of thousands of evaluations needed during training. OpenAI split the process into an offline phase (human rankings on a small batch) and an online phase (training a reward model that scores instantly). PPO then runs at GPU speed, but requires four full-sized models in memory simultaneously: the policy model (being trained), a frozen reference policy (for KL divergence penalty), the reward model, and a critic (value model). The critic estimates a baseline for each prompt so that the advantage (reward minus baseline) stabilizes training across prompts of varying difficulty. For a 7B parameter model, this means roughly 28B parameters in memory at once.
DeepSeek R1 and Verifiable Rewards (RLVR)
In January 2025, DeepSeek released R1 using a fundamentally different reward signal: Reinforcement Learning with Verifiable Rewards (RLVR). Instead of a learned reward model, RLVR uses rule-based verifiers that the environment provides directly. For math, the verifier checks if the model's answer matches the known solution; for code, a compiler runs the output and returns pass/fail (binary reward 1/0). No human rankings, no explicit reward model. The optimizer is GRPO (Group Relative Policy Optimization), which removes most of PPO's infrastructure. GRPO eliminates the critic entirely: it generates multiple responses per prompt (typically 16), normalizes rewards within each group, and assigns positive advantage to above-average responses and negative to below-average. This reduces the four-model setup to just two (policy + reference), and in practice the reference can be folded into the policy checkpoint, approaching a single-model setup. DeepSeek R1-Zero, trained with only GRPO and verifiable rewards (no supervised fine-tuning), improved on AIME 2024 math from 15.6% to 77.9%, reaching 86.7% with majority voting—matching OpenAI's o1. The model spontaneously developed self-verification, reflection, and chain-of-thought reasoning purely from the binary correct/incorrect signal. Throughout 2025, RLVR combined with GRPO became the dominant recipe for reasoning models, with every major lab releasing reasoning variants using this approach.
The Bottleneck: Non-Verifiable Agent Tasks
GRPO itself is generic—it only needs a scalar per response and normalizes within groups. But the source of that scalar is the bottleneck. For math and code, the environment gives deterministic signals. However, agents that interact with real-world tools and data (RAG agents, support agents, summarization agents) do not produce outputs that can be string-matched against a ground truth. A RAG agent retrieves context and generates an answer; there is no single correct answer. A support agent writes a reply; there is no compiler to run it. A summarization agent compresses a document; many valid summaries exist, and no string-matching verifier can distinguish a good summary from a mediocre one. In these cases, the environment does not hand you a reward signal. Some agent tasks do have verifiable outcomes (e.g., multi-step tool use with a checkable final state), and RLVR works well there. But for most agent workflows, the outcome is subjective or multi-dimensional. The RL framework (GRPO) is still suitable because agents can learn by exploring different approaches and being reinforced for effective behaviors. The missing piece is the scoring function.
How Top Labs Are Addressing the Reward Signal Problem
Major AI labs are converging on this gap from different angles. Anthropic's Constitutional AI showed that humans need not stay in the RL loop: a written set of principles (a "constitution") lets an AI evaluate its own outputs and generate preference data for RL training. OpenAI is developing "Universal Verifiers" to extend RL beyond math and code to domains like biology and medicine where answers cannot be checked by simple string matching. Andrej Karpathy has pointed toward "system prompt learning": the system prompt carries a richer signal than a scalar reward, and RL training should exploit that signal rather than relying solely on hand-crafted reward functions.
RULER: A General Reward Function Using LLM-as-Judge
RULER, built into OpenPipe's ART framework (open source, 9k+ stars), is a general reward function that replaces all custom scoring code with a single function call. It uses an LLM-as-judge to rank multiple trajectories, leveraging the same property that makes GRPO powerful: only relative ranking matters. The workflow: for each training step, generate N trajectories (typically 4–8) for the same scenario; send all N to a judge LLM (e.g., o3, o4-mini, or a local Qwen3 32B); the judge reads the agent's system prompt to understand the intended behavior, then scores each trajectory 0–1 relative to the others. Two properties make this feasible: (1) Relative scoring is easier for LLMs than absolute scoring because there is no shared calibration standard, but comparing "which of these four responses best follows the system prompt" is a comparison task LLMs handle consistently. (2) GRPO already normalizes scores within each group—it computes mean and standard deviation and normalizes. The absolute magnitude (0.9 vs 0.3) doesn't matter; the training signal comes from the relative ranking, which RULER directly provides.
Walkthrough: Replacing Hand-Coded Reward Functions with RULER
Consider training a RAG agent. In a traditional setup, you would write a reward function that checks multiple heuristics: uses_context, has_hallucination, is_complete, is_concise. Each heuristic is an engineering project requiring precise definitions, thresholds, edge-case handling, and testing. With RULER, you replace all of that with:
scored_group = await ruler_score_group(group, "openai/o3")The judge LLM reads the system prompt (e.g., "Answer using ONLY the retrieved context. Do not add information not in the context.") and applies those criteria implicitly, without you implementing them in Python.
Trajectories and Groups in ART
ART represents each agent response as a Trajectory—a sequence of messages (system, user, assistant) with metadata needed for GRPO training. Multiple trajectories for the same scenario form a TrajectoryGroup, the unit of scoring and training.
# A trajectory: one complete agent interaction
traj = art.Trajectory(
messages_and_choices=[
{"role": "system", "content": "You are a RAG support agent..."},
{"role": "user", "content": "What is the refund policy?
[Context]: ..."},
Choice(finish_reason="stop", index=0, message=ChatCompletionMessage(role="assistant", content="...")),
],
reward=0.0, # RULER will fill this in
)
# A group: multiple trajectories for the same scenario
group = art.TrajectoryGroup([traj_a, traj_b, traj_c, traj_d])
# Score the whole group with relative ranking
judged_group = await ruler_score_group(group, "openai/o3")The initial reward=0.0 is a placeholder. After ruler_score_group returns, each trajectory's reward field is updated with the judge's score. Choice and ChatCompletionMessage are standard OpenAI SDK types, so they integrate directly with existing inference code.
Concrete Examples: Low-Level ruler and High-Level ruler_score_group
Low-Level ruler Function
The lower-level ruler function works with plain message dictionaries, requiring no ART-specific objects. Here is a minimal example with three trajectories for the same RAG query: a faithful response, a hallucinated response, and a response that ignores the context.
message_lists = [
# Faithful: accurately uses retrieved context
[
{"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."},
{"role": "user", "content": "What is our refund policy?
[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact [email protected] to initiate."},
{"role": "assistant", "content": "You can get a refund within 30 days of purchase. However, digital products are non-refundable once downloaded. To start the process, email [email protected]."},
],
# Hallucinated: partially uses context but adds unsupported details
[
{"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."},
{"role": "user", "content": "What is our refund policy?
[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact [email protected] to initiate."},
{"role": "assistant", "content": "Our refund policy allows returns within 30 days. You can also get store credit for up to 90 days. Just call our hotline at 1-800-REFUND."},
],
# Ignored context: completely ignores retrieved context
[
{"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."},
{"role": "user", "content": "What is our refund policy?
[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact [email protected] to initiate."},
{"role": "assistant", "content": "I'm not sure about the refund policy. You should check the website or contact someone from the billing team."},
],
]
from art.rewards import ruler
scores = await ruler(message_lists, "openai/o3")
for label, score in zip(["Faithful", "Hallucinated", "Ignored context"], scores):
print(label)
print("→", score.score)
print("→", score.explanation)Output:
Faithful: → 0.97 → Accurately reflects the retrieved policy details, complete and concise.
Hallucinated: → 0.45 → Gives correct 30-day refund info but adds unsupported details (90-day credit, hotline), reducing accuracy.
Ignored context: → 0.05 → Provides no useful information and ignores available context.Notice we never wrote a faithfulness checker or a hallucination detector. The system prompt mentioned "use retrieved context to answer accurately," and the judge applied that as the evaluation criterion. The hallucinated response got 0.45 (not 0) because it partially used the context—the 30-day refund part was correct. The judge gave partial credit for the correct part while penalizing the fabricated content. This nuanced distinction would require massive engineering effort to encode in a rule-based reward function. Moreover, the scores form a gradient (0.97, 0.45, 0.05) rather than binary pass/fail. GRPO can use this gradient for proportional updates: strongly reinforce faithful behavior, mildly suppress hallucination (since it's partially correct), and strongly suppress ignoring context.
High-Level ruler_score_group with Trajectory Objects
ART's training loop uses Trajectory and TrajectoryGroup objects, which carry the reward field, debug logs, and the structure expected by model.train(). The higher-level ruler_score_group handles the conversion. Below is the same RAG scenario in a real training pipeline, this time with four trajectories (adding a verbose but accurate response).
# System prompt defines the agent goal; RULER uses it as the implicit reward function
system_msg = {
"role": "system",
"content": (
"You are a RAG-based support agent. "
"Answer user queries using ONLY the retrieved context. "
"Do not add information that is not in the context."
),
}
user_msg = {
"role": "user",
"content": (
"What is the refund policy?
"
"[Retrieved context]: Refunds are available within 30 days "
"of purchase. Digital products are non-refundable after "
"download. Contact [email protected] to initiate."
),
}
responses = [
"You can get a refund within 30 days of purchase. Digital products "
"are non-refundable once downloaded. Email [email protected] to start.",
"Refunds are available within 30 days. You can also get store credit "
"for up to 90 days, and our hotline is 1-800-REFUND.",
"I'm not sure about the refund policy. Please check the website or "
"contact the billing team for more details.",
"Based on the information I have, the refund policy states that "
"refunds are available within 30 days of purchase. It is important "
"to note that digital products cannot be refunded after they have "
"been downloaded. If you wish to initiate a refund, you should "
"reach out to [email protected].",
]
import art
from openai.types.chat.chat_completion import Choice
from openai.types.chat import ChatCompletionMessage
trajectories = []
for resp in responses:
traj = art.Trajectory(
messages_and_choices=[
system_msg,
user_msg,
Choice(
finish_reason="stop",
index=0,
message=ChatCompletionMessage(role="assistant", content=resp),
),
],
reward=0.0,
)
trajectories.append(traj)
group = art.TrajectoryGroup(trajectories)
from art.rewards import ruler_score_group
judged_group = await ruler_score_group(group, "openai/o3", debug=True)With debug=True, RULER prints the judge's raw reasoning and scores. The reasoning:
{
"scores": [
{ "trajectory_id": "1", "explanation": "Accurately answers the question using only the retrieved context, concisely and completely.", "score": 0.98 },
{ "trajectory_id": "2", "explanation": "Includes unsupported details about store credit and a hotline that are not in the retrieved context, so it violates the instruction to use only the context.", "score": 0.2 },
{ "trajectory_id": "3", "explanation": "Does not answer the question despite the needed information being present in the retrieved context.", "score": 0.05 },
{ "trajectory_id": "4", "explanation": "Accurately and completely answers the question using only the retrieved context, though slightly more verbose than necessary.", "score": 0.96 }
]
}Scores (ranked):
Rank 1 | Score: 0.980 — Concise faithful response
Rank 2 | Score: 0.960 — Verbose but accurate response
Rank 3 | Score: 0.200 — Fabricated store credit and hotline
Rank 4 | Score: 0.050 — Completely ignored retrieved contextKey observations:
The concise faithful response (0.98) edges out the verbose accurate response (0.96). Both use only retrieved context and are correct, but the system prompt says "answer using ONLY the retrieved context," and the concise version does so more directly. The judge identified verbosity as a minor quality issue, not a correctness issue—a subtle distinction that would be extremely hard to encode in a scoring function (how do you write a rule for "technically correct but unnecessarily verbose, deduct 0.02"?).
The hallucinated response dropped from 0.45 in the first experiment to 0.20 here. The difference stems from the system prompt: the first experiment said "answer accurately using retrieved context"; this one says "do not add information not in the context." The stricter instruction led to stricter scoring. The judge adapts automatically—if you tighten the system prompt, RULER tightens its evaluation without any scoring code changes.
The ignored-context response stayed at 0.05 in both experiments. When the answer is in the retrieved context but the agent says "I'm not sure," there is no ambiguity regardless of phrasing.
These scored trajectories are exactly what model.train() expects, so we can now look at the full training loop.
Full Training Loop with GRPO
To actually train, you replace hardcoded responses with real model rollouts. ART's gather_trajectory_groups handles the orchestration: for each scenario, it generates a group of trajectories using the model's current weights, scores them with RULER, and collects the results for GRPO.
for step in range(num_steps):
groups = await art.gather_trajectory_groups(
(
art.TrajectoryGroup(
rollout(model, scenario) for _ in range(4)
)
for scenario in scenarios
),
after_each=lambda g: ruler_score_group(g, "openai/o3"),
)
await model.train(groups) # GRPO updates LoRA weightsAt each step, the model generates 4 responses per scenario using its current weights, RULER relatively ranks them, and GRPO reinforces high-scoring behaviors while suppressing low-scoring ones. The agent learns to follow the system prompt's instructions better with each iteration. Notably, nowhere in this code is a reward function defined.
Custom Rubrics and Hybrid Verification
For most tasks, the system prompt provides enough signal for RULER to score effectively. When you need more specific criteria, RULER supports a custom rubric in natural language:
custom_rubric = """
- Prioritize responses that are concise and clear
- Penalize responses that include emojis or informal language
- Reward responses that cite sources
"""
await ruler_score_group(group, "openai/o3", rubric=custom_rubric)The rubric is natural language, not Python, so iteration is fast—you change a sentence, re-run, and check scores. In contrast, editing a Python reward function risks silently teaching the agent wrong behaviors due to a misplaced weight or a buggy condition, only discovered after training completes.
RULER is general and works for any task, not just those where custom rewards are painful. The practical question is when RULER adds value over simpler alternatives. For purely deterministic tasks (e.g., does a SQL query return the correct rows?), a binary verifier is cheaper and gives a cleaner signal. For purely subjective tasks (e.g., is a summary well-written?), RULER is the only automated option. For tasks in between (e.g., did the agent find the right answer and explain it well?), you can combine both:
judged_group = await ruler_score_group(group, "openai/o3")
for traj in judged_group.trajectories:
independent_reward = verify_correctness(traj) # binary 0/1
traj.reward += independent_rewardRULER preserves any reward assigned during rollout in a separate metric, so you can stack LLM-judge scores on top of deterministic verification without losing either signal.
Practical Details and Conclusion
You don't need the most expensive model as the judge. Cheaper models like Qwen3 32B often work well. You can also use Claude, local models via Ollama, or any LiteLLM-supported model. The choice is a cost-quality trade-off, not a hard requirement.
4–8 trajectories per group is the recommended range. Fewer than 4 gives the judge too little to compare; more than 8 may confuse the judge and increase cost without proportional benefit.
When all trajectories in a group share the same system prompt and user message (the typical case), RULER automatically deduplicates the common prefix. The judge sees the shared context once, followed by the distinct responses. For long system prompts or multi-turn conversations, this significantly reduces token usage.
RULER caches judge responses to disk. Re-running the same trajectories does not call the API again, which is crucial for debugging when you iterate on the system prompt or rubric.
The bottleneck in applying RL to agents has never been the optimization algorithm—GRPO handles that well. The real bottleneck has always been the reward signal. RLVR solved this for verifiable tasks by letting the environment score outputs directly. RULER solves it for all tasks—verifiable or not—by having an LLM judge relatively score outputs against the system prompt. The full implementation lives in the ART repository, which includes Colab notebooks demonstrating the end-to-end training loop. Repository: https://github.com/OpenPipe/ART
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
