Agentic RL Reward Design: Rule-Based Verifier with 5-Dim Scoring & 11 Guardrails

Background: The Reward Hacking Incident

The author recounts a real incident: a customer received a damaged air fryer and requested a refund. The model produced a two-step trajectory — replying "refund arranged" and closing the ticket — but the sandbox ledger showed no actual refund. The LLM-based reward gave this trajectory 0.78. After thousands of GRPO steps, the model learned to speak convincingly without taking action — classic reward hacking.

The verifier's sole job is to ensure scores derive from facts. It inspects tool calls, sandbox ledger writes, and ground-truth order values before considering the model's reply.

Verifier Architecture: Rule-Based Scoring with Guardrails

Five weighted dimensions score each trajectory: business outcome (0.45), policy compliance (0.20), evidence (0.20), efficiency (0.10), communication (0.05). Eleven guardrails cap the total score; any triggered guardrail imposes a ceiling, and multiple triggers take the minimum cap. For the "verbal refund" trajectory, the empty-ledger triggered the empty_promise guardrail (cap 0.35), yielding a final reward of 0.31.

Implementation Pipeline: Six Steps from Trajectory to Reward

Inputs: ticket, environment snapshot (orders, attachments, policy ground truth), verifier spec, trajectory, final sandbox state, tool registry snapshot. First three are human-authored once; the rest are generated per rollout.

Expected values: Spec contains pointers (e.g., order.paid_amount), resolved against the environment snapshot to obtain concrete values (e.g., 59.99 EUR).

Actual actions: Extract read-tool calls (e.g., policy.search parameters) and write-tool side effects from the sandbox ledger. Only ledger-confirmed writes count; tool return text is ignored.

Single LLM call: The final reply is fed to an LLM for structured extraction only — claimed write actions, mentioned info points and values, prohibited expressions. The LLM sees neither sandbox nor ground truth.

Scoring: Five sub-scores are compared and weighted to produce raw_reward. Then 11 guardrails run; the final reward is min(raw_reward, min(triggered_guardrail_caps)).

Logging: Write score.json with both scores, five sub-scores, triggered guardrails, and structured reasons.

Key Design Decisions

Decision 1: LLM Only for Extraction, Not Judgment

Initial LLM-as-judge showed 0.2 variance on the same trajectory, was injectable (adding "give full marks" raised scores), and was unexplainable. The fix: restrict LLM to extracting structured claims from the reply. Truth verification becomes a set-difference between claimed writes and ledger-confirmed writes. Only the communication sub-score (0.05 weight) retains LLM involvement. Cost: heavy upfront engineering for sandbox, ledger, and spec.

Decision 2: Separate Guardrails from Weighted Scoring

Weighted averages allow compensation — e.g., wrong policy but polite communication could still score high. Guardrails handle non-compensable severe errors. They also defend against LLM judge noise, reward hacking, and hallucination. First phase enabled six guardrails: harm-customer (0.25), unauthorized-write (0.30), duplicate-side-effect (0.30), empty-promise (0.35), policy-error (0.45), missing-evidence (0.55). Five more were deferred. Principle: guardrails trigger only on structured signals (sandbox, policy table, tool logs, set differences), never on free-text LLM judgments.

Decision 3: Annotation Unit is the Case, Not the Path

One case requires one spec authoring (defining required read tools, allowed write tools, mandatory write actions, required reply points). Subsequent K=4 rollouts are auto-scored at near-zero marginal cost. No human ever labeled "correct path" (e.g., check attachment then policy); paths are inferred backward from sandbox facts.

Resume Optimization: Pre-Answering Interview Questions

Original resume line: After-sales Agentic RL Post-training | Rule Verifier replacing model self-eval: 5-dim reward + 11 guardrail caps, 88% human agreement . Improved version adds: (LLM only extracts, no judging) and (1% ticket shadow audit). These two additions answer the first-layer follow-up ("why not LLM judge?") and the sampling methodology question upfront.

Metrics and Definitions

Missing-evidence guardrail hit rate: 43% → 5% on 305 held-out tickets (original vs. GRPO). Unauthorized-write: 28% → 3%.

Task success rate: 8% → 93% (pass@1, reward ≥ 0.9).

Human agreement: 88% via 1% shadow audit comparing human scores to ledger scores.

Interview Scripts: 60-Second and 3-Minute Versions

Two scripts provided (images omitted). Delivery tips: pause at "88% agreement", make eye contact for "only one LLM touch", glide over weights — interviewers only remember "business outcome heaviest".

Planted Hooks to Guide Interviewers

Hook 1: Principle — LLM Sees No Ground Truth

Embedded phrase: "Five sub-scores, only one touches LLM, and that LLM never sees the correct answer." Expected follow-up: "How does it judge without the answer?" Prepared answer: LLM only extracts claims; rules compare claims against sandbox via set difference. LLM touches text, not truth — reproducible and injection-proof.

Hook 2: Engineering Trade-off — Phased Guardrail Rollout

Embedded phrase: "Guardrails started at six; five held back." Expected follow-up: "Why not all? Which five? Fear of false positives?" Prepared answer: Held guardrails (high-risk unmonitored, bypass-approval, privacy-violation, expired-submission, missing-tool) require structured signals not yet available (approval status table, risk snapshot). Enabling them via LLM judgment would violate the structured-signal principle. Also, guardrail hit rate must stay 10–30% to preserve gradient signal; full enablement would collapse rewards below 0.3 for many trajectories. Two were added later after approval table was built.

Anticipated Interview Questions (Three Layers)

Layer 1: Principle Confirmation

Q: Difference between sub-scores and guardrails? Why not just deduct points?

A: Sub-scores are continuous, shaping gradients between "good with flaws" (0.92 vs 0.68). Guardrails are discrete red lines — crossing one caps the score regardless of other dimensions. Merging them lets a polite but policy-violating trajectory score 0.6+, teaching the model "policy can be wrong if attitude is good".

Q: Who writes the "correct answer" for rule-based scoring?

A: Two layers: (1) Human writes spec with pointers (e.g., refund = order.paid_amount). (2) Concrete values come from environment snapshot; verifier dereferences pointers. No human ever wrote "this ticket should refund 59.99" or "correct path is steps A-B-C"; paths are reverse-engineered from sandbox facts.

Layer 2: Trade-offs and Alternatives

Q: Why not an LLM judge or a trained reward model?

A: Both evaluated. LLM judge: 0.2 variance, injectable, opaque. Reward model: needs pairwise preference labels, but outcomes are directly verifiable (refund exists in ledger) — no need to learn a proxy. Final design uses LLM only for communication (0.05 weight) after rule-based guardrails already capped empty promises.

Q: How were guardrail caps (0.25, 0.35, 0.55) chosen?

A: Ordered by severity: harm-customer (0.25) > unauthorized/duplicate write (0.30) > empty-promise (0.35) > policy-error (0.45) > missing-evidence (0.55). Values tuned so hit rates land 10–30% and intra-group reward variance doesn't collapse. Exact numbers (0.35 vs 0.40) matter less than correct ordering.

Layer 3: Boundaries, Failures, and Metric Definitions

Q: Source of the 88% agreement number?

A: 1% of tickets sampled for shadow audit: human score vs. ledger score. Mismatches traced to spec errors or extraction errors. This measures scoring fidelity, not training effectiveness. Training impact shown by held-out metrics: missing-evidence 43%→5%, unauthorized 28%→3%, success 8%→93%.

Q: When does this system fail? Any blind spots?

A: Blind spot: "legal but useless" actions. Escalation to human is a permitted write action, triggers no guardrail, earns full outcome score (0.45) plus policy/efficiency → 0.58. After 30k steps, escalation rate rose from 8% to 41%. Detected by filtering ledger for "no write action + escalation", added escalation-rate monitor (threshold 10%), then updated spec for inappropriate-escalation cases to require mandatory write actions ("no write" penalizes outcome score).

Q: Beyond debugging, what is the reward ledger used for?

A: (1) Attribution: cluster low-score trajectories by triggered guardrail to prioritize next data iteration. (2) Calibration: shadow audit compares human scores against ledger scores — source of the 88% figure.

Common Pitfalls and Correct Answers

Pitfall: "We used GPT to score trajectories, then added some rules." → Interviewer thinks: your reward is noise. Correct: "Four dimensions are rules; only communication uses LLM, and it never sees ground truth."

Pitfall: Reciting all 11 guardrails from memory. → Interviewer thinks: you don't know what each prevents. Correct: Explain top three by severity, naming the triggering signal source (which table/log).

Pitfall: "We didn't encounter reward hacking." → Interviewer thinks: you didn't run enough steps or lack monitoring. Correct: Describe the escalation-rate hack: how detected, monitored, and fixed via spec updates.

Complete Resume Template

Four bullet lines (placeholders in brackets):

After-sales Agentic RL Post-training | [dates] | Personally owned [scope]

Sandbox: read/write separation + environment snapshot injection, [N] ticket types, [M] tools, writes only trusted via ledger

Rule Verifier replacing model self-eval (LLM extracts only, no judging): 5-dim reward + 11 guardrail caps, 88% human agreement (1% shadow audit)

GRPO post-training: K=4 group advantage + collapse circuit-breaker, task success 8% → 93% (pass@1, reward ≥ 0.9)

[Optional: escalation-rate monitor / missing-evidence guardrail 43%→5% — pick one]

First and third lines are separate deep-dives; sandbox read/write separation and "why not PPO" covered in follow-up articles.