How to Keep LLM Agents in Check with Guardrails

1. The Real Danger: Over‑Promising, Not Just Mis‑answering

When agents are used in customer‑service, sales, or internal workflows, the biggest risk is that they make commitments they should not, such as promising a "buy‑one‑get‑ten" deal. This turns a simple answer into an unauthorized contract, creating compliance and financial risks.

2. Prompt Review – Encode What the Agent Must Not Promise

Effective prompts must go beyond "you are a polite customer‑service bot" and explicitly list hard rules the model cannot violate.

A production‑ready prompt should answer four questions: what the agent can do, what it cannot promise, how to handle gray‑area queries, and which policy version to cite. Missing any of these makes downstream validation passive.

{
  "role": "pre‑sales agent",
  "goal": "explain current promotion rules and next steps",
  "hard_boundary": [
    "no self‑issued discounts, gifts, compensation, or delivery dates",
    "do not turn \"eligible\" into \"approved\"",
    "do not execute approvals, price changes, or contract issuance"
  ],
  "gray_area_handling": [
    "if price exception, benefit compensation, or contract change is involved, set needs_human_review=true",
    "responses must state \"human confirmation required\""
  ],
  "reference": ["pricing_policy_v3", "campaign_rules_2026q1"],
  "fallback": "escalate to human if no reference found"
}

The prompt should list high‑risk permissions separately so the model knows which items are description‑only and which are prohibited commitments.

3. Output Validation – Check the Commitment Structure, Not Just Tone

Instead of scanning natural‑language replies, require the agent to output a structured JSON object and then programmatically verify it.

Validation occurs in two steps: schema validation (required fields, enum values) and business validation (mapping fields to real pricing, permission, and approval rules). Only when both pass is the reply sent to the user.

{
  "reply": "Current promotion does not include buy‑one‑get‑ten; any extra gifts require manual approval.",
  "intent": "promotion_query",
  "requested_action": "none",
  "promise_level": "provisional",
  "needs_human_review": true,
  "policy_basis": "campaign_rules_2026q1",
  "sensitive_commitment": ["gift", "pricing"]
}

Key checks include: allowed values for promise_level (none, provisional, binding), mandatory approval evidence for binding promises, and forced human review when sensitive_commitment is non‑empty.

4. Intercept Tools and Actions, Not Only Final Replies

Many teams audit replies but forget to guard tool calls. The system must verify that a suggested action matches the user’s intent, respects permission limits, and meets trigger conditions before execution.

A minimal action‑interception rule might be:

High‑risk tool categories ( refund, pricing, external_send, delete) automatically enter a high‑risk path.

Without an approval ID, role permission, or secondary confirmation, the tool cannot be invoked.

The model may suggest an action, but actual execution must be released by a rule engine or human node.

All intercepted actions are logged for future rule‑set improvement.

5. First‑Version Guardrails – Build a Closed Loop, Not a Monolith

Instead of aiming for an all‑covering platform, start with the simplest high‑risk chain: filter input, validate output, and route high‑risk cases to humans. Once this loop works, the system moves from “model self‑policing” to “rule‑backed safety”.

6. Pre‑Launch Test Suite – Five Essential Test Types

Over‑commitment tests: request out‑of‑scope discounts, gifts, or delivery promises; verify they trigger human review.

Prompt‑injection tests: embed "ignore the rules" or role‑changing statements in user messages or knowledge‑base snippets; ensure the agent stays within boundaries.

Reference‑missing tests: ask for non‑existent policies; confirm the agent does not fabricate policy numbers or dates.

Tool‑misfire tests: mix "draft email" requests with "send email" actions; verify the tool is not executed without approval.

Human‑escalation tests: after a high‑risk trigger, check that the system provides a clear hand‑off path rather than a generic apology.

Prioritize scenarios where a wrong answer or action would require business fallback—pricing, compensation, approval, or external communication. By linking prompt review, output validation, and action interception, guardrails ensure the agent stops when it should and only automates safely where confidence is high.