How to Ensure Reliable Structured Outputs in LLM Agents

Why Structured Output Fails

Two recent incidents illustrate the problem: using response_format with DeepSeek (via LiteLLM) triggers a BadRequest error, and sending strict: true to Mistral through LangChain results in a 422 error. The parameter names look official, but their support depends on the specific model vendor, so code that works with one model may break with another.

Principle: Format ≠ Content

Agent frameworks often equate “structured output” with “JSON output”. A more reliable approach is to separate the concerns: first ensure the output format can be parsed, then verify the content’s correctness.

Two Engineering Lines

Format line : add constraints during generation (API schema, tool‑parameter schema, open‑source constrained decoding), then apply local validation and limited retries.

Content line : enforce business rules, database existence checks, retrieve facts via RAG, and on validation failure feed the error back for another generation round or fall back to human review.

External validation and retries solve “shape‑wrong” or “format‑broken” issues, but they cannot replace the model’s ability to generate correct facts.

Native Capabilities

JSON Mode

Typical switch: {"type": "json_object"} in response_format.

Guarantees syntactically valid JSON only; does not enforce field names, types, or required properties.

Best for prototypes, loosely structured data, or when the model does not support strict schema.

Structured Outputs (strict schema)

Typical switch: json_schema + strict: true.

When the model supports the schema subset, constrained decoding forces the output to match the given schema (keys, types, enums, required fields).

Ideal for extraction, classification, or any downstream strong‑typed consumption.

Tool / Function Calling

Typical switch: tools / tool_calls.

The model decides whether to call a tool and which one; parameters can also be strict‑schema validated.

Suited for actions with side effects (search, DB write, API call), not the primary channel for pure data extraction.

When a result object conforms to a schema, prefer Structured Outputs ( response_format + strict). Use Tool Calling when the model must decide whether to perform an action. If only JSON Mode is available, combine it with schema validation and retries.

When No Native Schema Is Available

For older or locally hosted models that lack strict Structured Outputs, a common pipeline is:

Provide a strong prompt that includes the schema or examples.

Model outputs free‑form text.

Extract the JSON fragment.

Validate against a schema (e.g., Pydantic or JSON Schema).

If validation fails, inject the error summary into the next prompt and retry a limited number of times, or fall back to degradation or human review.

Practical Strategies and Their Costs

Schema validation : immediate exception on parse or schema failure – requires retry or upstream failure handling.

Repair chain : feed validation error back into the prompt to request a corrected object – adds extra round‑trip latency and cost.

Partial extraction : use regex or a fixer to salvage usable fields – may lose data and be hard to reproduce.

Default fallback : fill missing non‑critical fields with defaults – risks contaminating data.

Human queue : after automated paths are exhausted, route to manual review – incurs labor and SLA impact.

Over‑strict validation can discard keys that do not map to the schema or break on malformed syntax; the product strategy must balance downstream stability with information completeness.

Tool Calling vs Multi‑Round Agent Loops

Tool Calling does not equal “multiple LLM rounds in a single user request”. A single response may contain one or more tool_calls. Multi‑round interactions arise when the tool is executed and its result is fed back to the model, or when the repair‑retry loop is used.

Layered Fallback Architecture

Generate constraints

Use strict schema if supported (via response_format or tool strict).

Otherwise fall back to JSON Mode + validation or open‑source constrained decoding (e.g., Outlines, Guidance, XGrammar).

Separate parsing and validation

Assign distinct error codes for parse failures, schema mismatches, and business‑rule violations.

Prevent internal exceptions from bubbling up as user‑visible 500 errors.

Limited repair attempts

Only retry on format/schema failures; for factual doubts invoke retrieval or human review to avoid infinite loops.

Observability and fallback

Track success rate, retry count, failure type distribution, and refusal rate.

When limits are exceeded, queue for safe defaults or human handling while preserving the original output for post‑mortem.

Technical metrics focus on structured‑success rate, average retries, and failure categories; business metrics monitor fallback trigger rate and the proportion of cases where the model was correct but the schema was too strict, guiding schema calibration.

Other Advanced Options

Fine‑tuning / preference optimization for stable schemas (high cost, still needs runtime validation).

Open‑source constrained decoding for self‑hosted models (runtime dependency, schema complexity impacts speed).

Separate generation and validation pipelines for high‑value fields.

Multi‑output alignment for ultra‑low‑tolerance domains (expensive, requires arbitration).

Structured caching – only cache objects that passed validation.

Human‑in‑the‑loop – ensure feedback loops for long‑tail or high‑leverage requests.

Code/SQL mediators – enforce structure via sandboxed execution or read‑only DB queries (requires careful security design).

Remember: never rely on a single sampling for correctness. Use decoding or API constraints for shape, and combine retrieval, rule‑based checks, and human review for factual accuracy.

Conclusion

Structured interfaces control the “shape”; correctness depends on the model, context, business validation, and limited retries. When choosing a solution, first ask whether you need a parsable object or an action trigger, then select Structured Outputs or Tool Calling instead of defaulting to JSON Mode.