How to Guarantee 100% Structured JSON Output from Large Language Models

Introduction

Ensuring that large language models (LLMs) output data in a structured JSON format is crucial for automating data processing and improving system interoperability. Direct JSON output simplifies downstream parsing and correct parameter passing when LLMs invoke tools or agents.

Why LLMs Struggle with Strict JSON

LLMs generate text token by token based on probability distributions, so even with prompts requesting JSON, occasional deviations occur, which can break engineering pipelines.

Existing Solutions

OpenAI JSON Mode (Dec 2023) – still requires example JSON in prompts and cannot guarantee 100% compliance.

Kimi JSON Mode – similar limitations.

OpenAI Structured Outputs (Aug 2024) – guarantees JSON when provided with examples.

Three‑Stage Optimization Strategy

Pre‑inference (Prompt Engineering) : Include explicit statements like “The JSON object: json” and provide a “## Output Format Specification” with a JSON example.

Example prompt snippet:

## Output Format Specification:
```json
[
  {"name":"<dimension>", "mentions":"<count>", "references":[{"time":"<time>","text":"<content>"}]}
]
The JSON object: json
```

In practice, this raised correct JSON output from ~50% to 95% in an AI teaching evaluation project.

Mid‑inference (Dynamic Constrained Decoding) : Freeze the decoding process, set the probability of tokens that would break JSON syntax to zero, and optionally enforce regular‑expression constraints for each field.

Define a schema and regex, then generate only the values while the keys are inserted from memory.

city_info_schema=[{"name":"city name","country":"country","latitude":"lat","population":"pop","top 3 landmarks":["...","...","..."]}]
city_regex = (
    r"""{
"""
    r"""  \"name\": \"[\w\d\s]{1,16}\",
"""
    r"""  \"country\": \"[\w\d\s]{1,16}\",
"""
    r"""  \"latitude\": [-+]?[0-9]*\.?[0-9]{0,2},
"""
    r"""  \"population\": [-+]?[0-9]{1,9},
"""
    r"""  \"top 3 landmarks\": \[\"[\w\d\s]{1,16}\", \"[\w\d\s]{1,16}\", \"[\w\d\s]{1,16}\"\]
"""
    r"""}\"""
)

Implementation requires local model deployment (e.g., Qwen2‑7B‑Instruct) and the sglang library.

Installation commands:

pip install --upgrade pip
pip install "sglang[all]"
pip install flashinfer-0.1.2+cu121torch2.3-cp310-cp310-linux_x86_64.whl
modelscope download --model=qwen/Qwen2-7B-Instruct --local_dir ./Qwen2-7B-Instruct
python -m sglang.launch_server --model-path Qwen2-7B-Instruct --port 30000

After deployment, a sample function uses @sgl.function to generate JSON constrained by the regex.

Post‑inference (JSON Repair)

Use the json_repair Python library to fix minor syntax errors.

Control random seed to reduce variability.

Conclusion

The three stages—prompt engineering, dynamic constrained decoding, and post‑processing—can be combined to achieve near‑perfect structured JSON output from LLMs, though the most reliable method currently requires local model deployment.