Why Prompt‑Based JSON Extraction Breaks LLM‑Driven Intent and NER

Intent Recognition and Named Entity Recognition (NER) in an AI‑driven after‑sales chatbot

The first step of the workflow is to convert a user’s natural‑language query into structured parameters that backend services can consume. This requires two classic NLP tasks:

Intent recognition – identifying the verb (the action the user wants).

Named‑entity recognition (NER) – extracting the nouns (key entities).

Example user input:

“我昨天买的 iPhone 15 发货没？订单号 8848，赶紧的。”

Desired extraction:

Intent: Query_Logistics Entities:

{"product": "iPhone 15", "order_id": "8848", "time": "昨天"}

Traditional approach

Before large language models (LLMs), engineers trained BERT‑style models on tens of thousands of labeled examples, a process that took months and still struggled with rare product names.

Prompt‑only LLM approach and observed failures

A simple prompt such as:

You are a data extraction expert. Extract intent, product, and order_id from the user input. Must output JSON only, no extra Chinese characters!

Works in a test environment but causes JSONParseException in production because the model adds:

Polite preambles or markdown (e.g., “好的，根据您提供的信息，提取结果如下：{…}”).

Renamed keys (e.g., "order_number" instead of "order_id").

Hallucinated fields (e.g., fabricating an "order_id" when the user did not provide one).

The root cause is that LLMs are probabilistic generators; after RLHF they are inclined to be polite and to fill missing slots, so a plain prompt cannot guarantee strict machine‑readable output.

Robust intent‑extraction solutions

Solution 1 – Placeholders for missing values

Define explicit handling for absent fields. Example description for order_id:

order_id: 用户的订单编号。如果用户输入中未提供，必须填充固定字符串 NOT_FOUND，绝对禁止自行推测或编造。

The backend can filter the sentinel value NOT_FOUND or trigger a clarification dialog.

Solution 2 – Function Calling

Expose a backend function, e.g., query_order(order_id, product_name), and instruct the model to call it with the appropriate arguments. Mainstream LLMs are fine‑tuned for tool use, so when the model detects a function call it generates a structured payload rather than free‑form text, which can be directly deserialized in Java or Go.

Solution 3 – Constrained Decoding

For zero‑tolerance domains (finance, healthcare), enable constrained decoding via OpenAI’s newer API or local runtimes such as vLLM. The decoder enforces a predefined JSON schema at the token‑generation level, zeroing out probabilities for any token that would violate the schema (e.g., forcing a double‑quote at a position where only a quote is allowed). This guarantees that the output conforms exactly to the required format.

Conclusion

Treating an LLM as a pure black box that emits free‑form text for downstream parsers is fragile. Incorporating explicit constraints—placeholders, function calling, or constrained decoding—provides reliable intent and entity extraction for AI‑driven pipelines.