How Large Language Models Can Boost Smart Chatbot Resolution Rates

Impact Factors of Resolution Rate

The most common business metric for intelligent customer‑service robots is the resolution rate, which directly influences the robot’s purchase value. A high resolution rate saves costs, while a low rate discourages procurement. Many factors affect this metric, such as gaps in the knowledge base, overly complex answers, mismatched routing, and algorithmic failures.

Knowledge base does not cover popular C‑end queries.

Answers are too complex or duplicated across topics.

Customers intentionally request human assistance.

Algorithm fails to retrieve the best match.

What Can Large Models Do?

By integrating large language models (LLMs) into the analysis pipeline, we can automate the identification of root causes for resolution‑rate fluctuations without heavy manual effort.

Analyze pre‑ and post‑transfer conversations, let the LLM summarize the reason for each handover, and perform semantic clustering to discover implicit categories.

Determine whether the visitor’s question before and after transfer is completely identical, partially identical, or completely different, and calculate the proportion of each case.

Extract core post‑transfer questions and compare the robot’s knowledge‑base answer with the human agent’s response to spot gaps or mismatches.

Case Study: Automotive Client

We selected an automotive company as a seed customer and processed 8,731 related dialogues from September‑October 2023 (LLM analysis took about one night). Example dialogue:

Before transfer: Visitor asks about microphone discounts and whether a microphone includes a receiver.

After transfer: Human agent replies that the microphone has a receiver but no discount.

Using a prompt, the LLM generated the following JSON:

{"转人工原因":"客服机器人没有回答访客关于话筒优惠和麦克风带有接收器的问题","访客在 A 和 B 对话中询问问题是否一致":"部分相同","访客核心问题":"麦克风带有接收器吗，有优惠吗","客服答案":"麦克风有接收器，没有优惠"}

Aggregating over 8,000+ conversations revealed:

55.5% of transfers were due to completely identical questions (unsatisfied robot answers).

24.8% were partially identical.

19.6% were completely different (customer intentionally seeks human help).

Further analysis showed that in 14.7% of cases the post‑transfer question had an exact knowledge‑base match, yet 98.4% of those answers differed significantly from the human response, indicating poor answer quality.

Summary

LLM‑driven analysis can precisely pinpoint why visitors are transferred to human agents, turning a previously manual, time‑consuming task into an automated, data‑rich process. By quantifying the consistency of pre‑ and post‑transfer queries and comparing knowledge‑base answers with human responses, businesses can prioritize knowledge‑base improvements, ultimately raising chatbot resolution rates and reducing unnecessary handovers.