How to Detect and Prevent Hallucinations in LLM‑Powered NL2SQL Systems

Introduction

Large language models (LLMs) have become essential for Natural Language to SQL (NL2SQL) tasks, yet they often suffer from hallucinations—outputs that are factually incorrect, ignore database schema, or violate business rules—seriously affecting reliability and user experience.

What Is Hallucination?

Hallucination includes factual hallucination (fabricated facts) and faithfulness hallucination (ignoring instructions or context). It can be confident when the model assigns high probability to wrong tokens, or uncertain when low‑confidence tokens indicate ambiguity.

Causes of Hallucination

Training data imbalance or incomplete coverage of business fields.

Out‑of‑distribution inputs that exceed the model’s knowledge.

Reasoning errors caused by ambiguous or contradictory user queries.

Detection Techniques

Unsupervised Methods

Semantic entropy: multiple samplings to measure output variability.

Claim Conditioned Probability (CCP): split text into claims and verify each claim’s confidence.

Self‑consistency sampling: generate several answers and check for agreement.

Supervised Methods

PINOSE: probe‑based classifier using hidden states of LLMs.

Factoscope: static and dynamic analysis of activation patterns to judge factuality.

Active Sampling based on Token‑level Confidence (ASD)

ASD works in three steps: (1) generate an initial answer with greedy decoding while recording top‑N token confidences; (2) identify low‑confidence tokens as potential risk points; (3) actively sample those tokens to determine whether the low confidence stems from order uncertainty or content uncertainty. Tokens are grouped into semantic fragments, their mean confidence and variance are computed, and a dynamic threshold (mean − 3 × variance) filters out unreliable fragments. The method avoids exhaustive random sampling, improving efficiency and stability.

Practical Applications

Classification tasks : detect out‑of‑range metrics in user queries using token confidence or perplexity.

Information extraction : locate uncertain conditions or grouping dimensions in NL2SQL queries.

NL2SQL : apply ASD to Ideal Auto sales data queries, achieving ~80% detection accuracy and 75% recall while reducing sampling cost.

Future Directions

Research is moving toward internal model interpretability, such as sparse auto‑encoders that map dense activations to sparse, human‑readable features, enabling “hallucination control loops” that intervene during generation. Combining these insights with domain‑specific fine‑tuning promises more stable, controllable LLM deployments.