Why Pure AI Black‑Box Text2SQL Fails in Enterprise Deployments

Problem with black‑box Text2SQL

Many publications claim 90%–95% accuracy for Text2SQL, yet public demos are scarce because most solutions are black‑box. Black‑box methods cannot survive casual testing: AI may produce hallucinated, syntactically invalid, or semantically wrong SQL. These methods fall into two categories. Early approaches let a large model generate SQL directly from a natural‑language query, which is the purest but most uncontrollable—hallucinations, syntax errors, wrong table names, and chaotic JOINs are common. Later approaches first translate the utterance into a structured intermediate representation (e.g., JSON or a custom DSL) and then convert that to SQL. The intermediate layer reduces complexity slightly and improves accuracy, but the final translation step remains AI‑driven, so hallucinations persist.

Empirical evidence

In the enterprise‑oriented Spider 2.0 benchmark, GPT‑4o’s overall success rate dropped from 86% on Spider 1.0 to 6% , and o1‑preview fell from 91.2% to 21.3% . Subsequent studies reported annotation error rates of 52.8% for BIRD Mini‑Dev and 62.8% for Spider 2.0‑Snow, indicating a large gap between academic tests and real‑world needs and questioning the reliability of “accuracy” metrics.

Root cause

The execution chain is opaque: a user inputs a sentence and receives a result, but the intermediate reasoning—table/column matching, JOIN path selection, filter logic—is invisible. Business users cannot verify SQL, and developers cannot trace AI decisions, making error diagnosis and system improvement impossible.

White‑box design principles

Human‑readable, confirmable intermediate layer . AI only translates natural language to this layer, and a human (or business expert) must confirm it before proceeding.

Deterministic rule‑based compilation from the intermediate layer to SQL . No AI is involved in the final translation, guaranteeing 100% correctness for that step.

Standardized text (规范文本)

User utterance: 帮我查一下去年北京发往青岛的订单

Standardized text: 去年 北京 发往 青岛 订单

The standardized text is fully understandable by humans. After confirmation, a rule engine deterministically compiles it into MQL and then into SQL, ensuring the standardized‑text‑to‑SQL stage is 100% accurate .

Demo reference

http://query.raqsoft.com.cn:6999/nlq.html

Sample queries

商品名称 ‘龙虾’ 且 库存量小于50 商品 编码 名称 单价

2025年 金额最大10 订单

去年 北京发货 订单数 大于1 客户信息

LLM‑assisted standardization

An “LLM 规范” feature uses a large model to translate natural language into standardized text; an optional “deep 规范” step provides higher precision when the initial translation is unsatisfactory.

Pass‑rate trade‑off

Without LLM assistance, users must write standardized text themselves, limiting coverage for casual speakers and resulting in a low oral‑query pass rate. With LLM integration (e.g., DeepSeek), most everyday questions are converted successfully, dramatically improving oral‑query throughput. Simpler intermediate representations shrink query expressiveness, while overly complex ones become unreadable for business users.

Boundary cases

Operations such as multi‑field sorting or complex window functions are better handled by UI interactions or a complementary “自然语言报表 (NLR)” layer that performs post‑processing on result sets using Chinese commands (e.g., calculating month‑over‑month growth). This preserves deterministic execution while extending analytical capabilities.

Conclusion

Embedding a human‑confirmable, rule‑driven intermediate layer yields deterministic compilation, stability, reproducibility, and explainability—requirements that pure AI black‑box solutions cannot meet.