Achieving 85%+ Accuracy: Qunar’s SQL Agent for Intelligent Data Retrieval and Efficiency Gains

Project Background

In 2025, Qunar launched an AI Agent initiative to automate data‑retrieval tasks that previously required manual SQL writing by data‑analysis teams. The existing workflow involved many back‑and‑forth communications, long turnaround times, and high failure rates, especially as the operation team grew while data‑analyst headcount remained low.

Problems and Challenges

Data Access Difficulty : Massive, poorly governed tables and inconsistent business definitions produced noisy, dirty data.

SQL Writing Difficulty : Most product‑operations staff lacked SQL skills; complex queries were beyond their capability.

Usage Difficulty : The end‑to‑end process—request, SQL generation, execution, and result delivery—was slow and cumbersome.

Q4‑2024 metrics for domestic flight data showed each user executed an average of 188 queries, with high failure rates, long runtimes, and P60 latency, indicating substantial room for improvement.

Solution Design and Architecture

The solution was built on a layered AI infrastructure:

Model Layer : Deployed internal DeepSeek models and integrated external providers (Gemini, GPT, other Chinese models).

Framework Layer : Leveraged langchain4j and langgraph4j on a Java‑centric stack.

Platform Layer : Developed an internal Q‑MOSS agent platform (knowledge base, workflow, MCP plugins, prompt management) and also deployed the open‑source Dify platform.

Agent Layer : Implemented a specialized SQL Agent to generate and execute queries.

User Layer : Exposed the service via web and Feishu clients.

Two primary improvement levers were identified: (1) invest in data governance to standardize tables and definitions, and (2) use Agent capabilities to generate standard SQL from natural‑language requests.

Agent Evolution

Initial design injected domain knowledge directly into a monolithic SQL Agent. During proof‑of‑concept, three issues emerged:

Rule‑dropping: the model often ignored a subset of the ten predefined rules, reducing accuracy.

Information overload: combining SQL generation with optimization tasks caused the model to forget optimization steps.

Industry jargon: users frequently used domain‑specific terms that the model did not understand.

Prompt‑engineering mitigations (stage‑wise reminders, explicit constraints) yielded limited improvement, prompting a redesign.

The Agent was split into Generation Agent (produces raw SQL) and Optimization Agent (formats, validates, and enriches SQL). The Generation Agent no longer needs to handle syntax correctness, allowing it to focus on logical query construction.

React Mechanism

A “React” loop was added: after the model generates SQL, a syntax‑check tool validates it. If errors are detected, the model revises the query up to a retry threshold, preventing infinite loops.

Intent Clarification

An additional Intent Clarification Agent rewrites ambiguous natural‑language requests, asks follow‑up questions when needed, and ensures the query intent is unambiguous before SQL generation.

RAG and Knowledge Base

Retrieval‑Augmented Generation (RAG) was introduced. When a user asks a question, the system retrieves the top‑2 most similar historical Q&A pairs, injects them into the prompt, and lets the model learn from real cases. Correct answers are up‑voted, triggering automatic ingestion into the RAG knowledge base for future reuse.

The knowledge base was redesigned into six modules:

Field semantics

Table schema information

Terminology dictionary (industry jargon)

Template library for high‑frequency complex SQL patterns

Table‑level default constraints

Table relationship metadata

Iterative Evaluation and Operations

Two case‑collection methods were defined: manual curation and automatic capture of real‑world executed SQLs. An evaluation Agent periodically runs quality checks on the entire case set, producing reports that guide knowledge‑base updates or Agent bug fixes. A feedback button, daily patrol staff, and automated evaluation ensure continuous monitoring.

Results and Experience Summary

Before automation, a typical data‑request required 1.9 days across seven steps. After deploying the AI‑driven workflow, the same request is fulfilled in a single step with near‑instant response, achieving >85 % SQL accuracy and eliminating repetitive manual work for the data‑analysis team.

Key lessons learned:

Collect realistic training data by reverse‑engineering actual executed SQLs rather than synthetic examples.

Start with a single Agent; split into multiple agents only when concrete failure modes appear.

Expose the full generation‑optimization pipeline to users to improve perceived latency.

Iterate quickly with small user groups; avoid “big‑bang” releases.

Select models based on cost, latency, and domain suitability rather than raw capability alone.

Future Outlook

Planned enhancements include:

Extending the capability from pure SQL generation to full data‑analysis workflows.

Automated business‑domain detection via a dedicated Domain Agent, removing manual domain selection.

Improving response speed to meet API‑level service expectations.

Reducing knowledge‑base maintenance overhead through smarter curation tools.

The roadmap envisions exposing the platform as APIs and reusable Skills, enabling broader consumption across Qunar’s product lines.