Introducing SCALE: An Open‑Source Benchmark Redefining LLM SQL Capabilities

As large language models (LLMs) become increasingly applied in data science, many existing benchmarks focus solely on Text‑to‑SQL conversion accuracy, which fails to capture a model's true SQL handling ability in complex, real‑world scenarios.

To fill this gap, we introduce SCALE —the SQL Capability Leaderboard for LLMs—an open, transparent, community‑driven framework that aims to become an industry‑standard evaluation.

Background: Limitations of Current LLM‑SQL Benchmarks

Recent years have seen rapid progress in LLMs' ability to handle Structured Query Language (SQL), spurring a wave of public benchmarks that primarily advance Text‑to‑SQL performance.

However, professional database management and software development present challenges far beyond converting a single sentence to SQL:

Performance is critical : A query that returns correct results but takes minutes is unacceptable in production, yet existing benchmarks rarely assess execution efficiency.

Environment diversity : Database migration and cross‑platform adaptation are common, but few benchmarks test a model's ability to handle dialect differences among MySQL, Oracle, PostgreSQL, etc.

Deep understanding : Maintaining, reviewing, and refactoring legacy code requires models to not only generate SQL but also comprehend its logic, intent, and risks—an area current benchmarks overlook.

This single‑dimensional focus makes it difficult for developers and decision‑makers to select truly suitable models for real, complex business needs.

Our Solution: The SCALE Benchmark Framework

To systematically address these issues, we designed and implemented SCALE (SQL Capability Leaderboard for LLMs), a fully open‑source evaluation framework built from the perspective of database experts and seasoned developers. We believe that open source code, data, and methodology are essential for broad industry trust.

Core Dataset: High‑Quality, Multi‑Level Test Cases

The credibility of a benchmark hinges on the quality and breadth of its data. We therefore constructed a high‑quality, multi‑level dataset that reflects real‑world scenarios and released it to the community.

Real‑world cases : Collected and anonymized queries from various industries to ensure alignment with production challenges.

AI‑assisted scenario construction : Used AI to generate fine‑grained test cases for complex patterns such as subqueries, multi‑table joins, nested queries, and stored procedures, targeting logical robustness and accuracy.

Scoring weights : Assigned higher weights to more technically complex cases to differentiate difficulty.

Answer verification : All reference answers were cross‑validated for correctness.

Three Core Evaluation Dimensions

Based on this dataset, SCALE assesses models across three independent dimensions:

⚡ SQL Optimization

Research question : Does the model exhibit a DBA‑level awareness of performance optimization?

Evaluation method : Provide low‑performance queries and check whether the model can rewrite them into more efficient equivalents while preserving logical equivalence, measuring both syntactic correctness and optimization rule complexity.

Use case : Guides database performance tuning and code refactoring efforts.

🔄 Dialect Conversion

Research question : Can the model act as a reliable cross‑database “code translator”?

Evaluation method : Test logical fidelity and syntax accuracy when converting between major database dialects, ensuring results are ready for production.

Use case : Supports teams facing database migration or multi‑platform data‑center construction.

📊 SQL Understanding

Research question : Beyond code generation, how deeply does the model understand SQL?

Evaluation method : Assess result correctness, syntax error detection, execution‑plan analysis, and query‑type classification to gauge deep analytical ability.

Use case : Assists in code review, legacy system maintenance, and automated code analysis by identifying the most “SQL‑savvy” AI assistant.

Value and Applications of SCALE

SCALE creates value for different professional roles:

Data and software developers : Improves development efficiency and delivery quality by quickly identifying the most suitable AI tools for optimization, migration, and code review tasks.

AI researchers and model developers : Provides transparent evaluation methods and open datasets to pinpoint strengths and weaknesses, guiding precise, quantifiable model improvements.

Enterprise CTOs and technology decision‑makers : Enables risk‑aware technology selection based on objective, neutral data, ensuring reliable AI integration and robust data infrastructure.

Conclusion and Outlook

We launched SCALE to offer the community a more professional, in‑depth, and practice‑aligned standard for assessing LLM SQL capabilities.

As an open‑source project, we recognize community contributions as vital. All benchmark scripts, datasets, and methodologies are publicly available, and we invite you to explore the results, use the tool for precise technical judgments, and join the community by contributing code, test cases, or feedback.

Let’s collaboratively refine SCALE and advance the application of large language models in the database domain.

SCALE: Choose the right AI model for professional SQL tasks.