How Does Kimi‑K2 Stack Up? Inside the September SCALE SQL‑LLM Benchmark

1. Monthly Overview and Key Highlights

In September 2025, SCALE continued its focus on AI applications in the SQL domain. The new model released by Moonshot AI—Kimi‑K2‑Instruct‑0905—was added to the leaderboard, and the platform received functional upgrades to provide more detailed references for developers, researchers and enterprise decision‑makers.

New model evaluation : Kimi‑K2 scored 70.4 in “SQL Understanding”, 64.4 in “SQL Optimization” and 63.0 in “Dialect Conversion”. It performed well on domestic database adaptation and basic syntax handling, but lagged behind leading models on long‑complex queries and deep optimization.

Platform feature upgrades : Added “model sub‑metric ranking” and “model comparison” capabilities, allowing users to view rankings on “Logical Equivalence”, “Execution Accuracy”, etc., and to conduct multi‑dimensional visual comparisons across models.

2. Evaluation Benchmark Description

The benchmark retains the three‑dimensional evaluation system established by SCALE since its inception, ensuring long‑term comparability and reproducibility of results.

SQL Understanding : assesses whether the model can accurately parse complex query logic and user intent.

SQL Optimization : evaluates the model’s awareness and ability to improve query efficiency and performance.

Dialect Conversion : measures accuracy of syntax migration between mainstream databases.

3. Focus Analysis – Kimi‑K2 First Evaluation

SQL Understanding: 70.4

Kimi‑K2 shows reliable Text‑to‑SQL fundamentals, with strong scores in “Execution Accuracy” (72.9) and “Syntax Error Detection” (82.9). However, its “Execution Plan Detection” score is only 42.9, indicating over‑reliance on surface syntax rather than deep semantic understanding.

SQL Optimization: 64.4

The model guarantees correct syntax (100 % in Syntax Error Detection) and decent “Logical Equivalence” (68.4), but its “Optimization Depth” score (55.6) ranks 14th, showing conservative strategies and failure to apply projection push‑down or LIKE‑prefix optimizations.

Dialect Conversion: 63.0

The model excels at domestic database conversion (92.1) but struggles with large‑scale SQL migration (38.7). For example, it incorrectly rewrites WHERE ProductID = @ProductID as WHERE ProductID = ProductID, revealing difficulty with complex heterogeneous migration scenarios.

4. Platform Upgrade: Deeper, More Intuitive Comparison

The new “model sub‑metric ranking” lets users drill down into 12 detailed indicators, while the “model comparison” feature supports multi‑model visual analysis, shifting focus from overall scores to scenario‑specific optimal solutions.

Model Sub‑Metric Ranking

Users can assess a model’s performance on each sub‑task to determine suitability for particular use cases.

Model Comparison

By selecting multiple models, users can visually compare strengths and weaknesses, e.g., Kimi‑K2 ranks 12th in overall SQL Optimization but scores 9th in Syntax Error Detection and 10th in Logical Equivalence, making it a reliable choice for correctness‑focused teams.

5. Expert Commentary

Xue Xiaogang , database expert, CCF Database Committee member, Oracle ACE, PostgreSQL ACE. The September benchmark highlights a shift from “all‑round champions” to “scenario experts” in AI‑database integration, emphasizing fine‑grained selection based on specific tasks. Kimi‑K2 shows strong performance in domestic database adaptation and basic syntax correctness, but its weaknesses in large‑SQL conversion and deep optimization expose its current limits. The platform’s strategic upgrades turn the leaderboard from an academic competition into a practical engineering tool for model selection.

6. Summary and Outlook

With the addition of Kimi‑K2 and other new models, the SCALE leaderboard now covers over 19 mainstream AI models and tools. The recent feature upgrades improve transparency and usability, helping the community make more informed technology choices.

We invite more model developers and database tool providers to submit their products for evaluation, contributing to an open and transparent benchmarking ecosystem. Visit https://sql-llm-leaderboard.com/ranking/2025-09 for the full list.

SCALE: Choose professional AI models for professional SQL tasks.

Reference: SCALE 202509 – https://sql-llm-leaderboard.com/ranking/2025-09