Two Top LLM Leaderboards Disagree on #1: Private Tests & 12-Point Open-Source Gap

LMArena: Millions of Human Votes Measure "Likeability"

LMArena (formerly LMSys Chatbot Arena) uses blind pairwise voting: users pick the better answer from two anonymous models. The October 21, 2026 text leaderboard Top 10 shows Gemini 4 Argon at 1524 Elo (first), but with only 4,900 votes and a ±9 confidence interval—labeled "Preliminary." Third-place Claude Opus 4.6 has 80,000+ votes and ±3 interval. The gap between first and third is statistically unstable.

Key reading points:

1. Gemini 4 Argon's lead is tentative. Low vote count makes its top rank unreliable.

2. Elo differences ≤25 points are essentially ties. Ranks 2–10 (1509–1494) are compressed within 15 points; real tier boundaries appear at 1450/1400.

3. Open-source ceiling: Kimi K3. Kimi K3 (Max) reaches 1487 Elo (18th overall), only 7 points behind 10th place. Other Chinese models: Qwen3.8 Max ~1482, GLM-5.3 ~1474, DeepSeek V4.1 Flash ~1464—all in competitive range.

Remember: LMArena measures which answer users prefer —style, formatting, and tone influence votes. A polished but factually wrong answer can beat a rigorous but plain one.

Artificial Analysis Intelligence Index v5: Measuring "Can It Get the Job Done"

Released October 27, 2026, v5 introduces two major changes:

Change 1: 60% weight moved to private test sets. Public benchmarks have been "cramming" (test leakage into training data). v5 raises private/held-out weight from 40% to 60% and adds three new benchmarks: Cognition's FrontierCode 2.0 (132 real open-source repo tasks judged by maintainer merge decisions), Terminal-Bench-Science (70 expert-reviewed terminal science tasks), and HLE-Diamond (1,000 curated "Humanity's Last Exam" questions). Saturated GPQA Diamond is removed.

Change 2: Four capabilities equally weighted. Knowledge work, coding, science, and general each 25%, preventing single-dimension dominance.

Results highlights:

1. Claude Opus 5.5 tops at 58 points, but no model wins every subcategory. Subcategory winners: coding (FrontierCode) and knowledge (HLE-Diamond, AA-Omniscience) → Opus 5.5; science/physics → GPT-6 Astra; workflow automation → Gemini 4 Argon; document reasoning → Meta's Muse Wonder. Terminal-Bench 4.0 (terminal tasks) winner is Kimi K3.1 Pro —open-source breaks through on a single dimension.

2. Open-source vs. closed-source gap widens from 3 to 12 points. In v4.3 the gap was 3 points. In v5, top open-source Kimi K3.1 Pro scores 46 (weights unpublished) vs. Opus 5.5's 58. Among published-weight open models, DeepSeek V4.1 Flash 1017 and GLM-5.3 tie at 32—behind closed-source small model Claude Haiku 5.5 at 36. Harder, more realistic tasks re-price open-source "cost-effectiveness."

3. Cost differences are stark. Per full evaluation task (max effort): Opus 5.5 $10.04, GPT-6 Astra $6.24, GPT-6.1 Sol $1.27 (≈1/8 of Opus) while only 3 points lower. Effort-level swing: Opus 5.5 low effort 35 points, high effort 58 points, 10× price difference. Chasing "flagship scores" without checking effort tier and cost misleads.

SWE-bench: Coding Benchmark Near Saturation

SWE-bench Verified (500 real GitHub issues, patch generation judged by test suite pass) latest standings:

Closed-source leaders 95%+: Claude Opus 5 measured 97% by third-party Vals.ai (self-reported 96%), Claude Fable 5 self-reported 95%, Opus 4.8 88.6%.

Open-source first tier stuck at ~80%: DeepSeek V4 Pro 80.6%, Kimi K2.6 80.2%, GLM-5.2 80.0%.

From 13% top score in early 2024 to near-perfect in 18 months.

Two caveats: distinguish score sources —97% is independent, 95% is vendor self-report; mixing distorts rankings. Benchmark contamination is high —the 95% vs. 97% gap lacks practical meaning. For coding model selection, prioritize price, speed, and your repo's language over the top two leaderboard points.

Why Leaderboards Are Going Private: The Anti-Gaming Arms Race

Industry trend: public benchmarks are collectively exiting.

AA v5: private/held-out weight 40% → 60%, GPQA Diamond removed due to saturation.

LMArena: million-human blind votes naturally resist cramming, but vulnerable to "sycophancy tuning."

SWE-bench: organizers acknowledge high-score segment pollution and design effects.

Vendor launch scores usually come from public sets and are self-reported. Third-party value: "the scorer doesn't sell the model." Iron rule: trust third-party private scores first, then third-party public measurements, last vendor self-reports.

Decision Time: Which Leaderboard to Trust, Which Model to Use

Pick a model for real work → trust Artificial Analysis. It measures task completion, close to actual work. Budget-sensitive? Check its "cost-performance frontier": GPT-6.1 Sol (55 pts, $1.27/task) is currently the best-value flagship.

Want a pleasant chat assistant → trust LMArena. Human preference is the ultimate chat metric, but avoid low-vote, wide-interval "preliminary" rankings.

Writing code → closed-source: Claude / GPT-6 family; need cost savings or private deployment → open-source first tier (DeepSeek V4 Pro / Kimi / GLM) covers 80% of daily scenarios, but don't expect open-source to close the 15-point gap in heavy agent coding soon.

Leaderboard reading habit → check three things at a glance: who scored it (vendor vs. third-party), public vs. private test set, and effort tier (effort level). Missing any, treat the score as reference only.

Leaderboards are maps, not territory. Two leaderboards, two #1s proves "strongest model" is never a single number—it's a function of your task, your budget, your deployment constraints .

Data sources: LMArena (2026-10-21 update), Artificial Analysis Intelligence Index v5 (2026-10-27), SWE-bench third-party aggregates (Vals.ai, steel.dev, etc.); scoring methodologies differ, cross-leaderboard direct comparison invalid; vendor self-reports and third-party measurements distinguished in text.