BestHub
Sign in
Tagged articles

LLM evaluation

72 articles · Page 1 of 1
AI Code to Success
AI Code to Success
Sep 24, 2026 · Artificial Intelligence

Seed-2.1-Pro 0915 Stress Test: Honesty Under Tool Failures & Missing Data

The author evaluates Seed-2.1-Pro 0915 against its predecessor using simulated tool failures, cropped financial PDFs, and a full analysis pipeline, finding both models honest when evidence is missing but revealing post-processing unit conversion errors, concluding that model execution requires system-level verification.

LLM evaluationSeed-2.1-Proagent honesty
0 likes · 15 min read
Seed-2.1-Pro 0915 Stress Test: Honesty Under Tool Failures & Missing Data
Machine Heart
Machine Heart
Sep 18, 2026 · Artificial Intelligence

Readout Bottleneck: LLMs Know Answers But Fail to Output Them

A Peking University and Yixin AI Lab paper accepted at EMNLP 2026 reveals that large language models often encode correct reasoning in hidden states but suffer a 'readout bottleneck' where output-layer biases collapse final predictions; a two-parameter calibration recovers up to 30+ accuracy points without retraining.

EMNLP 2026Hidden StatesLLM evaluation
0 likes · 14 min read
Readout Bottleneck: LLMs Know Answers But Fail to Output Them
dbaplus Community
dbaplus Community
Sep 15, 2026 · Artificial Intelligence

Meituan's Agent Evaluation System: From Scoring to Infrastructure Capability

Meituan's Turing team shares a comprehensive framework for evaluating AI agents, covering multi-layered assessment (result, process, efficiency, risk), human-machine alignment via binary rubrics, seed test sets, expert knowledge integration, and the shift toward infrastructure for long-horizon agents with task-based evaluation harnesses.

Agent EvaluationEvaluation InfrastructureHuman-Machine Alignment
0 likes · 33 min read
Meituan's Agent Evaluation System: From Scoring to Infrastructure Capability
JavaGuide
JavaGuide
Sep 15, 2026 · Artificial Intelligence

Codex 'Dumbing Down' Claim Tested: Relay Accounts vs. Temporary Downgrades

The author empirically tests viral claims that OpenAI secretly degrades Codex for relay accounts, analyzing two benchmark tasks — a candy probability puzzle and an SVG pelican animation — finding local Codex passes both, while clarifying OpenAI's documented temporary downgrades target suspicious login activity, not relay usage specifically.

CodexGPT-6 AstraLLM evaluation
0 likes · 8 min read
Codex 'Dumbing Down' Claim Tested: Relay Accounts vs. Temporary Downgrades
IT Xianyu
IT Xianyu
Sep 7, 2026 · Artificial Intelligence

GPT-6 Astra: Capabilities, Pricing, and When It's Worth the Cost

This article analyzes OpenAI's GPT-6 Astra model, detailing its 1M-token context, tool-use capabilities, and pricing, while advising developers on suitable tasks like complex refactoring and multi-step automation, and emphasizing engineering discipline over raw specs to maximize cost-effectiveness.

AI pricingGPT-6 AstraLLM evaluation
0 likes · 7 min read
GPT-6 Astra: Capabilities, Pricing, and When It's Worth the Cost
Machine Heart
Machine Heart
Sep 1, 2026 · Artificial Intelligence

Only 33% of Claude, GPT, and Gemini Survive Real‑World Websites—ClawBench Shows AI Agents Still Struggle

ClawBench evaluates 144 production websites across 153 everyday tasks and finds that top models like Claude Sonnet 4.6, GPT‑5.4, Qwen 3.5 and GLM‑5 achieve at best a 33.3% overall success rate, exposing last‑mile non‑commit failures, anti‑bot defenses and domain‑specific weaknesses that sandbox benchmarks miss.

AI AgentsClawBenchLLM evaluation
0 likes · 14 min read
Only 33% of Claude, GPT, and Gemini Survive Real‑World Websites—ClawBench Shows AI Agents Still Struggle
Old Zhang's AI Learning
Old Zhang's AI Learning
Sep 1, 2026 · Artificial Intelligence

Doubao-Seed-Evolving Tested: Coding, Multimodal, and Long-Horizon Agent Skills

The author evaluates Doubao-Seed-Evolving's latest upgrades across coding, agent, multimodal, and long-horizon skill execution using real-world tasks like PPT-to-website conversion, personal project management, SVG generation, 3D Rubik's cube animation, and a complex 30-minute web scraping skill, finding strong autonomous task decomposition, testing, and error recovery.

AI model testingDoubao-Seed-EvolvingLLM evaluation
0 likes · 10 min read
Doubao-Seed-Evolving Tested: Coding, Multimodal, and Long-Horizon Agent Skills
Woodpecker Software Testing
Woodpecker Software Testing
Aug 28, 2026 · Artificial Intelligence

Practical Guide for Testing AI Agents: Challenges, Layered Strategy, and Real-World Practices

The article presents a comprehensive, experience‑driven framework for testing large‑model‑driven AI agents, detailing why traditional methods fail, outlining a four‑layer testing pyramid (intent, planning, tool interaction, end‑to‑end), and sharing three production‑validated engineering practices from banking and e‑commerce projects.

AI Agent TestingLLM evaluationRoot Cause Diagnosis
0 likes · 10 min read
Practical Guide for Testing AI Agents: Challenges, Layered Strategy, and Real-World Practices
DataFunTalk
DataFunTalk
Aug 13, 2026 · Artificial Intelligence

Overnight DeepSeek V4 Pro Test Reveals Disappointing Performance – Not Fit for Codex

After integrating the newly released DeepSeek V4 Pro into Codex and running 41 million tokens, the author finds the model’s silent operation, excessive context copying, weak frontend and writing abilities, and higher latency make it unsuitable as a primary model despite solid throughput and low cost.

DeepSeek V4 ProLLM evaluationcontext reuse
0 likes · 7 min read
Overnight DeepSeek V4 Pro Test Reveals Disappointing Performance – Not Fit for Codex
Baobao Algorithm Notes
Baobao Algorithm Notes
Aug 4, 2026 · Artificial Intelligence

Agentic RL: Cutting‑Edge Techniques from GLM‑5.2 and Qwen

The article dissects recent Agentic RL breakthroughs—including GLM‑5.2’s shift from GRPO to critic‑based PPO, Qwen’s multi‑dimensional verification system, the generative‑critic GenAC, and the on‑policy skill‑distillation method OPID—showing how each tackles long‑trajectory credit assignment, reward hacking, and scalable evaluation across software‑engineering, front‑end, and real‑world tasks.

Agentic RLGLM-5.2Generative Critic
0 likes · 36 min read
Agentic RL: Cutting‑Edge Techniques from GLM‑5.2 and Qwen
PaperAgent
PaperAgent
Aug 4, 2026 · Artificial Intelligence

Tencent’s WorkBuddy Unveils Its Internal Benchmark in a New Paper

Tencent’s WorkBuddy team released a paper describing the open‑source WorkBuddy Bench, a multi‑model agent benchmark that details task generation, contamination prevention, four specialized tracks (Code, Web, Office, Security), and extensive leaderboard results that reveal how models like GLM‑5.2, Opus 4.8 and GPT‑5.5 perform across diverse real‑world scenarios.

AI benchmarkAgentic AILLM evaluation
0 likes · 13 min read
Tencent’s WorkBuddy Unveils Its Internal Benchmark in a New Paper
Sohu Tech Products
Sohu Tech Products
Jul 29, 2026 · Artificial Intelligence

The Three Paradoxes of AI Agents: Memory, Reasoning, and Self‑Improvement

Rapid advances in AI agents have exposed three intertwined contradictions—memory, reasoning, and self‑improvement paradoxes—where more data hurts decision quality, engineering scaffolds create new failures, and reliable evaluation becomes a structural bottleneck, as detailed through recent industry systems and academic studies.

AI AgentsHarness EngineeringLLM evaluation
0 likes · 18 min read
The Three Paradoxes of AI Agents: Memory, Reasoning, and Self‑Improvement
AntTech
AntTech
Jul 28, 2026 · Artificial Intelligence

Introducing PIBench: A Payment‑Scenario Benchmark for AI Coding Agents

PIBench, an open‑source benchmark from Alipay’s AI payment team, defines 459 rubric items across 9 payment products and 9 real‑world projects to evaluate AI coding agents’ ability to integrate secure payment flows, covering basic functionality and advanced risk‑handling, with detailed experimental results showing significant model gaps and the impact of structured skill guidance.

AI coding agentsAlipayLLM evaluation
0 likes · 14 min read
Introducing PIBench: A Payment‑Scenario Benchmark for AI Coding Agents
PaperAgent
PaperAgent
Jul 13, 2026 · Artificial Intelligence

What Three Days of Reproducing an ACL 2026 Paper Revealed About SFT Failures

Reproducing the ACL 2026 paper on Incomplete Learning Phenomenon shows that about 15.3% of SFT training samples remain unlearned despite low loss, and the authors' Multiple‑Choice conversion with pass@5 detection uncovers five root causes and effective remediation strategies.

ILPKnowledge gapsLLM evaluation
0 likes · 13 min read
What Three Days of Reproducing an ACL 2026 Paper Revealed About SFT Failures
Data Party THU
Data Party THU
Jul 8, 2026 · Artificial Intelligence

Closing the Legal Judgment Gap: Evidence‑Based Fact Prediction (LFP) & LFPBench

This article surveys the shortcomings of current legal judgment prediction, proposes the evidence‑driven Legal Fact Prediction (LFP) paradigm, details the construction of the LFPBench benchmark, and presents extensive experiments with GPT‑4o, Claude 3.5 Sonnet and domain‑specific LLMs that cut average accuracy loss by 38.5%, while also examining ethical concerns, bias, and future research challenges.

EthicsLFPBenchLLM evaluation
0 likes · 11 min read
Closing the Legal Judgment Gap: Evidence‑Based Fact Prediction (LFP) & LFPBench
Machine Heart
Machine Heart
Jul 6, 2026 · Artificial Intelligence

Why 15 Large Reasoning Models All Fail: Hidden Risks in Their Thought Chains

A systematic study of 15 large reasoning models reveals that their intermediate reasoning traces pose higher safety risks than final answers, categorizes three failure modes, and proposes an adaptive multi‑principle steering mitigation that reduces unsafe outputs while preserving model capability.

AI safetyLLM evaluationadaptive steering
0 likes · 10 min read
Why 15 Large Reasoning Models All Fail: Hidden Risks in Their Thought Chains
AI Engineer Programming
AI Engineer Programming
Jun 30, 2026 · Artificial Intelligence

How to Quickly Validate LLM Capabilities Without Standard Benchmarks

Standard benchmarks often suffer from data leakage, mismatched real‑world scenarios, and limited metrics, so this guide proposes a practical, self‑crafted evaluation framework with diverse question types, clear scoring dimensions, and a step‑by‑step SOP to reliably assess LLM code‑generation abilities.

AI model assessmentLLM evaluationbenchmarking
0 likes · 18 min read
How to Quickly Validate LLM Capabilities Without Standard Benchmarks
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 26, 2026 · Artificial Intelligence

Claude‑style 9B Model with 1M‑Token Context Runs Locally

Qwythos‑9B, a Qwen3.5‑9B model fine‑tuned with over 500 M Claude‑style tokens, offers a 1 M‑token YaRN context, native function calling and tool‑augmented self‑correction, outperforms its base on MMLU and gsm8k benchmarks, and provides GGUF quantizations for consumer‑grade GPU deployment.

1M tokenClaudeFunction Calling
0 likes · 15 min read
Claude‑style 9B Model with 1M‑Token Context Runs Locally
Meituan Technology Team
Meituan Technology Team
Jun 25, 2026 · Artificial Intelligence

Meituan LongCat’s VitaBench 2.0: A New Benchmark for Long‑Term Dynamic Agents

VitaBench 2.0, an open‑source benchmark from Meituan LongCat, evaluates large language models on long‑term, dynamic user interactions using 56 realistic users, 819 tasks, over 2 000 evolving preferences across up to 1 580 days, and reveals that even top models struggle with memory, personalization and proactive behavior.

LLM evaluationMeituan LongCatVitaBench 2.0
0 likes · 17 min read
Meituan LongCat’s VitaBench 2.0: A New Benchmark for Long‑Term Dynamic Agents
PaperAgent
PaperAgent
Jun 22, 2026 · Artificial Intelligence

How ORGEval Revealed DeepSeek‑V3’s Surprising Modeling Strength

The paper introduces ORGEval, a graph‑theoretic evaluation framework that replaces costly solvers with bipartite‑graph isomorphism checks, proves a sufficient condition for WL‑test correctness, and shows on the Bench4Opt benchmark that DeepSeek‑V3 outperforms leading inference models in speed, consistency, and overall modeling accuracy.

DeepSeek-V3LLM evaluationORGEval
0 likes · 12 min read
How ORGEval Revealed DeepSeek‑V3’s Surprising Modeling Strength
Data Party THU
Data Party THU
Jun 18, 2026 · Artificial Intelligence

Why Large Language Models Are Short‑Sighted and How Next‑ToBE Unlocks Anticipatory Reasoning

The article examines the short‑sighted nature of current next‑token prediction in LLMs, presents the Next‑ToBE (Next Token‑Bag Exploitation) method that reshapes the training objective to expose latent future‑token awareness, and shows through extensive experiments that this approach improves anticipatory reasoning and downstream task performance.

Anticipatory ReasoningFuture Token PredictionLLM evaluation
0 likes · 12 min read
Why Large Language Models Are Short‑Sighted and How Next‑ToBE Unlocks Anticipatory Reasoning
Amap Tech
Amap Tech
Jun 18, 2026 · Artificial Intelligence

MapSatisfyBench: From Task Completion to Decision Satisfaction for Map Agents

MapSatisfyBench, a new benchmark co‑developed by Amap and Peking University, shifts map‑agent evaluation from simple task‑completion to quantifiable decision‑satisfaction by extracting implicit user demands via a restore‑identify‑filter pipeline, constructing a five‑dimensional truth schema and revealing that current LLM agents often complete tasks yet fail to meet hidden user preferences.

Decision SatisfactionFive-Dimensional MetricsImplicit Demand
0 likes · 14 min read
MapSatisfyBench: From Task Completion to Decision Satisfaction for Map Agents
SuanNi
SuanNi
Jun 11, 2026 · Artificial Intelligence

Why the Human Turing Test Is No Longer Enough: Agents’ Last Exam Benchmark

The article introduces Agents’ Last Exam (ALE), a comprehensive benchmark created by Berkeley and over 250 experts to evaluate generalist computer‑use agents on real‑world, multi‑step workflows across 55 sub‑fields, revealing that even the strongest models achieve only single‑digit pass rates.

AI AgentsBenchmarkClaude
0 likes · 13 min read
Why the Human Turing Test Is No Longer Enough: Agents’ Last Exam Benchmark
AI Engineer Programming
AI Engineer Programming
Jun 11, 2026 · Artificial Intelligence

How to Build Truly Effective LLM-as-a-Judge Evaluators

The article explains how to construct reliable LLM-as-a-Judge evaluators by combining deterministic code checks for syntactic validation, designing clear semantic evaluation rubrics, choosing appropriate output formats, calibrating with human‑labeled data, mitigating known model biases, and integrating trace‑based monitoring into production workflows.

AI safetyLLM evaluationLLM-as-a-Judge
0 likes · 15 min read
How to Build Truly Effective LLM-as-a-Judge Evaluators
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 10, 2026 · Artificial Intelligence

Testing Anthropic’s Claude Fable 5: Two Queries Cost 90 CNY

The author evaluates Anthropic’s newly released Claude Fable 5 by running a fireworks‑generation prompt and a knowledge‑collection task, compares it with Qwen3.7‑Max, details token limits, safety switches, and total expenses of roughly $10 (≈90 CNY), concluding that price outweighs its raw capability.

AnthropicClaude Fable 5Knowledge Extraction
0 likes · 4 min read
Testing Anthropic’s Claude Fable 5: Two Queries Cost 90 CNY
SuanNi
SuanNi
Jun 2, 2026 · Artificial Intelligence

Why the Best AI Scores Only 45.9% on JobBench’s ‘Dirty Work’ Benchmark

Washington University’s JobBench benchmark, built on a 1,500‑person Workbank survey and 130 real‑world tasks, measures how well AI agents can handle the chores professionals most want to delegate, revealing that even the strongest model, Claude Opus 4.7 + Claude Code, achieves just 45.9% overall, far below human‑level performance.

AI benchmarkJobBenchLLM evaluation
0 likes · 13 min read
Why the Best AI Scores Only 45.9% on JobBench’s ‘Dirty Work’ Benchmark
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
May 30, 2026 · Artificial Intelligence

Decoding the Harness Stack: Balancing Human Effort and AI Intelligence

The article analyzes Harness, a 2026 proposal that extends traditional agents with a seven‑layer architecture to fully emulate human experience, discusses rapid upgrades from prompts to skills, outlines development‑stack challenges, and presents six engineering principles for building reliable AI agents.

AGIAI AgentsHarness framework
0 likes · 9 min read
Decoding the Harness Stack: Balancing Human Effort and AI Intelligence
Machine Heart
Machine Heart
May 19, 2026 · Artificial Intelligence

Why Your Evaluation System Is the Bottleneck Holding Back LLM Progress

The article argues that current evaluation methods excel at measuring existing models but fail to anticipate qualitative shifts in emerging LLM capabilities, making evaluation the true bottleneck for future breakthroughs and calling for self‑evolving, predictive evaluation infrastructures.

AI safetyDeepMindLLM evaluation
0 likes · 11 min read
Why Your Evaluation System Is the Bottleneck Holding Back LLM Progress
Meituan Technology Team
Meituan Technology Team
May 14, 2026 · Artificial Intelligence

General 365: Meituan LongCat’s Open‑Source Benchmark Redefines LLM Reasoning Evaluation

The General 365 benchmark, built from 365 original seed questions and 1,095 variants across eight reasoning challenges, reveals that most mainstream large language models struggle with everyday logical tasks, achieving at most 62.8% accuracy and requiring far more tokens than on traditional subject‑specific tests.

AI reasoningGeneral 365LLM evaluation
0 likes · 9 min read
General 365: Meituan LongCat’s Open‑Source Benchmark Redefines LLM Reasoning Evaluation
PaperAgent
PaperAgent
Apr 26, 2026 · Artificial Intelligence

ICLR 2026 Outstanding Papers Reveal the Real Test for LLMs

The ICLR 2026 Outstanding Paper awards spotlight two studies—one proving Transformers are mathematically succinct and another showing that all major LLMs lose about 39% performance in multi‑turn conversations, exposing a reliability gap missed by single‑turn benchmarks.

AI benchmarksICLR 2026LLM evaluation
0 likes · 7 min read
ICLR 2026 Outstanding Papers Reveal the Real Test for LLMs
Fighter's World
Fighter's World
Apr 26, 2026 · Artificial Intelligence

How to Make AI Agents Reliable: Skillify’s 10‑Step Continuous Improvement Process

Agent systems often repeat the same failures, like missing historical calendar data or miscalculating time zones, but Garry Tan’s Skillify framework turns each error into a testable skill with a ten‑step checklist—including contracts, deterministic scripts, unit and integration tests, LLM evals, resolver checks, DRY audits, smoke tests, and knowledge‑base filing—to make agents structurally unable to repeat mistakes.

AI AgentsLLM evaluationcontinuous improvement
0 likes · 22 min read
How to Make AI Agents Reliable: Skillify’s 10‑Step Continuous Improvement Process
DeepHub IMBA
DeepHub IMBA
Apr 13, 2026 · Artificial Intelligence

From Retrieval to Answer: Three Overlooked Failure Points in RAG Pipelines

The article reveals silent failures in production RAG systems—where high retrieval scores and fluent LLM outputs still deliver incorrect answers—and proposes a four‑step observability loop (relevance gating, post‑generation evaluation, session‑wide tracing, and user‑signal logging) to detect and remediate these faults.

LLM evaluationRAGobservability
0 likes · 12 min read
From Retrieval to Answer: Three Overlooked Failure Points in RAG Pipelines
Machine Heart
Machine Heart
Apr 6, 2026 · Artificial Intelligence

Introducing LifeSim: The First Long‑Horizon User Life Simulator Redefining Personalized LLM Evaluation

LifeSim introduces a long‑horizon user life simulation framework that jointly models user cognition via a BDI engine and external environment, enabling realistic evaluation of personalized LLM assistants through the LifeSim‑Eval benchmark, which reveals current models excel at explicit intents but struggle with hidden intents and long‑term user understanding.

BDI modelLLM evaluationLifeSim
0 likes · 9 min read
Introducing LifeSim: The First Long‑Horizon User Life Simulator Redefining Personalized LLM Evaluation
Machine Heart
Machine Heart
Mar 31, 2026 · Artificial Intelligence

Can LLM Judges Be Trusted? TrustJudge Leverages Full Probability Distributions

LLM judges often produce contradictory scores and non‑transitive preferences; the TrustJudge framework replaces discrete scoring with distribution‑sensitive scoring and likelihood‑aware aggregation, dramatically reducing both score‑comparison and pairwise‑transitivity inconsistencies across multiple model families, improving accuracy and even serving as a reward signal for RL training.

LLM evaluationReinforcement LearningTrustJudge
0 likes · 12 min read
Can LLM Judges Be Trusted? TrustJudge Leverages Full Probability Distributions
Data STUDIO
Data STUDIO
Mar 30, 2026 · Artificial Intelligence

Why a Single AI Falls Short: Building a Multi‑Agent Expert Team for Superior Reports

The article demonstrates how a monolithic LLM struggles with multi‑dimensional market analysis and shows, through step‑by‑step code, how assembling specialized AI agents for news, technical and financial analysis yields clearer structure, deeper insight, and higher evaluation scores.

AI architectureLLM evaluationLangChain
0 likes · 17 min read
Why a Single AI Falls Short: Building a Multi‑Agent Expert Team for Superior Reports
SuanNi
SuanNi
Mar 8, 2026 · Artificial Intelligence

PinchBench Reveals Real‑World Performance of LLMs on OpenClaw Tasks

PinchBench, a rigorous benchmark that turns large language models into digital employees, measures success rate, execution speed, and per‑call cost across dozens of realistic office tasks, providing developers with concrete data to choose the most efficient model for their workloads.

AIBenchmarkLLM evaluation
0 likes · 10 min read
PinchBench Reveals Real‑World Performance of LLMs on OpenClaw Tasks
Woodpecker Software Testing
Woodpecker Software Testing
Mar 5, 2026 · Artificial Intelligence

Open-Source Playbook for Practically Testing Large Language Models

With large language models moving from labs to production, systematic testing becomes a safety baseline; this article examines why traditional tests fail, showcases four open‑source toolchains (LlamaIndex + pytest, DeepEval, Promptfoo + LangChain, Great Expectations), presents an end‑to‑end e‑commerce case, and offers practical pitfalls to avoid.

AI safetyDeepEvalLLM evaluation
0 likes · 8 min read
Open-Source Playbook for Practically Testing Large Language Models
Data Party THU
Data Party THU
Mar 2, 2026 · Artificial Intelligence

How ReLE Redefines Chinese LLM Evaluation and Reveals Capability Anisotropy

The ReLE framework introduces a dynamic, variance‑aware evaluation system that diagnoses capability anisotropy across 304 Chinese large language models, exposing ranking instability, commercial‑vs‑open‑source gaps, and format barriers while cutting evaluation cost by 70%.

AI assessmentBenchmarkCapability anisotropy
0 likes · 9 min read
How ReLE Redefines Chinese LLM Evaluation and Reveals Capability Anisotropy
SuanNi
SuanNi
Feb 27, 2026 · Artificial Intelligence

Can Deep Thought Ratio Reveal the True Reasoning Power of LLMs?

This article introduces the Deep Thought Ratio (DTR) metric, explains how tracking token modifications across neural network layers quantifies genuine inference effort, and shows through extensive experiments that DTR predicts accuracy far better than token length while enabling a sampling strategy that halves computational cost.

AI metricsLLM evaluationchain-of-thought
0 likes · 9 min read
Can Deep Thought Ratio Reveal the True Reasoning Power of LLMs?
PaperAgent
PaperAgent
Feb 19, 2026 · Artificial Intelligence

Can Claude Sonnet 4.6 Outperform Opus 4.5? A Deep Dive into Anthropic’s Latest LLM

Anthropic’s newly released Claude Sonnet 4.6 model, featuring a 1 million‑token context window, is evaluated against the flagship Opus 4.5 across coding, long‑context reasoning, agent planning and other tasks, revealing mixed performance, user preferences, and detailed benchmark comparisons.

AI AgentsAnthropicClaude Sonnet 4.6
0 likes · 5 min read
Can Claude Sonnet 4.6 Outperform Opus 4.5? A Deep Dive into Anthropic’s Latest LLM
Aikesheng Open Source Community
Aikesheng Open Source Community
Feb 9, 2026 · Databases

What the Latest SCALE Benchmark Shows About SQL Optimization in GLM‑4.7 and Seed‑OSS‑36B

The January 2026 SCALE benchmark adds an index‑suggestion metric and evaluates two new LLMs—智谱 GLM‑4.7 and 字节跳动 Seed‑OSS‑36B—revealing strengths in dialect conversion, moderate SQL understanding, and notable gaps in complex execution‑plan analysis and practical index recommendations.

AI benchmarkingDatabase OptimizationDialect Conversion
0 likes · 15 min read
What the Latest SCALE Benchmark Shows About SQL Optimization in GLM‑4.7 and Seed‑OSS‑36B
ByteDance Data Platform
ByteDance Data Platform
Jan 15, 2026 · Artificial Intelligence

Why Model Evaluation Can Be Cool: Innovative Automated Testing for Data‑Driven LLM Agents

In the era of rapidly advancing large‑model technology, the article outlines the challenges of evaluating data‑centric LLM agents, proposes a three‑layer evaluation framework covering basic capabilities, component‑level checks, and end‑to‑end business impact, and shares practical innovations such as semantic‑equivalence SQL matching, agent‑as‑judge pipelines, and a unified assessment platform.

Agent-as-JudgeData AgentLLM evaluation
0 likes · 22 min read
Why Model Evaluation Can Be Cool: Innovative Automated Testing for Data‑Driven LLM Agents
Aikesheng Open Source Community
Aikesheng Open Source Community
Dec 17, 2025 · Databases

How SQLFlash Stands Up to the SCALE Benchmark: Deep Dive into AI‑Powered SQL Optimization

This report evaluates the AI‑driven SQLFlash tool against the upgraded SCALE benchmark dataset, presenting core metrics on syntax compliance, logical equivalence, and optimization depth, and analyzes strengths, limitations, and future improvement directions for production‑grade SQL tuning.

AI modelsLLM evaluationSCALE benchmark
0 likes · 10 min read
How SQLFlash Stands Up to the SCALE Benchmark: Deep Dive into AI‑Powered SQL Optimization
AntTech
AntTech
Dec 6, 2025 · Artificial Intelligence

FinEval‑KR: Diagnosing Knowledge vs. Reasoning Gaps in Financial Large Language Models

FinEval‑KR, a new EMNLP2025 evaluation framework co‑authored by Shanghai University of Finance and Economics and Ant Group, separates knowledge coverage from logical reasoning to reveal why financial LLMs often hallucinate on calculation tasks, introduces KS, RS, and CS metrics, and ranks 18 state‑of‑the‑art models on a rigorously curated finance dataset.

Knowledge vs reasoningLLM evaluationfinance AI
0 likes · 14 min read
FinEval‑KR: Diagnosing Knowledge vs. Reasoning Gaps in Financial Large Language Models
Alibaba Cloud Developer
Alibaba Cloud Developer
Nov 26, 2025 · Artificial Intelligence

Unlocking AI-Powered Customer Service: From RAG to Deep Evaluation and Optimization

This article explores how the rapid growth of large language models reshapes intelligent customer service, detailing the evolution from rule‑based NLP bots to Retrieval‑Augmented Generation (RAG) and AI‑native agents, and presents a comprehensive framework for evaluating, diagnosing, and continuously improving chatbot performance using LLM‑driven metrics and context engineering.

AICustomer ServiceLLM evaluation
0 likes · 46 min read
Unlocking AI-Powered Customer Service: From RAG to Deep Evaluation and Optimization
Continuous Delivery 2.0
Continuous Delivery 2.0
Nov 13, 2025 · Artificial Intelligence

Shopify’s Blueprint for Scalable AI Agents: Architecture, Evaluation, and Reward‑Hack Fixes

This article details how Shopify engineered the Sidekick AI agent platform, covering its evolving architecture, just‑in‑time instruction system, rigorous LLM evaluation framework, GRPO training method, and strategies to prevent reward‑hacking, offering practical guidance for building production‑ready agentic systems.

AI AgentsAgentic SystemsLLM evaluation
0 likes · 13 min read
Shopify’s Blueprint for Scalable AI Agents: Architecture, Evaluation, and Reward‑Hack Fixes
Instant Consumer Technology Team
Instant Consumer Technology Team
Sep 5, 2025 · Artificial Intelligence

Why Context Engineering Is the Next Frontier for Large Language Models

This article surveys over 1,400 papers to define context engineering as a systematic discipline that structures retrieval, memory, tools, and multi‑agent coordination for LLMs, highlighting the critical asymmetry between understanding long contexts and generating equally complex outputs.

LLM evaluationMulti-Agent AIRetrieval-Augmented Generation
0 likes · 8 min read
Why Context Engineering Is the Next Frontier for Large Language Models
Meituan Technology Team
Meituan Technology Team
Aug 28, 2025 · Artificial Intelligence

How Meeseeks Redefines LLM Instruction-Following Evaluation

Meeseeks, a new benchmark released by Meituan’s M17 team, systematically evaluates large language models’ instruction‑following ability with a three‑tier framework, multi‑round self‑correction, and extensive real‑world data, revealing performance gaps among models such as OpenAI o‑series, Claude, DeepSeek and Qwen2.5.

AIBenchmarkLLM evaluation
0 likes · 13 min read
How Meeseeks Redefines LLM Instruction-Following Evaluation
AI Frontier Lectures
AI Frontier Lectures
Jul 27, 2025 · Artificial Intelligence

Can LLMs Ask the Right Questions? Introducing AR‑Bench for Active Reasoning

Large Language Models excel at passive reasoning, but struggle when information is incomplete; this paper defines the active reasoning problem, presents the AR‑Bench benchmark with detective, puzzle, and number‑guessing tasks, and reveals through extensive experiments that even top models like GPT‑4o perform poorly, highlighting research gaps.

BenchmarkLLM evaluationactive reasoning
0 likes · 13 min read
Can LLMs Ask the Right Questions? Introducing AR‑Bench for Active Reasoning
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Jul 24, 2025 · Artificial Intelligence

Exploring Recent Large‑Model Agent Papers: Insights and Analyses

This article reviews a series of recent research papers on large‑model agents, covering topics such as reinforcement‑learning‑driven ML agents, premise‑critique ability of LLMs, long‑term tool‑augmented LLM evaluation, agentic RAG, set‑based retrieval for multi‑hop QA, mobile VLM agents, and broader surveys of LLM applications, summarizing each work’s problem statement, prior approaches, novel contributions, experimental results, limitations, and future directions.

Agentic AIBenchmarkLLM evaluation
0 likes · 46 min read
Exploring Recent Large‑Model Agent Papers: Insights and Analyses
Meituan Technology Team
Meituan Technology Team
Jul 17, 2025 · Artificial Intelligence

How OIBench & CoreCodeBench Expose the Real Coding Limits of LLMs

The Meituan‑M17 team and Shanghai Jiao Tong University introduced two new benchmarks, OIBench and CoreCodeBench, to more accurately evaluate large language models' algorithmic and engineering coding abilities, revealing a substantial gap between claimed performance and actual capability across a range of tasks and models.

Artificial IntelligenceLLM evaluationalgorithmic assessment
0 likes · 28 min read
How OIBench & CoreCodeBench Expose the Real Coding Limits of LLMs
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Jun 3, 2025 · Artificial Intelligence

Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation

The TailoredBench framework dramatically reduces large‑language‑model evaluation cost and error by using a global probe set, model‑specific source selection, extensible K‑Medoids clustering, and calibration, achieving up to 300× speedup and a 31.4% MAE reduction across diverse benchmarks.

AI researchK-MedoidsLLM evaluation
0 likes · 10 min read
Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation
dbaplus Community
dbaplus Community
Apr 7, 2025 · Databases

How Do LLMs Tackle Oracle Bad Block Errors? A Hands‑On Evaluation

This article presents a hands‑on evaluation of several large language models—including Mistral‑Small, Deepseek‑r1, Llama 3.3 and ChatGPT‑4‑go—on Oracle database bad‑block errors, RAG‑based document retrieval, and log‑driven reasoning, revealing performance gaps, scoring results, and practical DBA implications.

AILLM evaluationOracle
0 likes · 11 min read
How Do LLMs Tackle Oracle Bad Block Errors? A Hands‑On Evaluation
Baobao Algorithm Notes
Baobao Algorithm Notes
Mar 28, 2025 · Artificial Intelligence

Can Small 7B Models Beat the State‑of‑the‑Art? A Critical Analysis of R1‑Zero Training and Unbiased GRPO

This article critically examines R1‑Zero‑style training by analyzing foundation models and reinforcement learning, uncovering pre‑training and optimization biases, proposing an unbiased Dr. GRPO method, and demonstrating a minimalist 7B‑model recipe that achieves new state‑of‑the‑art performance on AIME 2024.

Foundation ModelsGRPOLLM evaluation
0 likes · 20 min read
Can Small 7B Models Beat the State‑of‑the‑Art? A Critical Analysis of R1‑Zero Training and Unbiased GRPO
AI Algorithm Path
AI Algorithm Path
Mar 3, 2025 · Artificial Intelligence

DeepSeek‑R1 Model Performance: Comparing 32B, 70B, and R1

This article evaluates DeepSeek‑R1’s 32B and 70B distilled models alongside the original R1 on a range of reasoning and coding tasks, detailing hardware setup, test methodology, per‑task results, and a comparative analysis of their strengths and weaknesses.

32B70BDeepSeek
0 likes · 6 min read
DeepSeek‑R1 Model Performance: Comparing 32B, 70B, and R1
Architect
Architect
Feb 3, 2025 · Artificial Intelligence

How DeepSeek‑R1 Uses Pure Reinforcement Learning to Match OpenAI’s o1

This article presents DeepSeek‑R1 and DeepSeek‑R1‑Zero, two next‑generation LLMs trained with pure reinforcement learning and multi‑stage fine‑tuning, details their GRPO training framework, model‑distillation pipeline, open‑source release, and evaluation results that rival OpenAI’s o1‑1217 across reasoning, knowledge, and coding benchmarks.

DeepSeekLLM evaluationOpenAI o1
0 likes · 10 min read
How DeepSeek‑R1 Uses Pure Reinforcement Learning to Match OpenAI’s o1
Huolala Tech
Huolala Tech
Jan 22, 2025 · Artificial Intelligence

How LalaEval Revolutionizes Domain‑Specific LLM Evaluation

LalaEval is a comprehensive human‑evaluation framework that tackles enterprise challenges in building domain‑specific large language models by automating QA set generation, reducing evaluator subjectivity through controversy and score‑fluctuation analysis, and providing extensible, data‑driven metrics for model construction and iterative improvement.

AI benchmarkingLLM evaluationLalaEval
0 likes · 11 min read
How LalaEval Revolutionizes Domain‑Specific LLM Evaluation
21CTO
21CTO
Nov 24, 2024 · Artificial Intelligence

What’s New in OpenAI’s API? GPT‑4o Snapshot, Evals Tool, and Audio Features Explained

OpenAI’s latest announcements introduce the GPT‑4o snapshot with superior creative writing and file‑upload capabilities, embed the Evals evaluation framework directly in the dashboard, and add audio support in Chat Completions, empowering developers to build more reliable and expressive AI applications.

API updatesGPT-4oLLM evaluation
0 likes · 2 min read
What’s New in OpenAI’s API? GPT‑4o Snapshot, Evals Tool, and Audio Features Explained
NewBeeNLP
NewBeeNLP
Jul 10, 2024 · Artificial Intelligence

Can Large Language Models Master Co‑Temporal Reasoning? Introducing COTEMPQA

This article presents the COTEMPQA benchmark for evaluating large language models on co‑temporal reasoning, details its four scenario types, construction pipeline, experimental results across models, error analysis, and proposes the MR‑COT strategy that leverages mathematical reasoning to significantly improve performance.

LLM evaluationMR-COTbenchmark dataset
0 likes · 11 min read
Can Large Language Models Master Co‑Temporal Reasoning? Introducing COTEMPQA
Baobao Algorithm Notes
Baobao Algorithm Notes
Jun 27, 2024 · Industry Insights

How Open LLM Leaderboard v2 Redefines LLM Evaluation with New Benchmarks and Fair Scoring

Open LLM Leaderboard v2 introduces a revamped, reproducible evaluation framework for large language models, replacing saturated benchmarks with six carefully curated, unpolluted datasets, applying standardized scoring, updating the harness, adding voting and maintainer‑recommended models, and providing richer visualizations to guide the AI community.

AI metricsCommunity VotingLLM evaluation
0 likes · 19 min read
How Open LLM Leaderboard v2 Redefines LLM Evaluation with New Benchmarks and Fair Scoring
Baobao Algorithm Notes
Baobao Algorithm Notes
May 13, 2024 · Artificial Intelligence

How to Detect Test Set Leakage in Black‑Box Language Models

The ICLR 2024 paper introduces a black‑box method for detecting test‑set leakage in large language models by comparing log‑probabilities of original and shuffled test orders, proposes a scalable sharded likelihood test, and demonstrates its effectiveness on several open‑source models, revealing a potential leak in Mistral‑7B.

LLM evaluationlanguage model securityshuffled likelihood test
0 likes · 7 min read
How to Detect Test Set Leakage in Black‑Box Language Models
AI Large Model Application Practice
AI Large Model Application Practice
Sep 14, 2023 · Artificial Intelligence

How LangSmith Turns LLM Debugging into Production‑Ready Insight

This article explores how LangSmith, an experimental platform from the LangChain team, bridges the gap between prototype LLM applications and production by providing comprehensive tracing, debugging, testing, evaluation, and run‑management features that help developers monitor and improve generative AI systems.

AI ObservabilityLLM debuggingLLM evaluation
0 likes · 11 min read
How LangSmith Turns LLM Debugging into Production‑Ready Insight
Programmer DD
Programmer DD
Jul 21, 2023 · Artificial Intelligence

Why Did GPT-4’s Performance Plummet Between March and June 2023?

A Stanford‑Berkeley study reveals that between March and June 2023 GPT‑4’s accuracy on prime‑checking fell from 97.6% to 2.4%, code generation quality dropped sharply, and sensitivity handling changed, underscoring the rapid, unpredictable shifts in large language model performance over short periods.

AI safetyArtificial IntelligenceGPT-4
0 likes · 6 min read
Why Did GPT-4’s Performance Plummet Between March and June 2023?
DataFunTalk
DataFunTalk
Apr 19, 2023 · Artificial Intelligence

Is the Daily Emergence of Large Language Models Beneficial?

The article examines the rapid proliferation of large language models, weighing both the opportunities for experimentation and the drawbacks of noise, and argues that establishing authoritative Chinese LLM evaluation benchmarks is essential to guide meaningful progress in the field.

AI researchLLM evaluationlarge language models
0 likes · 7 min read
Is the Daily Emergence of Large Language Models Beneficial?
21CTO
21CTO
Apr 2, 2023 · Artificial Intelligence

Can GPT‑4 Be Considered Early AGI? Insights from Microsoft’s 155‑Page Study

This article reviews Microsoft’s extensive 155‑page work on early experiments with GPT‑4, exploring how the model approaches artificial general intelligence, its testing methodology, multimodal capabilities, programming and mathematical performance, interaction with tools and humans, limitations, societal impact, and future research directions.

AI safetyArtificial General IntelligenceGPT-4
0 likes · 15 min read
Can GPT‑4 Be Considered Early AGI? Insights from Microsoft’s 155‑Page Study