Tagged articles
LLM Evaluation
92 articles · Page 1 of 1

java1234Nov 16, 2026 · Artificial Intelligence
The article evaluates Tencent's newly released Hunyuan Hy4 Flash model (224B total parameters, 12B activated per token, 1M token context) through three practical tests on WorkBuddy: deep research with web search, complex Excel data cleaning and summarization, and SVG scene generation, reporting timings and qualitative results.
1M contextAI model testingHy4 Flash
0 likes · 14 min read

Tencent's Hy4 Flash, a 224B MoE model with 12B active parameters, achieves 150+ tokens/s decoding speed at 1/20 the cost of its predecessor, outperforms DeepSeek-V4.1-Flash on cost efficiency, and matches Claude Haiku 5.5 user satisfaction in blind tests across 259 office tasks.
HunyuanHy4 FlashInference Optimization
0 likes · 6 min read

The article analyzes Qwen4-Max-Preview after its 4-day anonymous run as Qoduck in Qoder, evaluates conflicting community benchmarks including the "pelican riding a bicycle" SVG test, notes a limited-time 80% price cut, and advises trying it on real tasks but not migrating production workflows yet.
AI coding assistantsLLM EvaluationQoder
0 likes · 12 min read

The author benchmarks Qwen4-Max Preview on five creative coding tasks—including a physics-based claw machine, a procedural Chinese ink painting engine, a Three.js League of Legends map, and a time-collapse visualization—finding exceptional token efficiency and engineering quality despite slow generation speeds of ~33 tokens/s.
AI model reviewChinese LLMsLLM Evaluation
0 likes · 9 min read

UtilityBench introduces a utility-based evaluation framework that assigns real-world costs to errors, revealing that higher LLM accuracy does not reliably translate into responsible answering or refusal behavior across 23 tested models.
AI safetyLLM EvaluationUtilityAUC
0 likes · 17 min read

360's TEP platform automates LLM evaluation with standardized datasets, dual effect-and-performance benchmarks, built-in and custom evaluators, and reproducible task tracking to replace ad-hoc 'should be fine' decisions with evidence-based iteration cycles.
360AILLM EvaluationTEP platform
0 likes · 6 min read

Researchers simulated a multi-agent social network where a malicious agent injects false statements; all 10 tested LLMs — including GPT-5.5, Claude, Gemini, Qwen, and Llama — gradually accepted fabricated claims, which persisted after the attacker left and biased downstream recommendations.
AI safetyLLM EvaluationMandela effect
0 likes · 15 min read

HiRoute's new release introduces per-step model routing with a three-question decision framework, continuous trust-level evaluation of new models using real workflow evidence, task delegation to specialized agents, and cross-platform GUI support, all running locally and open-source.
Agent RoutingHiRouteLLM Evaluation
0 likes · 27 min read

Researchers from 2077AI, UC Berkeley, and other institutions created CyberArena, a symmetric attack-defense benchmark where 12 frontier AI models competed in 466 matches, revealing that static benchmarks overestimate capabilities and defense speed distinguishes top performers like GPT-5.6 Sol and Claude Opus 5.
AI agentsCyberArenaLLM Evaluation
0 likes · 20 min read

LangChain's Deep Agents v0.7 cuts default harness base input tokens by 65% — from 5,395 to 1,895 — by clearing the default system prompt, trimming tool descriptions 43%, and making TodoListMiddleware opt-in, backed by evaluations across GPT-5.6, Claude Opus 4.8, and GLM 5.2 showing no consistent accuracy gains from these components.
Agent HarnessDeep AgentsLLM Evaluation
0 likes · 13 min read

Researchers from HKUST Guangzhou, Fudan, Washington, Imperial College, and Stanford introduce OpenDS-World, a benchmark of 200 real-world deep search tasks spanning 4 modalities, 42 resource formats, and 7 access types; 12 frontier models including GPT-6 Astra, Claude Opus 5.5, and Gemini 3.8 Flash achieve at best 39% accuracy, revealing severe challenges in long-horizon decision-making, tool failure recovery, and search termination.
AI agentsDeep SearchLLM Evaluation
0 likes · 21 min read

This survey reviews 119 studies on benchmark compression for efficient LLM evaluation, categorizing methods into static compression, adaptive evaluation, and benchmark analysis, while emphasizing that reliable compression requires defining evaluation goals, verifying coverage, and accounting for full lifecycle costs rather than simply reducing test items.
LLM Evaluationadaptive evaluationbenchmark analysis
0 likes · 34 min read

AWS introduces Skill-level evaluation in Strands Evals and Bedrock AgentCore, adding three-layer checks — Skill Invoked, Skill Selection Accuracy, and Skill Instruction Following — to verify agents actually load, choose, and execute skills correctly, moving skill validation into CI/CD pipelines like unit tests.
AWSBedrock AgentCoreCI/CD
0 likes · 14 min read

iFixAi is an open-source tool that independently audits AI agents in 120 seconds, using 60 checks across five pillars to produce an A-F grade with mandatory minimum scores, requiring separate judge models to ensure impartiality, and integrates into CI pipelines for governance and compliance.
AI agent auditingCI/CD IntegrationCompliance
0 likes · 14 min read

ExplorationBench, a new benchmark from Fudan University and Tencent Hunyuan, evaluates AI systems' ability to discover rules in unfamiliar environments by placing them in two alien worlds—AlienCode and AlienLogic—where models must design experiments, infer rules from feedback, and solve novel tasks across four exploration rounds.
AI explorationAlienCodeAlienLogic
0 likes · 20 min read

Artificial Analysis's AA-PersonalityBench uses 900 questions to evaluate 30 personality traits and 21 communication features across leading AI models, ranking them on indices like expressiveness, transparency, and sycophancy via Elo ratings, revealing distinct model personalities without system prompts.
AA-PersonalityBenchAI benchmarkingArtificial Analysis
0 likes · 7 min read

The article analyzes GPT-6 Astra's impact on software development, arguing it shifts AI programming from generating code snippets to executing complete tasks, and outlines practical implications for Java teams including legacy system understanding, change verification, and documentation consistency, while advocating for structured task boundaries and empirical evaluation before adoption.
AI programmingAI-assisted codingGPT-6 Astra
0 likes · 6 min read

Alibaba Cloud's AgentCore product lead details the platform's architecture for building, governing, and scaling enterprise AI agents, covering collaboration patterns, asset management, observability, evaluation methods, and security governance.
AI agentsAgentCoreLLM Evaluation
0 likes · 27 min read

The author evaluates Seed-2.1-Pro 0915 against its predecessor using simulated tool failures, cropped financial PDFs, and a full analysis pipeline, finding both models honest when evidence is missing but revealing post-processing unit conversion errors, concluding that model execution requires system-level verification.
LLM EvaluationSeed-2.1-Proagent honesty
0 likes · 15 min read

The article describes an AI evaluation platform that addresses non-determinism, evaluator bias, and dataset chaos by introducing consistency metrics for scenario stability, confidence scores for evaluator reliability, a pluggable architecture supporting four evaluator types, unified dataset management with golden sets, and continuous experiment orchestration across 155 scenarios.
AI-evaluationConfidence ScoringContinuous Evaluation
0 likes · 48 min read

A Peking University and Yixin AI Lab paper accepted at EMNLP 2026 reveals that large language models often encode correct reasoning in hidden states but suffer a 'readout bottleneck' where output-layer biases collapse final predictions; a two-parameter calibration recovers up to 30+ accuracy points without retraining.
CalibrationEMNLP 2026LLM Evaluation
0 likes · 14 min read

The author built an open-source knowledge learning system called Knowledge Dance using doubao-seed-2.1-pro-0915 via Trae IDE in three dialogue iterations over three hours, featuring automated research, PPT generation, quiz games, review reports, and knowledge base integration, showcasing the model's multimodal coding and self-debugging via screenshot analysis.
AI-assisted developmentFastAPILLM Evaluation
0 likes · 15 min read

Meituan's Turing team shares a comprehensive framework for evaluating AI agents, covering multi-layered assessment (result, process, efficiency, risk), human-machine alignment via binary rubrics, seed test sets, expert knowledge integration, and the shift toward infrastructure for long-horizon agents with task-based evaluation harnesses.
Agent EvaluationEvaluation InfrastructureHuman-Machine Alignment
0 likes · 33 min read


JavaGuideSep 15, 2026 · Artificial Intelligence
The author empirically tests viral claims that OpenAI secretly degrades Codex for relay accounts, analyzing two benchmark tasks — a candy probability puzzle and an SVG pelican animation — finding local Codex passes both, while clarifying OpenAI's documented temporary downgrades target suspicious login activity, not relay usage specifically.
CodexGPT-6 AstraLLM Evaluation
0 likes · 8 min read


IT XianyuSep 7, 2026 · Artificial Intelligence
This article analyzes OpenAI's GPT-6 Astra model, detailing its 1M-token context, tool-use capabilities, and pricing, while advising developers on suitable tasks like complex refactoring and multi-step automation, and emphasizing engineering discipline over raw specs to maximize cost-effectiveness.
AI pricingGPT-6 AstraLLM Evaluation
0 likes · 7 min read

This article analyzes three leading open-source AI model evaluation frameworks—LangChain Eval, DeepEval, and RAGAS—comparing their design philosophies, capabilities, and industrial use cases to help teams build reliable, continuous quality assurance for LLM applications.
AI quality assuranceCI/CD IntegrationDeepEval
0 likes · 8 min read

ClawBench evaluates 144 production websites across 153 everyday tasks and finds that top models like Claude Sonnet 4.6, GPT‑5.4, Qwen 3.5 and GLM‑5 achieve at best a 33.3% overall success rate, exposing last‑mile non‑commit failures, anti‑bot defenses and domain‑specific weaknesses that sandbox benchmarks miss.
AI agentsClawBenchLLM Evaluation
0 likes · 14 min read

The author evaluates Doubao-Seed-Evolving's latest upgrades across coding, agent, multimodal, and long-horizon skill execution using real-world tasks like PPT-to-website conversion, personal project management, SVG generation, 3D Rubik's cube animation, and a complex 30-minute web scraping skill, finding strong autonomous task decomposition, testing, and error recovery.
AI model testingCoding AgentDoubao-Seed-Evolving
0 likes · 10 min read

The article presents a comprehensive, experience‑driven framework for testing large‑model‑driven AI agents, detailing why traditional methods fail, outlining a four‑layer testing pyramid (intent, planning, tool interaction, end‑to‑end), and sharing three production‑validated engineering practices from banking and e‑commerce projects.
AI Agent TestingLLM EvaluationRoot Cause Diagnosis
0 likes · 10 min read

After integrating the newly released DeepSeek V4 Pro into Codex and running 41 million tokens, the author finds the model’s silent operation, excessive context copying, weak frontend and writing abilities, and higher latency make it unsuitable as a primary model despite solid throughput and low cost.
DeepSeek V4 ProLLM Evaluationcontext reuse
0 likes · 7 min read

The article dissects recent Agentic RL breakthroughs—including GLM‑5.2’s shift from GRPO to critic‑based PPO, Qwen’s multi‑dimensional verification system, the generative‑critic GenAC, and the on‑policy skill‑distillation method OPID—showing how each tackles long‑trajectory credit assignment, reward hacking, and scalable evaluation across software‑engineering, front‑end, and real‑world tasks.
GLM-5.2Generative CriticLLM Evaluation
0 likes · 36 min read

Tencent’s WorkBuddy team released a paper describing the open‑source WorkBuddy Bench, a multi‑model agent benchmark that details task generation, contamination prevention, four specialized tracks (Code, Web, Office, Security), and extensive leaderboard results that reveal how models like GLM‑5.2, Opus 4.8 and GPT‑5.5 perform across diverse real‑world scenarios.
AI benchmarkAgentic AILLM Evaluation
0 likes · 13 min read

Rapid advances in AI agents have exposed three intertwined contradictions—memory, reasoning, and self‑improvement paradoxes—where more data hurts decision quality, engineering scaffolds create new failures, and reliable evaluation becomes a structural bottleneck, as detailed through recent industry systems and academic studies.
AI agentsHarness EngineeringLLM Evaluation
0 likes · 18 min read


AntTechJul 28, 2026 · Artificial Intelligence
PIBench, an open‑source benchmark from Alipay’s AI payment team, defines 459 rubric items across 9 payment products and 9 real‑world projects to evaluate AI coding agents’ ability to integrate secure payment flows, covering basic functionality and advanced risk‑handling, with detailed experimental results showing significant model gaps and the impact of structured skill guidance.
AI coding agentsAlipayLLM Evaluation
0 likes · 14 min read

Reproducing the ACL 2026 paper on Incomplete Learning Phenomenon shows that about 15.3% of SFT training samples remain unlearned despite low loss, and the authors' Multiple‑Choice conversion with pass@5 detection uncovers five root causes and effective remediation strategies.
ILPKnowledge gapsLLM Evaluation
0 likes · 13 min read

This article surveys the shortcomings of current legal judgment prediction, proposes the evidence‑driven Legal Fact Prediction (LFP) paradigm, details the construction of the LFPBench benchmark, and presents extensive experiments with GPT‑4o, Claude 3.5 Sonnet and domain‑specific LLMs that cut average accuracy loss by 38.5%, while also examining ethical concerns, bias, and future research challenges.
EthicsLFPBenchLLM Evaluation
0 likes · 11 min read

A systematic study of 15 large reasoning models reveals that their intermediate reasoning traces pose higher safety risks than final answers, categorizes three failure modes, and proposes an adaptive multi‑principle steering mitigation that reduces unsafe outputs while preserving model capability.
AI safetyLLM Evaluationadaptive steering
0 likes · 10 min read

Standard benchmarks often suffer from data leakage, mismatched real‑world scenarios, and limited metrics, so this guide proposes a practical, self‑crafted evaluation framework with diverse question types, clear scoring dimensions, and a step‑by‑step SOP to reliably assess LLM code‑generation abilities.
AI model assessmentLLM Evaluationbenchmarking
0 likes · 18 min read

Qwythos‑9B, a Qwen3.5‑9B model fine‑tuned with over 500 M Claude‑style tokens, offers a 1 M‑token YaRN context, native function calling and tool‑augmented self‑correction, outperforms its base on MMLU and gsm8k benchmarks, and provides GGUF quantizations for consumer‑grade GPU deployment.
1M tokenClaudeFunction Calling
0 likes · 15 min read

VitaBench 2.0, an open‑source benchmark from Meituan LongCat, evaluates large language models on long‑term, dynamic user interactions using 56 realistic users, 819 tasks, over 2 000 evolving preferences across up to 1 580 days, and reveals that even top models struggle with memory, personalization and proactive behavior.
LLM EvaluationMeituan LongCatPersonalization
0 likes · 17 min read

The paper introduces ORGEval, a graph‑theoretic evaluation framework that replaces costly solvers with bipartite‑graph isomorphism checks, proves a sufficient condition for WL‑test correctness, and shows on the Bench4Opt benchmark that DeepSeek‑V3 outperforms leading inference models in speed, consistency, and overall modeling accuracy.
DeepSeek-V3LLM EvaluationORGEval
0 likes · 12 min read

The article examines the short‑sighted nature of current next‑token prediction in LLMs, presents the Next‑ToBE (Next Token‑Bag Exploitation) method that reshapes the training objective to expose latent future‑token awareness, and shows through extensive experiments that this approach improves anticipatory reasoning and downstream task performance.
Anticipatory ReasoningFuture Token PredictionLLM Evaluation
0 likes · 12 min read


Amap TechJun 18, 2026 · Artificial Intelligence
MapSatisfyBench, a new benchmark co‑developed by Amap and Peking University, shifts map‑agent evaluation from simple task‑completion to quantifiable decision‑satisfaction by extracting implicit user demands via a restore‑identify‑filter pipeline, constructing a five‑dimensional truth schema and revealing that current LLM agents often complete tasks yet fail to meet hidden user preferences.
Decision SatisfactionFive-Dimensional MetricsImplicit Demand
0 likes · 14 min read

Claude Fable 5’s newly added safety guardrails silently downgrade its answers, causing it to refuse every ProgramBench task and score zero, yet the model still tops the benchmark leaderboard, highlighting a paradox between model capability, safety restrictions, and practical usability.
AI safetyClaude Fable 5LLM Evaluation
0 likes · 9 min read


SuanNiJun 11, 2026 · Artificial Intelligence
The article introduces Agents’ Last Exam (ALE), a comprehensive benchmark created by Berkeley and over 250 experts to evaluate generalist computer‑use agents on real‑world, multi‑step workflows across 55 sub‑fields, revealing that even the strongest models achieve only single‑digit pass rates.
AI agentsClaudeGPT-5.5
0 likes · 13 min read

The article explains how to construct reliable LLM-as-a-Judge evaluators by combining deterministic code checks for syntactic validation, designing clear semantic evaluation rubrics, choosing appropriate output formats, calibrating with human‑labeled data, mitigating known model biases, and integrating trace‑based monitoring into production workflows.
AI safetyLLM EvaluationLLM-as-a-Judge
0 likes · 15 min read

The author evaluates Anthropic’s newly released Claude Fable 5 by running a fireworks‑generation prompt and a knowledge‑collection task, compares it with Qwen3.7‑Max, details token limits, safety switches, and total expenses of roughly $10 (≈90 CNY), concluding that price outweighs its raw capability.
AnthropicClaude Fable 5Knowledge Extraction
0 likes · 4 min read


SuanNiJun 2, 2026 · Artificial Intelligence
Washington University’s JobBench benchmark, built on a 1,500‑person Workbank survey and 130 real‑world tasks, measures how well AI agents can handle the chores professionals most want to delegate, revealing that even the strongest model, Claude Opus 4.7 + Claude Code, achieves just 45.9% overall, far below human‑level performance.
AI benchmarkJobBenchLLM Evaluation
0 likes · 13 min read

The article analyzes Harness, a 2026 proposal that extends traditional agents with a seven‑layer architecture to fully emulate human experience, discusses rapid upgrades from prompts to skills, outlines development‑stack challenges, and presents six engineering principles for building reliable AI agents.
AGIAI agentsHarness framework
0 likes · 9 min read

The article argues that current evaluation methods excel at measuring existing models but fail to anticipate qualitative shifts in emerging LLM capabilities, making evaluation the true bottleneck for future breakthroughs and calling for self‑evolving, predictive evaluation infrastructures.
AI safetyDeepMindLLM Evaluation
0 likes · 11 min read

The General 365 benchmark, built from 365 original seed questions and 1,095 variants across eight reasoning challenges, reveals that most mainstream large language models struggle with everyday logical tasks, achieving at most 62.8% accuracy and requiring far more tokens than on traditional subject‑specific tests.
AI ReasoningGeneral 365LLM Evaluation
0 likes · 9 min read

The ICLR 2026 Outstanding Paper awards spotlight two studies—one proving Transformers are mathematically succinct and another showing that all major LLMs lose about 39% performance in multi‑turn conversations, exposing a reliability gap missed by single‑turn benchmarks.
AI benchmarksICLR 2026LLM Evaluation
0 likes · 7 min read

Agent systems often repeat the same failures, like missing historical calendar data or miscalculating time zones, but Garry Tan’s Skillify framework turns each error into a testable skill with a ten‑step checklist—including contracts, deterministic scripts, unit and integration tests, LLM evals, resolver checks, DRY audits, smoke tests, and knowledge‑base filing—to make agents structurally unable to repeat mistakes.
AI agentsLLM Evaluationcontinuous improvement
0 likes · 22 min read

The article reveals silent failures in production RAG systems—where high retrieval scores and fluent LLM outputs still deliver incorrect answers—and proposes a four‑step observability loop (relevance gating, post‑generation evaluation, session‑wide tracing, and user‑signal logging) to detect and remediate these faults.
LLM EvaluationRAGobservability
0 likes · 12 min read

LifeSim introduces a long‑horizon user life simulation framework that jointly models user cognition via a BDI engine and external environment, enabling realistic evaluation of personalized LLM assistants through the LifeSim‑Eval benchmark, which reveals current models excel at explicit intents but struggle with hidden intents and long‑term user understanding.
BDI modelLLM EvaluationLifeSim
0 likes · 9 min read

LLM judges often produce contradictory scores and non‑transitive preferences; the TrustJudge framework replaces discrete scoring with distribution‑sensitive scoring and likelihood‑aware aggregation, dramatically reducing both score‑comparison and pairwise‑transitivity inconsistencies across multiple model families, improving accuracy and even serving as a reward signal for RL training.
LLM EvaluationReward ModelingTrustJudge
0 likes · 12 min read

The article demonstrates how a monolithic LLM struggles with multi‑dimensional market analysis and shows, through step‑by‑step code, how assembling specialized AI agents for news, technical and financial analysis yields clearer structure, deeper insight, and higher evaluation scores.
AI ArchitectureLLM EvaluationLangChain
0 likes · 17 min read


SuanNiMar 8, 2026 · Artificial Intelligence
PinchBench, a rigorous benchmark that turns large language models into digital employees, measures success rate, execution speed, and per‑call cost across dozens of realistic office tasks, providing developers with concrete data to choose the most efficient model for their workloads.
AILLM EvaluationModel Performance
0 likes · 10 min read

With large language models moving from labs to production, systematic testing becomes a safety baseline; this article examines why traditional tests fail, showcases four open‑source toolchains (LlamaIndex + pytest, DeepEval, Promptfoo + LangChain, Great Expectations), presents an end‑to‑end e‑commerce case, and offers practical pitfalls to avoid.
AI safetyDeepEvalLLM Evaluation
0 likes · 8 min read

The ReLE framework introduces a dynamic, variance‑aware evaluation system that diagnoses capability anisotropy across 304 Chinese large language models, exposing ranking instability, commercial‑vs‑open‑source gaps, and format barriers while cutting evaluation cost by 70%.
AI assessmentCapability anisotropyChinese LLMs
0 likes · 9 min read


SuanNiFeb 27, 2026 · Artificial Intelligence
This article introduces the Deep Thought Ratio (DTR) metric, explains how tracking token modifications across neural network layers quantifies genuine inference effort, and shows through extensive experiments that DTR predicts accuracy far better than token length while enabling a sampling strategy that halves computational cost.
AI metricsLLM Evaluationchain-of-thought
0 likes · 9 min read

Anthropic’s newly released Claude Sonnet 4.6 model, featuring a 1 million‑token context window, is evaluated against the flagship Opus 4.5 across coding, long‑context reasoning, agent planning and other tasks, revealing mixed performance, user preferences, and detailed benchmark comparisons.
AI agentsAnthropicClaude Sonnet 4.6
0 likes · 5 min read

The January 2026 SCALE benchmark adds an index‑suggestion metric and evaluates two new LLMs—智谱 GLM‑4.7 and 字节跳动 Seed‑OSS‑36B—revealing strengths in dialect conversion, moderate SQL understanding, and notable gaps in complex execution‑plan analysis and practical index recommendations.
AI benchmarkingDialect ConversionLLM Evaluation
0 likes · 15 min read

The CL‑Bench benchmark reveals that current large language models fail to learn and apply new, long‑context knowledge, exposing critical gaps in context learning, scoring design, and error patterns across ten cutting‑edge models.
AI researchContext LearningLLM Evaluation
0 likes · 7 min read

In the era of rapidly advancing large‑model technology, the article outlines the challenges of evaluating data‑centric LLM agents, proposes a three‑layer evaluation framework covering basic capabilities, component‑level checks, and end‑to‑end business impact, and shares practical innovations such as semantic‑equivalence SQL matching, agent‑as‑judge pipelines, and a unified assessment platform.
Agent-as-JudgeData AgentLLM Evaluation
0 likes · 22 min read

This report evaluates the AI‑driven SQLFlash tool against the upgraded SCALE benchmark dataset, presenting core metrics on syntax compliance, logical equivalence, and optimization depth, and analyzes strengths, limitations, and future improvement directions for production‑grade SQL tuning.
AI modelsDatabase PerformanceLLM Evaluation
0 likes · 10 min read


AntTechDec 6, 2025 · Artificial Intelligence
FinEval‑KR, a new EMNLP2025 evaluation framework co‑authored by Shanghai University of Finance and Economics and Ant Group, separates knowledge coverage from logical reasoning to reveal why financial LLMs often hallucinate on calculation tasks, introduces KS, RS, and CS metrics, and ranks 18 state‑of‑the‑art models on a rigorously curated finance dataset.
Knowledge vs reasoningLLM Evaluationfinance AI
0 likes · 14 min read

This article explores how the rapid growth of large language models reshapes intelligent customer service, detailing the evolution from rule‑based NLP bots to Retrieval‑Augmented Generation (RAG) and AI‑native agents, and presents a comprehensive framework for evaluating, diagnosing, and continuously improving chatbot performance using LLM‑driven metrics and context engineering.
AICustomer ServiceLLM Evaluation
0 likes · 46 min read

This article details how Shopify engineered the Sidekick AI agent platform, covering its evolving architecture, just‑in‑time instruction system, rigorous LLM evaluation framework, GRPO training method, and strategies to prevent reward‑hacking, offering practical guidance for building production‑ready agentic systems.
AI agentsAgentic SystemsLLM Evaluation
0 likes · 13 min read

Shopify’s engineering team explains how they transformed the Sidekick AI assistant from a simple tool‑calling system into a robust, production‑grade AI agent platform, sharing architectural, evaluation and training lessons to help others avoid common pitfalls.
AI agentsGRPOJust-in-Time instructions
0 likes · 12 min read

This article surveys over 1,400 papers to define context engineering as a systematic discipline that structures retrieval, memory, tools, and multi‑agent coordination for LLMs, highlighting the critical asymmetry between understanding long contexts and generating equally complex outputs.
LLM EvaluationMulti-Agent AIRetrieval-Augmented Generation
0 likes · 8 min read

Meeseeks, a new benchmark released by Meituan’s M17 team, systematically evaluates large language models’ instruction‑following ability with a three‑tier framework, multi‑round self‑correction, and extensive real‑world data, revealing performance gaps among models such as OpenAI o‑series, Claude, DeepSeek and Qwen2.5.
AILLM EvaluationMeeseeks
0 likes · 13 min read

Large Language Models excel at passive reasoning, but struggle when information is incomplete; this paper defines the active reasoning problem, presents the AR‑Bench benchmark with detective, puzzle, and number‑guessing tasks, and reveals through extensive experiments that even top models like GPT‑4o perform poorly, highlighting research gaps.
LLM Evaluationactive reasoningbenchmark
0 likes · 13 min read

This article reviews a series of recent research papers on large‑model agents, covering topics such as reinforcement‑learning‑driven ML agents, premise‑critique ability of LLMs, long‑term tool‑augmented LLM evaluation, agentic RAG, set‑based retrieval for multi‑hop QA, mobile VLM agents, and broader surveys of LLM applications, summarizing each work’s problem statement, prior approaches, novel contributions, experimental results, limitations, and future directions.
Agentic AILLM EvaluationRetrieval-Augmented Generation
0 likes · 46 min read

The Meituan‑M17 team and Shanghai Jiao Tong University introduced two new benchmarks, OIBench and CoreCodeBench, to more accurately evaluate large language models' algorithmic and engineering coding abilities, revealing a substantial gap between claimed performance and actual capability across a range of tasks and models.
LLM Evaluationalgorithmic assessmentartificial intelligence
0 likes · 28 min read

The TailoredBench framework dramatically reduces large‑language‑model evaluation cost and error by using a global probe set, model‑specific source selection, extensible K‑Medoids clustering, and calibration, achieving up to 300× speedup and a 31.4% MAE reduction across diverse benchmarks.
AI researchK-MedoidsLLM Evaluation
0 likes · 10 min read

This article presents a hands‑on evaluation of several large language models—including Mistral‑Small, Deepseek‑r1, Llama 3.3 and ChatGPT‑4‑go—on Oracle database bad‑block errors, RAG‑based document retrieval, and log‑driven reasoning, revealing performance gaps, scoring results, and practical DBA implications.
AILLM EvaluationOracle
0 likes · 11 min read

This article explains how semantic enrichment, vector retrieval, hybrid search, and clustering can be combined to evaluate large language model inputs and outputs, improve debugging, ensure compliance, and enhance user intent understanding in AI applications.
AI operationsHybrid RetrievalLLM Evaluation
0 likes · 9 min read

This article critically examines R1‑Zero‑style training by analyzing foundation models and reinforcement learning, uncovering pre‑training and optimization biases, proposing an unbiased Dr. GRPO method, and demonstrating a minimalist 7B‑model recipe that achieves new state‑of‑the‑art performance on AIME 2024.
GRPOLLM EvaluationR1-Zero
0 likes · 20 min read

This article evaluates DeepSeek‑R1’s 32B and 70B distilled models alongside the original R1 on a range of reasoning and coding tasks, detailing hardware setup, test methodology, per‑task results, and a comparative analysis of their strengths and weaknesses.
32B70BDeepSeek
0 likes · 6 min read


ArchitectFeb 3, 2025 · Artificial Intelligence
This article presents DeepSeek‑R1 and DeepSeek‑R1‑Zero, two next‑generation LLMs trained with pure reinforcement learning and multi‑stage fine‑tuning, details their GRPO training framework, model‑distillation pipeline, open‑source release, and evaluation results that rival OpenAI’s o1‑1217 across reasoning, knowledge, and coding benchmarks.
DeepSeekLLM EvaluationOpenAI o1
0 likes · 10 min read

LalaEval is a comprehensive human‑evaluation framework that tackles enterprise challenges in building domain‑specific large language models by automating QA set generation, reducing evaluator subjectivity through controversy and score‑fluctuation analysis, and providing extensible, data‑driven metrics for model construction and iterative improvement.
AI benchmarkingLLM EvaluationLalaEval
0 likes · 11 min read


21CTONov 24, 2024 · Artificial Intelligence
OpenAI’s latest announcements introduce the GPT‑4o snapshot with superior creative writing and file‑upload capabilities, embed the Evals evaluation framework directly in the dashboard, and add audio support in Chat Completions, empowering developers to build more reliable and expressive AI applications.
API updatesGPT-4oLLM Evaluation
0 likes · 2 min read

This guide explains how to use Alibaba Cloud's PAI platform and the Themis judge model to efficiently evaluate large language models on custom or public datasets, covering data preparation, task submission, result analysis, multi‑model comparison, and API integration.
Alibaba CloudLLM EvaluationOpen Source LLM
0 likes · 10 min read


NewBeeNLPJul 10, 2024 · Artificial Intelligence
This article presents the COTEMPQA benchmark for evaluating large language models on co‑temporal reasoning, details its four scenario types, construction pipeline, experimental results across models, error analysis, and proposes the MR‑COT strategy that leverages mathematical reasoning to significantly improve performance.
LLM EvaluationMR-COTbenchmark dataset
0 likes · 11 min read

Open LLM Leaderboard v2 introduces a revamped, reproducible evaluation framework for large language models, replacing saturated benchmarks with six carefully curated, unpolluted datasets, applying standardized scoring, updating the harness, adding voting and maintainer‑recommended models, and providing richer visualizations to guide the AI community.
AI metricsCommunity VotingLLM Evaluation
0 likes · 19 min read

The LMSYS Chatbot Arena benchmark, using blind user voting and an Elo scoring system, placed China's Yi‑Large model among the top global large language models, detailing its methodology, ranking results, and the broader implications for the AI industry.
AI benchmarkingChatbot ArenaElo ranking
0 likes · 12 min read

The ICLR 2024 paper introduces a black‑box method for detecting test‑set leakage in large language models by comparing log‑probabilities of original and shuffled test orders, proposes a scalable sharded likelihood test, and demonstrates its effectiveness on several open‑source models, revealing a potential leak in Mistral‑7B.
LLM Evaluationlanguage model securityshuffled likelihood test
0 likes · 7 min read

This article explores how LangSmith, an experimental platform from the LangChain team, bridges the gap between prototype LLM applications and production by providing comprehensive tracing, debugging, testing, evaluation, and run‑management features that help developers monitor and improve generative AI systems.
AI observabilityLLM EvaluationLLM debugging
0 likes · 11 min read

A Stanford‑Berkeley study reveals that between March and June 2023 GPT‑4’s accuracy on prime‑checking fell from 97.6% to 2.4%, code generation quality dropped sharply, and sensitivity handling changed, underscoring the rapid, unpredictable shifts in large language model performance over short periods.
AI safetyGPT-4LLM Evaluation
0 likes · 6 min read

The article examines the rapid proliferation of large language models, weighing both the opportunities for experimentation and the drawbacks of noise, and argues that establishing authoritative Chinese LLM evaluation benchmarks is essential to guide meaningful progress in the field.
AI researchLLM Evaluationlarge language models
0 likes · 7 min read


21CTOApr 2, 2023 · Artificial Intelligence
This article reviews Microsoft’s extensive 155‑page work on early experiments with GPT‑4, exploring how the model approaches artificial general intelligence, its testing methodology, multimodal capabilities, programming and mathematical performance, interaction with tools and humans, limitations, societal impact, and future research directions.
AI safetyArtificial General IntelligenceGPT-4
0 likes · 15 min read
