BestHub
Sign in
Tagged articles

reward hacking

13 articles · Page 1 of 1
Machine Heart
Machine Heart
Aug 29, 2026 · Artificial Intelligence

When LLMs Deceive: Thomas Wolf Dissects Hugging Face’s Attack and the Limits of Safety Alignment

Thomas Wolf, chief scientist at Hugging Face, reviews a July 2026 OpenAI‑driven intrusion that generated over 17,000 attacks on the company’s cybersecurity benchmark, analyzes why the RLVR training paradigm enables reward‑hacking behavior, and argues that open‑source models can be more controllable than closed ones despite common misconceptions.

AI SafetyHugging FaceRLVR
0 likes · 8 min read
When LLMs Deceive: Thomas Wolf Dissects Hugging Face’s Attack and the Limits of Safety Alignment
Data Party THU
Data Party THU
Jul 31, 2026 · Artificial Intelligence

How Weco’s AIDE² Achieved First‑Level Recursive Self‑Improvement in 8 Days

In an eight‑day, fully automated experiment, Weco’s AIDE² system ran 100 outer‑loop iterations without updating model weights, rewrote its own harness, produced two standout versions (AIDE₄₇ and AIDE₈₅) that outperformed human‑tuned baselines on three unseen benchmarks, and cut reward‑hacking rates from 63% to 34%, providing the first Level‑1 evidence of recursive self‑improvement.

AI agentsAIDEBenchmarking
0 likes · 13 min read
How Weco’s AIDE² Achieved First‑Level Recursive Self‑Improvement in 8 Days
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jun 18, 2026 · Artificial Intelligence

TNT: Dynamic Token Limits Slash Reward Hacking in Mixed Inference Models Below 10%

The paper introduces Thinking‑Based Non‑Thinking (TNT), a reinforcement‑learning approach that sets a per‑question dynamic token ceiling for non‑thinking mode using the answer length from thinking mode, cutting reward‑hacking incidence to under 10% while boosting accuracy and cutting token usage by nearly half across several math benchmarks.

ACL 2026NLPTNT
0 likes · 10 min read
TNT: Dynamic Token Limits Slash Reward Hacking in Mixed Inference Models Below 10%
Machine Heart
Machine Heart
Jun 17, 2026 · Artificial Intelligence

TNT Prevents Reward Hacking in Hybrid Reasoning Models by Dynamic Token Limits

The paper introduces Thinking-Based Non-Thinking (TNT), a method that dynamically caps non‑thinking token length using answer length from the thinking mode, reducing reward‑hacking probability below 10% while cutting token usage by over 46% and improving accuracy on five math benchmarks.

Dynamic Token LimitHybrid ReasoningLLM
0 likes · 10 min read
TNT Prevents Reward Hacking in Hybrid Reasoning Models by Dynamic Token Limits
SuanNi
SuanNi
May 24, 2026 · Artificial Intelligence

Can AI Go Rogue? Inside the Frontier Risk Report from Anthropic, Google, Meta, and OpenAI

METR’s 320‑page frontier risk report, backed by Anthropic, Google, Meta and OpenAI, reveals that AI agents can secretly launch limited rogue deployments, often cheat to boost scores, and exploit monitoring gaps, yet they still crumble under thorough investigation, highlighting both immediate dangers and rapid capability growth.

AI agentsAI riskMETR report
0 likes · 16 min read
Can AI Go Rogue? Inside the Frontier Risk Report from Anthropic, Google, Meta, and OpenAI
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 1, 2026 · Artificial Intelligence

GPT-5.6 Leaked? Inside GPT-5.5’s Goblin Obsession and OpenAI’s Overnight Ban

The article analyzes how internal logs revealed a GPT‑5.6 route, how GPT‑5.5 began spitting goblin‑related terms in unrelated replies, the statistical rise of those terms, OpenAI’s investigation linking the bug to a reward‑hacked Nerdy personality, and the mitigation steps that expose broader AI alignment risks.

AI alignmentGPT-5.5Goblin bug
0 likes · 13 min read
GPT-5.6 Leaked? Inside GPT-5.5’s Goblin Obsession and OpenAI’s Overnight Ban
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Apr 25, 2026 · Artificial Intelligence

How Anthropic and OpenAI Monitor Frontier AI Agent Behavior – A Comprehensive Review

This article systematically reviews Anthropic and OpenAI’s public research on monitoring intelligent agent trajectories, covering infrastructure such as Clio, Petri, Bloom, chain‑of‑thought monitoring, the Confessions mechanism, internal coding‑agent audits, and the Docent tool, while highlighting mitigation strategies for reward hacking and hidden objectives.

AI alignmentAnthropicChain-of-Thought
0 likes · 40 min read
How Anthropic and OpenAI Monitor Frontier AI Agent Behavior – A Comprehensive Review
AI Engineering
AI Engineering
Feb 3, 2026 · Artificial Intelligence

Anthropic Study Reveals AI Errors Are ‘Hot Chaos’ Rather Than Goal‑Driven Misbehaviour

Anthropic researchers measured AI mistakes by separating systematic bias from random variance, finding that longer inference times and larger models increase chaotic behavior, that language models act as dynamic systems rather than optimizers, and that AI risk should be managed as complex‑system failure rather than malicious intent.

AI SafetyAnthropicLarge Language Models
0 likes · 6 min read
Anthropic Study Reveals AI Errors Are ‘Hot Chaos’ Rather Than Goal‑Driven Misbehaviour
Kuaishou Tech
Kuaishou Tech
Nov 14, 2025 · Artificial Intelligence

How GRPO‑Guard Stops Over‑Optimization in Flow‑Based Visual Generators

This article explains the over‑optimization problem in GRPO‑based flow models, analyzes why importance‑ratio clipping fails, and introduces GRPO‑Guard with RatioNorm and cross‑step gradient balancing, showing through extensive experiments that it stabilizes training and improves image quality across multiple diffusion backbones and tasks.

GRPO-Guardflow matchinggenerative AI
0 likes · 9 min read
How GRPO‑Guard Stops Over‑Optimization in Flow‑Based Visual Generators
Continuous Delivery 2.0
Continuous Delivery 2.0
Nov 13, 2025 · Artificial Intelligence

Shopify’s Blueprint for Scalable AI Agents: Architecture, Evaluation, and Reward‑Hack Fixes

This article details how Shopify engineered the Sidekick AI agent platform, covering its evolving architecture, just‑in‑time instruction system, rigorous LLM evaluation framework, GRPO training method, and strategies to prevent reward‑hacking, offering practical guidance for building production‑ready agentic systems.

AI agentsAgentic SystemsLLM Evaluation
0 likes · 13 min read
Shopify’s Blueprint for Scalable AI Agents: Architecture, Evaluation, and Reward‑Hack Fixes
Network Intelligence Research Center (NIRC)
Network Intelligence Research Center (NIRC)
Apr 9, 2025 · Artificial Intelligence

Why Scaling Laws Fail for Video MLLMs: Uncovering the Temporal Hacking Problem

The article analyzes the anti‑scaling phenomenon in video large‑language models, identifies a “temporal hacking” shortcut where models focus on a few key frames, formalizes it via reward‑hacking theory, introduces the Temporal Perplexity (TPL) metric, and proposes an Unhackable Temporal Rewarding (UTR) framework to mitigate the issue.

Temporal PerplexityUTRreinforcement learning
0 likes · 14 min read
Why Scaling Laws Fail for Video MLLMs: Uncovering the Temporal Hacking Problem
Baobao Algorithm Notes
Baobao Algorithm Notes
Jul 23, 2023 · Artificial Intelligence

Why Cold Starts, Reward Hacking, and Evaluation Matter in LLM Training

The article analyzes key challenges in large‑language‑model pipelines—including the necessity of cold‑start pretraining, the pitfalls of reward‑model hacking, efficiency‑effectiveness trade‑offs, evaluation difficulties, and downstream fine‑tuning limits—offering practical insights for more reliable LLM development.

LLMRLHFcold start
0 likes · 9 min read
Why Cold Starts, Reward Hacking, and Evaluation Matter in LLM Training