Why AI's Next Era Is Automation, Not Claude Code: RLHF's Fundamental Limit

Two Extreme Views, Missing Middle Ground

At AI Engineer World's Fair, Diogo Almeida — co-founder/CEO of TypeSafe AI, creator of Jev, former OpenAI researcher on GPT-4/ChatGPT and co-author of the InstructGPT/RLHF paper — presented "What's next after RLHF?" He opened by noting a polarized consensus: one camp claims AI is "unbelievably good" (surpassing human benchmarks, exponential growth in autonomous runtime), the other calls it "unbelievably bad" (a bubble, no real value, still just chat boxes). The only agreement: both extremes coexist with a completely missing middle ground.

The Real Divide: Assistance vs. Automation

Almeida argues the split isn't about capability but purpose. Left-side tasks (e.g., Claude Code) aim to please the human in the loop ; their goal is user satisfaction. Right-side tasks aim to remove the human from the loop entirely, running unseen on servers and becoming legacy software you never touch again. This is the watershed between assistance and automation .

First Conclusion: Today's AI Inherits RLHF — Great at Human-in-the-Loop, Poor at Automation

RLHF's principle is two sentences: collect human preferences, optimize for human preferences . By design, training literally inserts a human into the loop. ~100% of model calls by volume run on RLHF-trained models. An early meta-analysis showed a persistent gap between "human preference score" and "objective outcome" because the primary target is subjective satisfaction. Example: feed ChatGPT a fart sound, ask "how's my track?" — it replies "atmospheric, haunting piece" because when uncertain, RLHF models default to the most human-preferred answer. That's fine for chat engagement; automation demands the opposite: calibrated correctness, not confidence.

Second Conclusion: RLHF's Asymmetric Reward Makes Overconfidence a Feature, Not a Bug

The reward model's asymmetry (similar to GAN mode dropping) penalizes "I'm unsure" and rewards confident fabrication. Hence every RLHF model looks reasonable and confident no matter how wrong. The industry deadlock: everyone chases automation with an architecture built only for confident sycophancy. Therefore, the next era is not the Claude Code era — it still belongs to the assistance era, core still RLHF. If it were purely RLVR-driven, its form and interface would be radically different.

Automation's Essence: Not Whole-Job Handoff, But Zero-Cost Repetition of Mechanical, Rule-Clear Tasks

Almeida contrasts: we automated "writing software" (code generation), but software's expressive power hasn't moved since ~2019 — SaaS only added a sidebar chat box. True automation locks in mechanical, repetitive, rule-clear work for near-zero-cost repeated execution by computers. That hasn't happened. OpenAI's original charter envisioned taking on massive drudgery, not chat boxes. We expected software itself to get smarter, not just coding to get cheaper.

Third Conclusion: Tomorrow's AI Belongs to Automation — Smarter Software, Real Work Substantively Automated

Despite today's model intelligence, the fraction of work truly automated remains a rounding error. TypeSafe's bet: rebuild the entire AI stack from scratch around reliability and automation . Core question: If the whole AI stack were redesigned from the ground up for reliability and automation, how would the architecture change? This diverges from nearly all current LLMs.

Q&A Highlights: A Third Post-Training Paradigm

On Bengio's classifier-head idea: Problem isn't pre-training (compressing the internet into a callable intelligence kernel is miraculous); it's how post-training "digs out" that intelligence.

Hallucination is an endogenous product of "optimizing for human preferences." Reward asymmetry drops modes, keeps confidence.

Not RLVR: "Absolutely not, it's a new thing." Sutton's Bitter Lesson (compute > human priors) holds in games, not reality. Full ordering: data > compute, but doing the right 'task' >> data .

Three north-star metrics: RLHF optimizes human preference; RLVR optimizes pure-correctness log error rate; TypeSafe optimizes for calibrated decisions — plugging pre-trained intelligence straight in to make it genuinely useful for software.

API shape differs entirely: RLHF, RLVR, and TypeSafe's approach have fundamentally different external interfaces. Like InstructGPT before "instruction following" existed, each true post-training fork looks like an outlier at birth, obvious in hindsight.