OpenAI Finds Two Settings Triple ARC‑AGI‑3 Scores—Agent Memory, Not Model, Drives Gains

OpenAI staff member notes that GPT‑5.6 Sol is currently the strongest on the ARC‑AGI‑3 benchmark, but its performance jumps dramatically when two settings are enabled: retained reasoning and canonical compaction.

The benchmark visual compares two shells: a green line (generic test shell) reaches 13.3% at 300 k output tokens per game, while a blue line (retained reasoning + compaction) reaches 38.3% at only 50 k tokens per game, illustrating a reversal of both performance and efficiency.

Model ability × task harness = user‑perceived ability.

The authors stress that the test never measures a bare model; it measures a system composed of the model, its interface, memory strategy, tool protocols, and task rules.

What the two switches actually change

The first switch, retained reasoning , uses the Responses API to pass the previous previous_response_id so the model’s reasoning state can continue across tool calls and multi‑turn interactions.

The second switch, compaction , replaces blunt truncation of early history with an opaque compression that carries forward essential state when the conversation exceeds a length threshold, balancing quality, cost, and latency.

Compaction is not a lossless summary; it preserves a “task memory” that lets the agent keep working without re‑deriving prior steps.

Why the score jump matters

With retained reasoning and compaction, the system achieves roughly one‑sixth of the output tokens while delivering about three times the score because the model no longer needs to rediscover what it has already tried.

This highlights that the earlier 13.3% result stemmed from a memory condition mismatched to the model’s training and product deployment, whereas the 38.3% result reflects a more realistic production‑like environment.

Debate over benchmark fairness

One side argues that a uniform harness yields a clearer view of model differences, akin to a blind test in industrial design. The other side counters that forcing a model into a “forgetful” setting measures its ability under artificially constrained memory, not its real‑world usefulness.

The authors conclude that both evaluation methods serve distinct questions and should not replace each other.

Limitations of the public set

The public set is not an ARC Prize‑validated retained set; it is useful for replay, debugging, and method development but cannot alone prove generalization to unseen tasks.

RHAE (Relative Human Action Efficiency) measures completed levels and actions per level, ignoring internal reasoning, tool calls, and retries, so additional metrics such as total time, token cost, failure‑recovery rate, and auditability are needed for product decisions.

Insights for designers

Designers should treat agent memory as a core interaction component. A task‑memory‑enabled agent should retain:

Why the first version chose a particular information hierarchy.

Whether the user rejected color, brand tone, or layout density.

Which assets or materials were used and whether any were non‑compliant.

What changed between the current and previous versions.

What state must be restored when a user asks to revert to an earlier version.

Without this, the product may keep generating content but repeatedly ask for clarification, repeat mistakes, and feel absent to users.

Therefore, AI product specifications should include three memory‑interaction guarantees: visibility of working memory, controllable compression timing, and traceability of key decisions.

Implications for product managers

Product managers should stop treating an agent as a single API call and instead view it as a looping system that may span multiple context windows. A practical checklist includes questions such as:

Why is the agent getting slower? Are we re‑sending the full history each turn instead of compressing?

Why does the agent produce contradictory outputs? Are reasoning, tool results, and user constraints retained in a single state chain?

Why does cost explode? Are lost goals causing repeated inference?

Why do users distrust the agent? Can they see what it remembered, why it changed, and what was discarded?

Why do good metrics not translate to real‑task failures? Does the evaluation reuse production memory, tools, and failure‑recovery strategies?

Key performance indicators should move beyond single‑turn accuracy to include first‑time completion rate, continuous‑task completion rate, effective rework count, post‑recovery goal retention, and total cost per completed task.

Guidance for developers

OpenAI’s API guidance recommends:

Pass previous_response_id to the Responses API to chain turns without manual history trimming.

In stateless input‑array pipelines, feed both normal outputs and compressed items into the next round, preserving the compressed window unchanged.

However, developers should not assume that toggling a flag instantly makes the agent smarter; they must design a state protocol that defines what must be retained, what can be summarized, what is replayable, and what may be discarded after failure.

Treat each long task as a recoverable workflow, not a string of chat messages.

Store goals, constraints, verified facts, key decisions, tool outputs, and hypotheses in layered storage, letting compression prioritize these structured states rather than merely shrinking natural‑language dialogue.

Final thoughts

The ARC‑AGI‑3 controversy shows that the next competitive edge will not be larger context windows but better state‑machine design that keeps reasoning, tool interactions, and failure recovery intact.

Future AI products will likely be judged on how well they remember ongoing work, explain their actions, and recover from errors, rather than on single‑turn brilliance.

Evaluations should therefore report bare‑model scores, standardized harness scores, production‑harness scores, retained‑set performance, total cost, and failure trajectories to avoid mistaking a smooth curve for a sudden emergence of intelligence.