How Two Settings Tripled GPT‑5.6 Sol’s ARC‑AGI‑3 Score

OpenAI disclosed that altering just two API parameters can dramatically improve the performance of its GPT‑5.6 Sol model on the ARC‑AGI‑3 benchmark.

What is ARC‑AGI‑3?

ARC‑AGI‑3 is a benchmark designed to evaluate an AI agent’s ability to learn and reason over long‑term interactive tasks. Unlike standard question‑answer tests, it presents a novel 2‑D game environment without explicit rules, requiring the agent to explore, observe outcomes, and gradually infer the game mechanics.

Initial performance and the problem

When run with the official test harness, GPT‑5.6 Sol achieved only 13.3% on the public test set. OpenAI’s investigation revealed that the low score was not due solely to model capability but to two limitations of the harness:

Reasoning discarded: After each game action, the model’s private reasoning trace was deleted, forcing it to re‑interpret the current state from scratch.

Context truncation: The harness used a rolling‑cutoff mechanism that removed the earliest messages once the context window overflowed, causing loss of early observations and plans.

These factors prevented the agent from retaining long‑term memory and building cumulative strategies.

Two API settings that fix the issues

The breakthrough came from switching to a runtime mode closer to ChatGPT and Codex, specifically enabling:

Retained reasoning: The model’s internal reasoning steps are kept as part of the context, allowing subsequent actions to build on previously generated thoughts.

Context compression: Instead of bluntly deleting old messages, the system summarizes earlier information, preserving essential content while staying within the context‑length limit.

With these settings, the model no longer needs to re‑analyze the game at every step and can reuse past insights.

Resulting performance gains

Under the modified framework, GPT‑5.6 Sol’s ARC‑AGI‑3 score rose to 38.3% , roughly a three‑fold increase, and the number of output tokens dropped by about six times.

Implications

OpenAI emphasizes that benchmark results are influenced not only by the model itself but also by API configurations, evaluation harness design, and prompting style. Developers seeking maximal performance should align their API settings with real‑world usage, and researchers comparing models should consider evaluations that incorporate these settings for a fairer assessment.