OpenAI: API Tweaks Triple ARC-AGI-3 Scores

The Devil's in the Harness Details

OpenAI's revelation that simple API settings—retained reasoning and compaction—can dramatically improve performance on benchmarks like ARC-AGI-3 is a powerful demonstration of how evaluation environments shape AI capabilities. This finding underscores a critical, often overlooked, aspect of AI development: the 'harness' or testing framework is not a neutral observer but an active participant that can either amplify or obscure a model's true potential. The implication is that benchmark scores, especially for complex reasoning tasks, are not solely a reflection of the model's inherent intelligence but also of the sophistication of its deployment context. This raises questions about the comparability of scores across different research labs and benchmark implementations if these 'less visible choices' are not standardized or transparently reported.

While the results are impressive, the article could benefit from a deeper dive into the qualitative changes in GPT-5.6 Sol's problem-solving approach when these settings are enabled. Beyond improved scores and reduced token usage, understanding how the model's strategy or reasoning process evolves would offer more profound insights. Furthermore, the reliance on a 175,000-token limit for context, even with compaction, still represents a constraint. The long-term implications for models operating with truly unbounded or significantly larger context windows remain an open area for exploration. The recommendation to switch from the legacy Chat Completions API to the Responses API, while practical for developers aiming for similar performance, also nudges users towards OpenAI's newer, potentially more feature-rich, but also proprietary, infrastructure.

Key Points

• Two specific API settings, 'retained reasoning' and 'compaction', significantly boosted GPT-5.6 Sol's performance on the ARC-AGI-3 benchmark.
• The ARC-AGI-3 benchmark's generic harness discarded private reasoning and used rolling truncation, hindering the model's ability to learn and strategize.
• Enabling retained reasoning allowed the model to remember its thought process, reducing per-action thinking time and improving learning over time.
• Compaction, replacing rolling truncation, helped preserve learned information over longer runs, leading to higher scores with fewer output tokens.
• Together, these settings tripled scores and reduced output tokens by 6x on the public task set, demonstrating the crucial impact of evaluation environment design.
• OpenAI recommends using their Responses API with retained reasoning and compaction for maximizing performance and for fair model comparisons, aligning with their production setups in ChatGPT and Codex.

---

📖 Source: How enabling two settings tripled our scores on the ARC-AGI-3 benchmark