What was tested
On July 29, OpenAI investigated why GPT-5.6 Sol scored poorly on ARC-AGI-3, a set of unfamiliar 2D games in which an agent must infer rules from interaction rather than instructions.
The generic harness discarded private reasoning after every action and removed older actions as the context grew. OpenAI rebuilt it with the Responses API, retaining reasoning through previous response IDs and replacing rolling truncation with compaction.
Reported results
OpenAI reports that GPT-5.6 Sol scored 13.3% RHAE on the public set with the official harness and 38.3% with retained reasoning and compaction. Output-token use fell by roughly six times. OpenAI estimated an average human score of 48% from official gameplay logs.
The company attributes the difference to preserving plans and observations instead of forcing the model to reinterpret the game after every move. Compaction also retained early discoveries and failed attempts longer than a rolling cutoff.
Why it matters
A benchmark measures more than model weights. It also measures API settings, tools, context policy, and harness design. Rankings can change substantially if one model loses its reasoning state or runs under a context policy unlike its production environment.
For developers, the practical lesson is to preserve compressed goals, observations, failures, and decisions in long-running agents, then measure task success and total cost together.
Limitations
This is a vendor-run experiment using OpenAI's own model and API. The multiplier comes from one public task set and two harnesses; it does not establish the same effect for other models or workloads. Fair comparisons should disclose harness settings and separate common-condition results from product-optimized results.