The 37-point gap in GPT-6 Astra's benchmark

When OpenAI unveiled GPT-6 Astra on September 3, 2026, the headline number was impossible to miss: 99.9% on ARC-AGI-3, a benchmark designed to test whether AI can learn unfamiliar tasks the way humans do. Greg Brockman, OpenAI's president, called it the start of the AGI era. Two days later, the benchmark's own creators published a different number: 62.7%. Same model. Same test. A 37-point gap that has nothing to do with the model and everything to do with how you measure it.

Two harnesses, two report cards

ARC Prize Foundation, the organization that built ARC-AGI-3, ran GPT-6 Astra through both of its evaluation setups on September 3, 2026. The results, published September 5, 2026, tell a story about scaffolding rather than raw intelligence:

Source: ARC Prize Foundation, September 5, 2026

The Standard harness is deliberately minimal. After each action, the model's private reasoning is discarded. Only the notes it chose to write survive, carried forward in visible form. The Provider Adapter keeps the model's opaque reasoning state alive between requests — state that evaluators cannot see — and compacts long conversations. One setup plays the game as a nearly amnesiac loop, re-deriving everything each turn. The other plays as an agent with working memory.

Turn memory on, and effort barely matters: every Provider Adapter row sits between 96.7% and 99.9%. Under the Standard harness, effort is the whole game, ranging from 17.5% to 62.7%. OpenAI's launch blog reported the 99.9% figure — the Provider Adapter result at high effort — against GPT-5.6 Sol at 7.8% and Claude Opus 5 at 30.2%, both measured under the Standard harness.

OpenAI's official ARC-AGI-3 chart, as published in the launch blog. Source: OpenAI

A footnote in the blog acknowledged that Astra ran on OpenAI's responses-API harness, which "changes two settings to better match real-world performance." The footnote also estimated that Sol would score "in the ballpark of ~30%" under that same harness — meaning the 7.8% figure used for comparison was not apples-to-apples.

The blog post that kept changing

The harness gap is only half the story. Forensic analysis of archived versions of OpenAI's launch blog, first reported by Detailler on September 5, 2026, shows that benchmark figures were altered multiple times after the post went live — and in some cases, competitor numbers moved in ways that made Astra look better.

The launch itself was chaotic. OpenAI scheduled the blog for 2:00 p.m. ET (UTC-4) on September 3, 2026. It briefly appeared, then vanished. The official X account posted a link at 3:32 p.m. (UTC-4) that returned an error. Sam Altman posted it himself at 3:50 p.m. (UTC-4), saying "We hit a little snag getting the blog post deployed." It wasn't until roughly 5:00 p.m. (UTC-4) that the post loaded reliably for most users.

Among the changes documented across archived snapshots:

• Hallucination rate: Astra's figure was 4.2% in the 2:23 p.m. (UTC-4) snapshot, dropped to 2% by 5:20 p.m. (UTC-4), then reverted to 4.2% in later checks. Sol's rate moved from 12.2% to 9.4% and back.
• FrontierMath Tier 4: Anthropic's Fable 5.1 started at 87.8%, fell to 78% by 5:17 p.m. (UTC-4), then rebounded to 83%. Astra's score stayed at 97.6% throughout.
• ExploitBench: Sol's score moved from 5.5% to 11.5%, with OpenAI later saying it was investigating reverting to 5.5% because the higher number reflected "a reasoning level that is not commercially available for Sol."
• ARC-AGI-3: An embargoed draft provided to media listed 98.6%. The live blog showed 99.99%.

An OpenAI spokesperson told Fortune: "We care deeply about getting evaluations right. Most evaluations have noise within a few percentage points based on the exact checkpoint, scaffold, and evaluation run used in reporting. For our launch blog, we made fixes to ensure the numbers represent our best estimate of available model performance."

This isn't just an OpenAI problem

The practice of optimizing benchmark conditions until the most favorable number appears has a name in the industry: benchmaxxing. It's not illegal, and it's not unique to OpenAI. In 2025, Meta faced backlash after reports that it published Llama 4 scores from an internal, more optimized version rather than the one released publicly. Yann LeCun later acknowledged the company had "fudged" the results.

Stanford researchers Anka Reuel and Mike Hardy, from the Intelligent Systems Laboratory and Trustworthy AI Lab, told Detailler that rapid benchmark adjustments "can be done in a very tight timeframe, and it's better for their marketing." They also criticized the GPT-6 Astra system card for providing "barely any details about the evaluation" of internal benchmarks, not even the number of test items.

Vincent Sunn Chen, an AI engineer at Snorkel AI, offered a more charitable view: "It's usually a function of final launch logistics. A benchmark score reflects a specific measurement setup: the model checkpoint, configuration, harness, eval/grading configuration. All of those are typically still shifting in the final days before a launch."

ARC Prize's dual-harness leaderboard, showing the cost-score tradeoff. Source: ARC Prize Foundation

What this actually means

The 37-point gap is not evidence that GPT-6 Astra is weak. Under the Standard harness, 62.7% is still a substantial lead over Claude Opus 5 at 30.2% and GPT-5.6 Sol at 7.8%. ARC Prize itself called Astra "a noticeable step-function change in frontier model capabilities."

The problem is that the headline number — 99.9%, "saturates" the benchmark, AGI era — came from a setup where the model's own production scaffolding does much of the work. When ARC Prize tested the same model under the neutral Standard interface, the story changed. And the post-launch edits to the blog, including temporary dips in competitor scores, have made the whole launch look less like a clean measurement and more like a marketing document that was still being written after it went live.

ARC Prize has already committed to a fix: its public leaderboard will now carry both harness conditions, clearly labeled, so no vendor figure can float free of its scaffold. The rest is replication — a third party running Astra and its rivals on the Standard harness at matched reasoning effort, publishing cost per percentage point alongside raw scores.

Until that exists, "Astra saturates ARC-AGI-3" should be read the way ARC Prize's designers intend it: a statement about a model inside a very capable harness. A real achievement, and not yet a claim about general intelligence. The 37-point gap isn't a bug in the reporting. It's the finding.