Why does GPT-6 Astra have two different ARC-AGI-3 scores?

Because ARC Prize ran it on two different harnesses. On the Standard harness, GPT-6 Astra scored 62.7% on the ARC-AGI-3 Semi-Private set for roughly $26,000 of compute. With a Provider Adapter that preserves opaque reasoning state across turns and compacts longer conversations, the same model scored 99.9% for about $19,000. Both numbers are real. Only the 62.7% is comparable to other models' standard-harness results, because the adapter gives the model a capability — persistent reasoning state — that the standard harness deliberately withholds.

What is a benchmark harness?

A harness is the scaffolding that connects a model to a benchmark: how the task is presented, how many attempts are allowed, whether tools are available, whether reasoning state persists between turns, and how long conversations are compacted or truncated. Two harnesses running the identical model on the identical benchmark can produce wildly different scores. This is why a benchmark result without its harness described is close to meaningless, and why vendor comparison tables that mix harness classes are unreliable.

What is the real ARC-AGI-3 comparison between models?

On ARC Prize's official Standard harness as of September 3, 2026: GPT-6 Astra 62.7%, Claude Opus 5 30.2%, GPT-5.6 Sol 7.8%. That is the apples-to-apples ranking, and Astra's lead is genuine — it more than doubles Opus 5 and beats GPT-5.6 Sol by roughly 8x. The widely circulated near-perfect figure came from the Provider Adapter harness and should never be placed in the same column as another model's standard-harness score.

How do I read AI benchmark claims safely?

Check five things. First, who ran it — vendor self-report or independent evaluator. Second, which harness, and whether every model in the table used the same one. Third, what reasoning effort level, since max effort inflates scores and costs. Fourth, the compute cost, which ARC Prize publishes and most vendors do not. Fifth, whether the benchmark is the public set or a semi-private held-out set, since public sets leak into training data. If a chart omits all five, treat it as marketing.

Quick Answer

ARC-AGI-3: Why GPT-6 Astra Scored 62.7% and 99.9%

Published: September 4, 2026

The Short Answer

GPT-6 Astra scored 62.7% and 99.9% on the same benchmark, and both numbers are correct. The difference is the harness — the scaffolding around the model — not the model itself.

62.7% on ARC Prize’s Standard harness, for roughly $26,000 of compute. This is the comparable number.
99.9% with a Provider Adapter that preserves opaque reasoning state and compacts long conversations, for about $19,000.

Charts that put the near-perfect figure next to another model’s standard-harness score are comparing different experiments. Last verified: September 4, 2026.

The Apples-to-Apples Table

ARC Prize published official ARC-AGI-3 results on September 3, 2026, on the Semi-Private evaluation set using the Standard harness:

Model	ARC-AGI-3 (Standard harness)
GPT-6 Astra	62.7%
Claude Opus 5	30.2%
GPT-5.6 Sol	7.8%

Astra’s lead here is real and large — it more than doubles Claude Opus 5 and beats GPT-5.6 Sol by roughly 8x. Nothing about the harness controversy undermines that. The problem is only that a second, much higher number entered circulation without its context attached, and comparison tables started mixing the two.

What a Harness Actually Is

A benchmark measures a model. A harness is everything around the model that determines how it gets to attempt the benchmark:

How the task is presented — raw, templated, or with worked examples.
How many attempts are permitted, and whether failures are retried.
Which tools the model may call.
Whether reasoning state persists between turns, or is discarded each time.
How long conversations are handled — truncated, compacted, or preserved whole.
What reasoning effort level the model runs at.

Change any of these and the score moves. Change several and the score can move by tens of points. This is not cheating; it is a legitimate design space. It only becomes misleading when the harness is dropped from the reporting.

Why the Provider Adapter Changed So Much

The Standard harness deliberately discards reasoning state between turns. The model gets the task, thinks, answers, and its internal chain is not carried forward. That is a hard test of whether the model can re-derive its own reasoning from scratch.

The Provider Adapter preserves opaque reasoning state across turns and compacts longer conversations rather than truncating them. In effect the model keeps its own working memory intact across a multi-turn attempt.

For most benchmarks that would help modestly. For ARC-AGI-3 — an abstract-reasoning suite where the whole difficulty is holding and refining a hypothesis — it is close to changing the task. Hence 62.7% becoming 99.9%.

Note the cost inversion too: the higher score cost less ($19,000 vs $26,000), because preserved state means less re-derivation and fewer tokens. That is a genuinely useful engineering finding, and it is the reason the adapter result is worth publishing at all — just not worth publishing in the same column as a standard-harness score.

This Is Not an Isolated Case

Harness ambiguity is the single most common source of bad AI benchmark charts in 2026. Three patterns to watch for:

Effort-level mixing. OpenAI notes that unless stated otherwise, its published GPT-6 Astra evaluations ran at maximum effort. Models with a low→max reasoning dial score very differently across it, and cost differently too. A table comparing one model at max against another at default is not a comparison.

Self-reported versus independent. The leaked GPT-6 Astra launch table showed Astra ahead of Claude Fable 5.1, Fable 5 and Opus 5 on every comparable benchmark. Independent measurement told a different story: Artificial Analysis scored Fable 5.1 at 65.6 on the Intelligence Index against Astra’s 61.1. Anthropic’s own Fable 5.1 figures were published with production safeguards enabled and are reproducible. Not all vendor numbers are equal.

Public versus held-out sets. ARC Prize evaluates on a Semi-Private set precisely because public benchmark sets leak into training data. A model scoring well on a public set may be recalling, not reasoning. Always check which set.

How to Read a Benchmark Claim in Five Questions

Before you let a number influence a model decision, ask:

Who ran it? Vendor self-report, or an independent evaluator like Artificial Analysis, ARC Prize or LMArena.
Which harness — and did every model in the table use it? If the harness is unnamed, the table is unusable for cross-model comparison.
What reasoning effort? Max-effort results describe a configuration you probably will not run in production.
What did it cost? ARC Prize publishes compute cost per evaluation. Most vendors do not. A score you cannot afford to reproduce is not a capability you have.
Public or held-out set? Semi-private and private sets are the trustworthy ones.

A chart that answers none of these is marketing. A chart that answers all five is rare and worth trusting.

The Practical Takeaway

For model selection: use the standard-harness ARC-AGI-3 ranking. GPT-6 Astra genuinely leads abstract reasoning by a wide margin, and that is a good reason to route hard reasoning work to it.

For your own evaluations: the Provider Adapter result is a hint worth acting on. Preserving reasoning state across turns made the same model dramatically more capable and cheaper on a hard multi-turn task. If your agent framework discards model reasoning state between steps — many do, by default, to save tokens — you may be paying more for worse results. Test it.

For anything you publish or forward: cite the harness. The reason ARC Prize’s numbers are trustworthy is that they arrive with the harness, the set, and the compute cost attached. That is the standard to hold vendors to.