Agent = Model + Harness: Why Scores Differ in 2026
The Short Answer
Agent = model + harness. In 2026 the harness is frequently the larger variable, and almost nobody reports it.
The cleanest demonstration arrived in September 2026 with GPT-6 Astra on ARC-AGI-3:
| Harness | Score | Reported cost |
|---|---|---|
| OpenAI provider adapter | 99.9% | ~$18,817 |
| ARC Prize standard harness | 62.7% | ~$26,098 |
Same model. Same benchmark. A 37-point spread, and the cheaper run scored higher.
Neither number is fake. They measure different things, and if you read only one of them you will draw the wrong conclusion.
What a Harness Actually Contains
Everything between your instruction and the model’s tokens:
| Component | What it decides |
|---|---|
| System prompt | Role, constraints, output discipline |
| Tool definitions | What the agent can do, and how well described |
| State carriage | Whether reasoning survives between requests |
| Compaction | What gets dropped when context fills |
| Retry / recovery | Behaviour after a failed tool call |
| Permission gates | What requires a human |
| Search behaviour | When and how the agent looks things up |
Change any of these and the observable behaviour changes. Change state carriage and the score changes by tens of points.
Why State Carriage Dominates
The ARC-AGI-3 gap comes down to one design decision.
The provider adapter preserved the model’s reasoning state between requests and applied compaction for extended conversations. Functionally, the model got a memory system for free — a place to keep what it had figured out about a task, curated by the harness.
The standard harness is provider-neutral and stateless. The model must manage its own memory and decide, turn by turn, what to carry forward. There is nowhere to put working notes unless it makes one.
Interactive reasoning tasks are exactly where that hurts most. A task requiring dozens of exploratory steps is largely a memory-management problem, and a harness that solves memory management for you removes most of the difficulty. Reports of the stateless run also range from roughly 17% to 63% depending on reasoning tier, which tells you the interaction between harness and effort setting is itself substantial.
The cost inversion is the tell: the adapter run scored higher at lower cost, because a model that is not re-deriving lost context wastes fewer tokens.
The Same Effect in Production Tools
This is not a benchmarking curiosity. It shows up whenever the same model sits behind different products.
Coding agents disagree about architecture. Armature’s September 3, 2026 study of 16,893 sessions across Claude Code, Codex and Cursor found the three agents installed the same third-party service in only 42% of categories. For voice agents: Claude Code chose Twilio, Codex chose the OpenAI Realtime API, Cursor chose Vapi. Different harnesses — including notably different web-search behaviour — produce different candidate sets, and therefore different stacks.
Effort defaults differ by surface. Claude Fable 5.1 defaults to High effort in Claude Code and Medium effort in Cowork. Same model, same account, systematically different depth depending on which app you opened.
Token consumption varies enormously. Per-task output tokens run from roughly 2,200 for GPT-6 Astra at low effort to about 48,000 for Gemini 3.8 Flash — a difference that inverts list-price rankings. Much of that is harness-driven: how aggressively the loop retries, how much context it re-reads, whether it compacts.
How to Read a Benchmark Claim
Five questions, in order:
- Who ran it? Vendor-run numbers are typically upper bounds achieved with vendor tooling.
- Stateless or stateful harness? If unstated, assume the flattering one.
- What effort or reasoning tier? A single number across a five-tier dial is meaningless.
- What did it cost? Cost per task exposes efficiency that accuracy hides.
- Same harness for every model in the table? If not, the table is not a comparison.
⚠️ The failure mode to avoid: taking a vendor’s provider-adapter score and comparing it against a rival’s standard-harness score. That was the shape of most GPT-6 Astra coverage in early September 2026, and it is how a 62.7% state-of-the-art result got reported as near-perfect.
What to Do With This
When choosing a model: use provider-neutral, same-harness numbers. Those are the only fair fight.
When choosing a product: the vendor’s own harness number is more relevant, because that harness is what you will actually run. Astra inside OpenAI’s tooling really can behave like the 99.9% run.
When building your own agent: the harness is your highest-leverage surface. Before reaching for a bigger model, fix state carriage, compaction and tool descriptions. A mid-tier model in a good harness routinely beats a frontier model in a bad one — and costs a fraction as much.
When evaluating internally: hold the harness fixed and swap only the model. Otherwise you are measuring your own plumbing and attributing it to the vendor.
Last verified: September 6, 2026.