Unreal Agent vs Codex vs Pi: Same Score, 39% Cheaper?
The short answer
Unreal Agent matched Codex’s Terminal-Bench 4.0 score with the same model for 39% less money, and it is open source. Unreal Labs released the harness on September 22, 2026 under MIT, with published Harbor runs for anyone to reproduce. Running GPT-6 Astra at xhigh effort, Unreal Agent hit 57.9% for $1,428 total, the Codex leaderboard entry hit 57.9% for $2,350, and Pi hit 55.0% for $1,827. The saving comes from architecture, not model choice: tool calls run asynchronously so the model never burns turns waiting, prompts are minimal, and there are no sub-agents. It is the clearest public demonstration yet that the harness, not the model, decides a large share of an agent’s bill.
Side by side
All runs use GPT-6 Astra at xhigh effort. Totals are dollars for the full benchmark; per-trial figures are input tokens / output tokens / model turns / tool calls.
| Benchmark | Unreal Agent | Codex | Pi |
|---|---|---|---|
| Terminal-Bench 4.0 | 57.9%, $1,428 (1.73M / 32k / 28 / 37) | 57.9%, $2,350 (leaderboard) | 55.0%, $1,827 (2.83M / 35k / 44 / 57) |
| DeepSWE 1.1 | 72.4%, $1,367 (1.60M / 28k / 26 / 38) | 69.0%, $1,633 (2.19M / 30k / 30 / 29) | 69.6%, $1,584 (2.21M / 30k / 40 / 75) |
| SWE-Atlas Codebase QnA | 65.8%, $936 (898k / 15k / 16 / 27) | 63.3%, $1,303 (1.69M / 17k / 22 / 21) | 64.0%, $1,033 (1.29M / 16k / 24 / 60) |
| Agents’ Last Exam (ALE-CLI) | 30.0% full pass, mean 59.7, $217 | 29.0%, mean 58.1, $292 | 29.0%, mean 59.2, $262 |
| License / language | MIT, Go | Open source CLI, TypeScript/Rust | Open source, TypeScript |
| Sub-agents / workflows | None by design | Yes | Yes |
| Interfaces | Go library, runner exe, Harbor benchmark runner | CLI, IDE, cloud sandbox, codex exec | CLI, SDK |
Unreal Labs says the pass-rate differences are within benchmark variance. The cost differences are not: input tokens per trial are the consistent gap, and they run 25–40% lower for Unreal Agent on every benchmark.
Why the harness is cheaper
Three mechanisms, all described in the September 22 write-up:
1. Asynchronous tool execution. When Unreal Agent issues a tool call, it immediately appends an “in-progress” event to the session log and continues. The model can launch a dev-environment setup that takes minutes, then explore the codebase and search the web in parallel, without any polling turns or heartbeat messages. When the tool finishes, its result is appended and the model is called again. Unreal Labs notes that doing this without breaking prompt caching was a real engineering problem; the payoff is that long tools cost zero tokens while they run.
2. More tool work per model turn. Because the async model is explained plainly to the LLM, it issues more heavy calls per turn. On Terminal-Bench, that is 37 tool calls across 28 turns for Unreal Agent versus 57 calls across 44 turns for Pi, meaning fewer round trips to the model, which is where the input-token bill accrues.
3. Minimal footprint. Simple prompts, token-optimized tool results, and no sub-agents or workflow orchestration. Sub-agents multiply context; Unreal Labs chose to leave them out.
The company also argues that security and approvals belong outside the harness, in deterministic sandbox constraints such as allowed hosts, granular tokens and approval-gating proxies, rather than in hook-based harness features it found brittle.
What this means against Codex and Claude Code
Codex and Claude Code are products, not just harnesses: they bundle hosted sandboxes, IDE integrations, subscription pricing and provider-tuned behaviour. If you pay a flat monthly plan, harness token efficiency is your provider’s problem, not yours. The comparison bites when you are on API pricing at scale, or building an agent-first product where the harness is your runtime.
Unreal Labs’ motivation section is a fair summary of that second case: CLI-oriented SDKs assume local sessions and subprocesses, provider switches break tools or compaction, and SDK upgrades change message formats. Their answer was a Go library you embed directly. That is the same direction as OpenAI’s Agents API and Codex harness and DeepSeek’s open-source harness, with a stronger cost claim and public Harbor job IDs behind it.
The context matters too. The HarnessTax study in mid-September 2026 found harness choice barely moves success rate but substantially changes token cost. Cognition reported an 18-step median against 48 for its previous Devin version. Rabbit shipped OS3 the same day Unreal Agent launched. The industry is converging on the view that the harness is where the remaining efficiency lives, now that frontier models are close on capability. Unreal Agent is the first open-source harness to put a reproducible dollar figure on it.
Who should use which
- Unreal Agent: production agents billed per token, embedded agent runtimes in Go, workloads with long-running tools (builds, test suites, environment setup), and teams that want users to steer mid-task. Verify the numbers on your own tasks with the Harbor runner first.
- Codex: OpenAI-native teams that want the hosted sandbox, cloud tasks, and ChatGPT plan billing; the reference harness for GPT-6 Astra.
- Pi: a mature, flexible harness with sub-agents and workflow support; accept a roughly 20% cost premium over Unreal Agent on these benchmarks for that flexibility.
- Claude Code: if your model is Claude Opus 5.5, whose $0.20 cache reads and lower token use per task already cut the bill, and whose classifier, sandbox and code review are built in.
The pricing behind all of this: GPT-6 Astra is $10/$50 per million tokens with $1.00 cached input as of September 2026, which is why a 1M-token reduction per trial shows up as hundreds of dollars per benchmark. On a cheaper model the absolute saving shrinks, but the percentage does not.
Last verified: September 24, 2026.