What is the difference between OSWorld, Terminal-Bench and SWE-bench?

They test three different jobs. OSWorld measures GUI computer use — driving desktop applications through a perception-action loop. Terminal-Bench measures command-line agent work, with version 4.0 focusing on long-horizon tasks that run for extended periods. SWE-bench measures whether a model can produce a correct patch for a real GitHub issue, which is a code-editing task rather than an agent task. A model can lead one and trail badly on another.

Why does the same model get different scores on the same benchmark?

Because the harness differs. Google's launch table listed Claude Opus 5 at 75.4% on OSWorld-2.0 while the official leaderboard and independent trackers put it at 70.2-70.6%. OpenAI's 72.6% for GPT-6 Astra came from a latency simulation on the offline set. Scaffolding, retry policy, timeout budget and whether the run is online or offline all move results by several points, so only compare rows measured by the same evaluator.

Which agent benchmark should I actually trust?

The one whose task shape matches your workload, measured by a third party rather than the vendor. If your agent drives a GUI, OSWorld is relevant and SWE-bench is not. If it runs long shell workflows, Terminal-Bench 4.0 is the signal and Terminal-Bench 2.1 is nearly saturated. Better still, build a private evaluation from fifty to a hundred real tasks from your own backlog, because that is the only benchmark that cannot be trained against.

Is a high benchmark score worth paying more for?

Often not. On the Artificial Analysis coding-agent harness in September 2026, Claude Fable 5.1 led on quality at index 70.4 but cost $9.18 per task, while GPT-6 Astra scored 67.0 at $3.27 and Muse Spark 1.3 scored 64.2 at $1.72. Roughly six index points cost about five times more. Whether those points matter depends entirely on what your agent does when it fails, not on the leaderboard position.

Quick Answer

OSWorld vs Terminal-Bench vs SWE-bench: Which to Use

Published: September 5, 2026

The Short Answer

Vendors quote these three interchangeably. They measure entirely different things:

Benchmark	Tests	Relevant if your agent…
OSWorld 2.0	GUI computer use on a real desktop	Clicks, types and navigates applications
Terminal-Bench	Command-line agent work	Runs shell workflows and tooling
SWE-bench Verified	Patch correctness on real GitHub issues	Edits code to fix defined bugs

A model can lead one and trail badly on another. Picking a model on the wrong benchmark is the most common and most expensive evaluation mistake in 2026.

Last verified: September 5, 2026.

OSWorld 2.0 — GUI Computer Use

What it measures: whether an agent can complete real desktop tasks — navigating applications, moving files, filling forms — through a perception-action loop against an actual operating system.

September 2026 standings:

Model	Score
GPT-6 Astra	72.6%
Claude Opus 5	70.2–70.6%
Muse Spark 1.3	66.9%
GPT-5.6 Sol	65.7%
Gemini 3.8 Flash	59.0%

Use it when your agent drives a GUI. Ignore it for API-driven or code-editing workloads — GUI perception is a distinct capability and a strong SWE-bench score predicts nothing about it.

⚠️ The harness caveat is severe here. OpenAI’s 72.6% figure comes from a latency simulation on the offline set, reported at roughly 40 minutes per task against GPT-5.6 Sol’s 65.7% at roughly 75 minutes. Google’s Gemini 3.8 Flash launch table lists Claude Opus 5 at 75.4% on OSWorld-2.0 — five points above what the official leaderboard shows. Neither party is lying; they ran different setups. Never compare a number from one vendor’s blog against a number from another’s.

Terminal-Bench — Command-Line Agents

What it measures: agent competence in a shell — running tools, interpreting output, recovering from errors, chaining operations toward a goal.

Version matters more than the score. Terminal-Bench 2.1 is close to saturated at the top: Muse Spark 1.3 posts 88.8%, and Gemini 3.8 Flash sits around 89–91% against Gemini 3.7 Flash’s 81.6%. When a benchmark clusters near 90%, remaining differences are mostly noise.

Terminal-Bench 4.0 is the one that discriminates, because it targets long-horizon tasks:

Model	Terminal-Bench 4.0
Claude Fable 5.1	55.8%
Claude Opus 5	51.8%
Gemini 3.8 Flash	19.1%

That 32-point collapse for Gemini 3.8 Flash between a task suite it nearly aces (2.1) and one it fails (4.0) is the single most useful data point in this article. Duration is a capability, distinct from per-step competence, and cheap models fall off a cliff on it.

Use it when your agent runs shell workflows. Insist on version 4.0 if your tasks run longer than a few minutes.

SWE-bench Verified — Patch Correctness

What it measures: whether a model produces a patch that resolves a real GitHub issue and passes the repository’s tests.

This is not an agent benchmark. It is a code-editing benchmark. The task is bounded, the context is provided, and success is a passing test suite. Strong SWE-bench performance tells you a model can write a correct fix; it tells you very little about whether the model can find the problem, decide what to do, or recover when its first three attempts fail.

Notably, DeepSeek V4 Pro statistically ties Claude Opus 4.7 on SWE-bench Verified at roughly a thirty-fourth of the input price — which is a real result about patch quality, and not a claim that the two are interchangeable as agents.

Use it when you are choosing a model for bounded code edits. Do not use it to select an autonomous agent.

DeepSWE v1.1 — software engineering; Muse Spark 1.3 posts 75.4%, ahead of Opus 5 at 74.0 and GPT-5.6 Sol at 73.0
CursorBench 3.2.0 — IDE-context coding; Claude Fable 5.1 at 73.4%
ARC-AGI-3 — abstract reasoning. The canonical harness-discrepancy example: OpenAI’s announcement table lists GPT-6 Astra at 99.9%, while ARC Prize’s own official standard harness scores it at 62.7%. Same model, same benchmark name, 37-point spread. Opus 5 scores 30.2% on both.
MRCR — long-context retrieval; Muse Spark 1.3 at 98.5% / 98.1% where GPT-5.6 Sol drops to 73.8%

Why Scores Move — and What to Do About It

Four variables shift results by several points before the model is even involved:

Scaffolding. The agent framework wrapping the model does much of the work. A good harness on a weaker model routinely beats a poor harness on a stronger one.
Retry policy. How many attempts, and whether failures are retried, changes the number substantially.
Timeout budget. Directly determines long-horizon results, which is exactly what Terminal-Bench 4.0 probes.
Online vs offline sets. Offline snapshots are reproducible; live runs are not, and they are not comparable to each other.

The rule that follows: only compare rows produced by the same evaluator. Vendor tables are internally consistent and externally meaningless.

What to Actually Do

Benchmarks narrow the field. They do not pick the model.

The number that decides real projects is cost per completed task on your own work. From the Artificial Analysis coding-agent harness, September 2026:

Model · effort	Index	Cost per task
GPT-5.6 Luna (max)	57.1	$0.29
Muse Spark 1.3 (xhigh)	64.2	$1.72
GPT-6 Astra (xhigh)	67.0	$3.27
Claude Opus 5 (xhigh)	68.1	$8.17
Claude Fable 5.1 (max)	70.4	$9.18

Six index points cost roughly five times more. Whether that trade is correct depends on what your agent does when it fails — a wrong answer that a human catches in review is cheap; a wrong action against production is not.

The practical process:

Use benchmarks to eliminate models whose task shape is wrong for you
Build a private evaluation of 50–100 real tasks from your own backlog
Measure cost per completed task, not per token and not per benchmark point
Keep the model identifier behind a config flag so switching is a deploy, not a refactor

A private eval is the only benchmark that cannot be trained against, and it is the only one that reflects your actual token consumption.