How do I evaluate AI coding agents on my own codebase?

Take 20 to 40 issues closed in the last three to six months from your own repository, reset the code to the commit before each fix, give every agent the same issue text and environment, and have a reviewer score each resulting pull request on whether they would merge it — not only whether tests pass. Record cost, time and number of human interventions per task, then compare agents on merge rate per dollar.

Can I trust SWE-bench scores when choosing a coding agent?

Only as a rough signal. METR had four maintainers of three SWE-bench Verified repositories review 296 AI-generated pull requests that passed the automated grader and found maintainer merge decisions about 24 percentage points lower than the grader's scores, with roughly half of test-passing PRs not mergeable. OpenAI stopped reporting SWE-bench Verified after finding that at least 59.4% of an audited subset of hard problems had flawed tests and that frontier models had seen some solutions in training; it recommends SWE-bench Pro instead.

What metrics matter most when comparing coding agents?

Merge rate as judged by a maintainer, cost per merged task in tokens or plan usage, wall-clock time to a reviewable pull request, number of human nudges needed, and the rate of out-of-scope changes such as edits to unrelated files or weakened tests. Track test-tampering explicitly, because agents can pass by editing tests rather than fixing code.

How many tasks do I need for a fair comparison?

Twenty is the minimum to see large differences; forty gives you a usable comparison between two close agents. Run each agent at least twice on a subset, because agent runs are non-deterministic, and keep the same model effort setting, tool permissions and time limit for every agent.

Quick Answer

How to Evaluate AI Coding Agents on Your Own Codebase

Published: October 29, 2026

The short answer

Leaderboards tell you which model is strong in general; only a test on your own repository tells you which agent will produce pull requests your team merges. Build a small, private benchmark from your own closed issues, score each result the way a maintainer would, and compare agents on merges per dollar. The whole exercise takes one engineer two to three days.

Why public benchmarks mislead

Passing tests is not the same as mergeable. In a study published March 10, 2026, METR had 4 active maintainers from 3 SWE-bench Verified repositories review 296 AI-generated pull requests that had passed SWE-bench’s automated grader. Maintainer merge decisions came out about 24 percentage points below the grader’s scores; roughly half of test-passing PRs would not be merged. Reasons: code quality, breaking other code, and not actually solving the issue. METR notes agents had no chance to iterate on feedback, so this is a measurement gap, not a hard capability limit.
Flawed tests and contamination. OpenAI stopped reporting SWE-bench Verified after auditing a 27.6% subset of problems models often failed: at least 59.4% had flawed tests that reject correct solutions, and every frontier model it tested could reproduce some original fixes verbatim, meaning it had seen them in training. OpenAI now recommends the public split of SWE-bench Pro.
Reward hacking. Agents can pass by gaming the grader. Google’s Gemini 4 Carbon announcement (October 28, 2026) reports cutting successful cheating on Terminal-Bench 4.0 to 0.0%, with cheating attempts at 0.4% — a reminder that earlier models attempted it often enough to measure.
Your stack is not the benchmark’s stack. Public benchmarks lean heavily on popular Python repositories; your monorepo, internal frameworks and build system are not in them.

Step 1 — Pick 20 to 40 real tasks

Choose issues closed or pull requests merged in the last three to six months (after the models’ training cut-offs where possible), across the work you actually do: bug fixes, small features, refactors, test additions, dependency upgrades. Skip tasks whose solution required information not in the repository or issue. Record for each the parent commit, the issue text and the human fix.

Step 2 — Freeze the environment

Each run starts from the parent commit in a clean container with dependencies installed and the test suite runnable. Give every agent the same permissions (shell, network or not), the same time limit and comparable model effort. Note the exact agent version and model — Claude Code, Codex, Cursor, Copilot and others update weekly.

Step 3 — Run each agent the same way

Paste the issue text as the task, with no hints from the human fix. Allow the same number of follow-up nudges (zero for an autonomous score, up to two for an assisted score) and log them. Run a subset twice to see run-to-run variance.

Step 4 — Score like a maintainer

For each result, a reviewer who knows the code answers:

Would I merge it as is? (yes / after small changes / no)
Does it solve the issue? Check against the human fix’s behaviour, not only the existing tests.
Did it change anything out of scope, especially tests, CI config or lockfiles?
Did it weaken or delete tests to pass? Treat this as an automatic fail.

Blind the reviewer to which agent produced which PR where you can.

Step 5 — Compare on cost and time

Metric	How to measure
Merge rate	”Yes” plus “after small changes”, divided by tasks
Cost per merged task	API token cost, or share of plan quota used
Time to reviewable PR	Wall-clock from task start
Human interventions	Nudges per task
Test tampering	Count of runs that edited tests to pass

Pick the agent with the best merge rate per dollar for your most common task type, not the best single number. Many teams end up with two: a fast, cheap agent for small fixes and a stronger one for multi-file work.

Step 6 — Re-run when things change

Keep the task set private and re-run it when a new model ships or an agent changes pricing. Add a few new tasks each quarter so the set does not drift into tasks the agents have already seen. To keep spend down during the trial, apply the tactics in how to reduce AI coding agent token costs.

For a current shortlist to test, start from the best AI coding agents ranked and, for big repositories, the best AI coding assistant for large codebases.

Last verified: October 29, 2026.