Why does AI coding break CI?

Because agents remove the two bottlenecks that used to throttle CI: writing code and reviewing it. Anthropic reported on September 14, 2026 that its engineers ship 8x as much code per quarter as in 2021-2025, Claude authors about 80% of it and helps approve PRs, its test count grew 10x, and CI jobs rose 25x in six months with roughly the same headcount. Agents also prefer smaller, more frequent PRs and push overnight and on weekends, so job volume rises and the floor never drops.

What is test impact analysis?

Test impact analysis (also called test selection) is a service that decides which tests to run on each change instead of running the whole suite. It typically has two parts: a listener that records the outcome of every test on every CI run, and a selector that uses that history plus package or dependency relevance to pick the tests that matter for an open PR. It cuts CI cost and time, and it gives agents a specific, trustworthy set of failures to iterate on instead of a wall of unrelated red.

How did Anthropic fix its test selection service?

After three patches that lasted 70 days, 29 days and under a day (a bigger machine, sharding the listener per package, and daily restarts), Anthropic rebuilt the service around an in-memory data store. Listener workers became stateless: any worker takes any result and appends it to a journal; a small consumer rolls the journal into per-test history every few seconds; the selector reads from that. It is horizontally scalable, easier to memory-profile, and the backlog that had grown week over week went flat. One engineer did it in three weeks.

How much CI capacity should an agentic team plan for?

Anthropic's advice is to assume your architecture will be at 25x today's load within two quarters, and to design v0 services for 10-20x perceived scale if budget allows. Instrument the pipeline so that jobs in equals jobs out, alert when the listener falls behind (Anthropic paged at 50,000 queued results), and expect a higher baseline plus bursts because agents work around the clock while humans still approve most merges.

Quick Answer

How to Scale CI for AI Coding Agents: Test Impact Analysis

Published: September 16, 2026

The short answer

Stop running every test on every PR, split the “record results” and “select tests” jobs into separately scalable services, and plan for 25x load within two quarters. That is the distilled lesson of Anthropic’s September 14, 2026 engineering post, in which Claude writes about 80% of the company’s code, engineers ship 8x more per quarter than in 2021-2025, test count grew 10x and CI jobs grew 25x in six months. Every team adopting coding agents in 2026 is on the same curve a few months behind.

Step 1: Measure the exponential you are on

Before touching architecture, get four numbers per week:

Metric	Why it matters
PRs opened per engineer	Agents shrink PRs and multiply them
CI jobs per day, plus the overnight/weekend floor	Agents raise the baseline; humans still drive bursts
Tests in the repo	Agents write tests too; Anthropic’s grew 10x
Listener lag (results recorded vs results produced)	The silent failure mode; 20 minutes of lag can mean tens of thousands of unapplied test updates

If PRs per engineer and tests are both growing, CI jobs grow as the product. Plot it on a log axis; if the line is straight, you have less time than you think.

Step 2: Adopt test impact analysis

Test impact analysis (test selection) runs only the tests relevant to a change. Anthropic’s version is deterministic: it uses past performance (which tests fail for which packages) and package relevance (what the change touches). Vendors sell this (Launchable, Datadog Intelligent Test Runner, Buildkite Test Engine, Gradle Develocity Predictive Test Selection); Bazel-style build graphs give a coarse version for free.

Whether you build or buy, it has two components that must stay in sync:

Listener — records every test result from every CI run.
Selector — reads the history and decides which tests run on each open PR.

The payoff is not only cost. Agents are worse than humans at ignoring irrelevant failures; a human shrugs off an unrelated flaky red, an agent starts investigating it. A selected, valid test set lets the agent self-verify and iterate.

Step 3: Know the failure sequence in advance

Anthropic’s v0 ran the listener as a single process because per-test history needed one writer. Here is how it died, so you can skip the stages:

Patch	What they did	How long it held
1. Bigger machine	Doubled cores (October 2025)	~70 days
2. Sharding	One writer per package instead of one global writer; Claude generated the sharding code (February 2026)	29 days
3. Daily restarts	Process hit memory limit by mid-afternoon (March 2026); allocator swap did nothing; found only four bugs	Less than a day, and the service fell further behind after each restart

When the listener fell more than an hour behind, results went unrecorded and the selector ran on stale data: flaky and universally failing tests kept running on everyone’s PRs. Untested code did not ship, but CI wasted capacity exactly when it had none to spare.

Step 4: Rebuild around a journal, not a singleton

The redesign that held:

Give the service a data store. Anthropic used an in-memory store. Each listener worker takes any result, appends it to a journal, and moves on without holding state. Workers are stateless, so they scale horizontally.
Roll up asynchronously. A small consumer process folds the journal into per-test history every few seconds.
Selector reads the rollup. Fast lookups, no coupling to ingestion rate.

Cost went up (more processes, a store), but the backlog chart went from “grows week over week” to flat, and memory profiling became possible because no single process held everything. Claude did the fine-tuning of journal size and worker count largely autonomously. Total effort: one engineer, three weeks, versus roughly a quarter a year earlier.

Step 5: Instrument for agents, not just dashboards

Anthropic ran a months-long session in an internal Claude agent that watched the service and pinged the owner whenever listener lag exceeded 50,000 jobs, then resumed the conversation with full context. Two practical rules follow:

Expose one invariant: jobs in equals jobs out. Alert on the gap, not on CPU.
Give the agent eyes: metrics and logs it can query, so it can hill-climb fixes incrementally instead of you re-explaining the system each time.

Step 6: Plan capacity for 25x

Anthropic’s explicit recommendation: assume your architecture will be at 25x load within two quarters, and design v0 for 10-20x perceived scale if budget allows. The old warning against over-engineering weakens when a redesign takes three weeks instead of a quarter. Half-measures now buy weeks, not years; each of the three patches above bought a fraction of what it would have a year earlier.

Also budget for shape changes:

Agents prefer small, granular PRs, so more CI jobs per unit of work.
The activity floor rises (agents push at 3 a.m.) but bursts remain (humans approve in daytime).
Test-authoring by agents is a cost center; Anthropic’s 10x test growth was its own load multiplier.

Checklist

Weekly plot of PRs/engineer, CI jobs/day, tests, listener lag
Test impact analysis in place (bought or built)
Listener and selector deployable and scalable independently
Stateless ingestion writing to a journal; async rollup
Alert on results-in vs results-recorded gap
Capacity model assumes 25x in two quarters
Flaky-test quarantine so agents are not chasing noise