Anthropic's CI Crisis: Scaling Test Impact Analysis After AI Wrote 80% of Code

Background: AI-Generated Code Overwhelms CI

Anthropic's Claude now writes 80% of their code. Engineers deliver 8x more code per quarter compared to 2021-2025. Writing code is no longer the bottleneck; PR review accelerated, shifting pressure to CI. Test volume grew 10x, and CI job count increased 25x in six months.

The Test Impact Analysis (TIA) Service and Its Failure

The TIA service decides which tests to run for each change. It has two components:

listener : records the result of every CI run.

selector : uses historical results to decide which tests an open PR should run.

This design worked under normal load, but when multiple CI jobs ran per second, the listener fell behind. A 20-minute listener delay meant tens of thousands of test updates were not applied to the selector, leading to bad merges, dependency flakiness, and new tests not being run.

The service was originally a single-process system. To preserve history, a single writer had to apply results sequentially, making horizontal sharding impossible.

Three Failed Patches

First Patch: Bigger Machine (October 2025)

Doubling CPU cores bought 70 days of stability.

Second Patch: Sharding by Package

The listener doesn't need a single writer to order all tests globally; it only needs per-package ordering. Claude generated code to split each package's state into independent shards. This lasted 29 days.

Third Patch: Daily Restarts

By March, the process hit memory limits most weekday afternoons. Bug hunting and switching memory allocators failed. Restarts only kept the service alive for less than a day, and daily restarts caused the service to fall further behind; once lag exceeded an hour, large numbers of task results went unrecorded.

Redesign: Distributed Journal Architecture

The team adopted Claude's suggestion: add an in-memory data store. Listener workers become stateless, appending results to a journal and immediately releasing memory. A separate, small consumer process merges the journal into per-test histories every few seconds. The selector can then query quickly.

This distributed architecture costs more to run but is far easier to scale and analyze for memory issues. One engineer completed the project in three weeks; a year earlier it would have taken nearly a quarter.

Queue backlog shifted from daily accumulation to a stable, flat line.

Lessons Learned

The author's key takeaways:

Always plan for exponential growth. CI job count scales exponentially with the number of agents and PR approval speed.

Advice for engineering teams: whether building or buying, assume load will reach 25x within two quarters.

Design v0 with 10-20x headroom if budget allows.

Avoid running critical services on a single instance.

Keep state separate from the process.

Original article: Agentic coding is straining CI (https://claude.com/blog/agentic-coding-is-straining-ci-heres-how-we-scaled-test-impact-analysis-at-anthropic)