Feeling 20% Faster but Actually 19% Slower: Who Gains and Who Loses with AI Coding?

1. Counter‑intuitive RCT (July 2025)

METR ran a randomized controlled trial with 16 senior open‑source developers who each maintained a repository averaging five years old (≈22 k stars, >1 M lines of code). For 246 real tasks, each task was randomly assigned to either allow AI assistance (Cursor Pro + Claude 3.5/3.7 Sonnet) or prohibit it.

Result: Tasks with AI assistance took 19 % longer on average.

Perception bias:

Before the trial developers expected a 24 % speedup.

After being slowed by 19 % they still believed AI made them about 20 % faster.

Why the slowdown? Developers were modifying deep‑context code in repositories they knew intimately. AI missed many implicit constraints, so the time spent reviewing and fixing AI output exceeded the typing time saved. METR notes the conclusion does not extend to novices, unfamiliar codebases, or newer tools.

2. Follow‑up (February 2026)

METR reran the experiment with newer tools: 57 developers, >800 tasks, 143 repositories.

Subset of the original team: ~18 % faster.

Newly recruited developers: ~4 % faster.

Both confidence intervals crossed zero → statistically insignificant.

METR added a banner “These results are out of date.” The authors reported a selection effect—many participants refused to work without AI, and 30‑50 % omitted tasks because they did not want to hand‑code them—suggesting the true benefit could be larger.

3. Large‑scale Cursor usage study (Suproteem Sarkar, University of Chicago)

⚠️ Disclaimer: The study was conducted with Cursor’s cooperation; data come from the Cursor platform and may be biased.

Core findings (≈1 000‑person behavior sample, 24 organizations using Cursor vs. 8 comparable non‑Cursor firms):

When “Agent” became the default mode, PR merge rate rose 39 % relative to the control group.

Quality metrics (PR rollback rate, lines changed per PR, files changed per PR) remained stable.

Experience correlated with Agent usage: each additional standard deviation of experience increased Agent adoption by ~6 %.

Senior developers tended to plan first and then let the Agent write code; junior developers relied more on line‑by‑line tab completion.

Sarkar summarized: AI is shifting development from “syntactic coding” to “semantic orchestration and evaluation of agents.”

4. Conditions that determine speedup vs. slowdown

Task type

More likely to speed up: new code, scaffolding, migration, test addition.

More likely to slow down: deep‑context debugging, changes with many hidden constraints.

Code‑base familiarity

Speed up when working in unfamiliar repositories or languages.

Slow down when working in a massive legacy codebase the developer knows intimately.

Output verifiability

Speed up when tests and clear acceptance criteria are present.

Slow down for vague “looks‑good‑to‑me” work.

Usage pattern

Speed up when the developer plans the whole chunk and hands it to the Agent.

Slow down when using line‑by‑line completion and repeatedly fixing AI errors.

Rule of thumb: The harder implicit knowledge is to convey to AI, the less help you’ll get; the easier the output can be verified, the more help you’ll receive.

5. Action checklist derived from senior developers’ behavior

Plan before generating. Let the AI propose a solution, confirm it, then let it write code – cheaper than fixing generated code later.

Give the Agent whole tasks; use tab‑completion for minor edits. Data show senior developers who delegate whole chunks reap the +39 % PR merge benefit.

Make work verifiable. Add tests and clear acceptance criteria to maximize AI benefit.

Apply AI to unfamiliar code, languages, or APIs. AI helps least on code the developer already knows intimately.

Trust objective metrics. Measure merge rate, rollback rate, and cycle time instead of relying on subjective feeling of speed.

6. References

METR original study (2025‑07): https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
METR follow‑up (2026‑02): https://metr.org/blog/2026-02-24-uplift-update/
Sarkar paper (SSRN): https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5713646
Cursor official blog: https://cursor.com/blog/productivity