Do AI Coding Assistants Slow Down Experienced Developers? Surprising Study Results

Research Motivation

Benchmarking studies of AI‑assisted coding often use isolated, synthetic tasks that ignore real‑world context and human interaction. METR argued that in‑situ, empirical experiments with experienced developers are needed to assess how generative AI actually influences software‑development productivity.

Methodology

METR recruited 16 seasoned open‑source contributors (average five years of experience, each maintaining repositories with >22 k stars and millions of lines of code). The participants supplied 246 real‑world issues from their own projects, covering bug fixes, feature additions, and refactorings. Each issue was randomly assigned to one of two conditions:

AI‑allowed: developers could use any coding assistant (the majority used Cursor Pro with Claude 3.5/3.7 Sonnet).

AI‑blocked: developers worked without generative assistance.

Developers recorded their work sessions, reported total elapsed time for each task, and were compensated at $150 USD per hour.

Key Findings

When AI assistance was permitted, developers spent on average 19 % more time to complete tasks than when AI was prohibited, directly contradicting the common expectation of a ~20 % speed‑up. Despite the slowdown, participants still believed AI would make them 24 % faster and remained confident in a 20 % productivity gain.

Time‑budget analysis showed that AI users spent less time writing code and searching for information, but allocated more time to:

crafting prompts,

waiting for model outputs,

reviewing generated code, and

idle periods.

Factor Analysis

METR examined 20 experimental attributes (e.g., model version, prompt length, task difficulty, interruption frequency). Five factors were identified as likely contributors to the observed slowdown, while eight factors showed mixed or ambiguous effects.

Robustness Checks

The slowdown persisted across alternative outcome metrics (e.g., lines of code changed, number of commits), different estimation techniques, and subgroup analyses (e.g., by task type or developer experience).

Limitations and Future Outlook

The study’s scope is limited to a small, highly skilled sample and to specific AI tools (Cursor Pro + Claude 3.x). Results may not generalize to all software‑engineering contexts or to developers with lower AI experience. METR acknowledges that newer or more specialized models could yield different outcomes and calls for continued diversified evaluation methods to track AI’s evolving impact on productivity.

Reference

Paper title: Measuring the Impact of Early‑2025 AI on Experienced Open‑Source Developer Productivity

URL: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf

Code example

收
藏
，
分
享
、
在
看
，
给
个
三
连
击呗！