Can AI Boost Developers? Study Finds Benefits for Experts, Drawbacks for Learners

A research project sponsored by an anthropology association examined how AI tools affect developers with different experience levels. Fifty‑two participants were split into two groups: junior software engineers and developers with over a year of Python experience. All participants performed two coding tasks using the Trio library ( https://trio.readthedocs.io/), which they had never used before.

One half of the participants received AI assistance during the tasks, while the other half worked without it. After completing the tasks, everyone took a 14‑question assessment covering debugging, code reading, code writing, and understanding of the Trio library. No AI assistance was allowed during the test.

Pre‑study pilot trials identified potential compliance issues (e.g., participants secretly using AI) and unrelated difficulties such as Python syntax problems. The main findings were twofold:

AI assistance did not produce a statistically significant increase in overall task completion time, contradicting earlier studies that reported up to an 80% reduction in time for skilled users.

Participants who used AI scored on average 17% lower (about two grade points) on the post‑task test, with the largest gap observed in debugging questions.

The researchers argue that while AI agents with higher autonomy might eventually improve productivity, relying on AI for code generation can suppress human skill development, making it harder for developers to verify and debug AI‑generated code later.

“If human abilities weaken, supervising increasingly powerful AI systems becomes ever more difficult.”

Anthropic, the creator of the Claude Code AI coding assistant, praised the study for reporting negative results and noted that the sample size was small, leaving many questions unanswered. The company previously claimed up to an 80% reduction in task time for skilled users, attributing the discrepancy to participants’ prior expertise.

Overall, the study suggests that widespread AI‑assisted coding could hinder the development of essential debugging and verification skills, especially among less experienced developers.