How Developers Supervise AI: Insights from 10k Real Coding Conversations

Researchers from the University of Notre Dame and Vanderbilt University examined a large corpus of real‑world programming conversations to characterize how developers interact with AI coding assistants.

Dataset and Methodology

The team employed the open‑source tool SpecStory to automatically export chat histories from public GitHub repositories. After removing conversations that were generated entirely by automated CLI agents, the final dataset comprised 11,579 complete chat sessions drawn from 1,300 repositories and contributed by 899 developers . The corpus contains 74,998 developer‑issued messages , with language distribution of 59.7 % English, 18.5 % Chinese, 8.3 % Japanese, and the remainder in other languages.

To interpret developer intent, the authors applied a qualitative “causal coding” approach in four iterative rounds, producing a detailed behavior‑intent taxonomy. This taxonomy was then used to label every message via a large language model (LLM) for high‑precision classification.

Key Findings

Progressive Specification (5.86 % new‑feature requests, 34.53 % code writing, 24.84 % iterative refinement) – Developers rarely begin with a complete specification. Most interactions involve drafting code, then repeatedly refining AI‑generated drafts.

Fault Diagnosis Outsourced (24 % of messages) – When errors occur, developers frequently paste raw error logs (8.84 %) and rely on the assistant to diagnose and fix the problem.

Black‑Box Queries (8.19 %) – Instead of reading source files, developers ask the assistant to explain the behavior of unfamiliar code.

Validation Delegation (3.99 % total) – Static code review accounts for 2.74 % and dynamic runtime checks for 1.26 %. Developers also invoke compilation scripts in 10.5 % of messages to verify generated code.

External Documentation (6.85 %) – Developers ask the assistant to generate Markdown task lists and other documentation to serve as a memory aid.

Context Injection (14.08 %) – Commands that inject real‑time facts or restrict the assistant’s actions are used to manage the model’s context.

Conversation Length Distribution – Median session length is three messages; however, a long‑tail of sessions exceeds 150 turns, focusing on deep code iteration and error fixing.

Repeated Conversation Archetypes – Six dominant patterns were identified: planning & consulting (15.77 %), fault‑driven debugging (19.90 %), iterative optimization (23.81 %), continuation‑driven delegation (9.46 %), extended co‑creation (18.42 %), and tool‑chain operations (12.64 %).

Self‑Reinforcing Dialogue – Consecutive messages with similar intents occur with high probability, creating feedback loops that keep developers and assistants locked in debugging cycles.

Cross‑Session Continuity – When a conversation becomes too long, developers start a new session but preserve the core intent, demonstrating a workflow for managing context overload.

Implications

The study reveals a shift from traditional upfront software design toward an incremental, AI‑augmented development process. Developers act as high‑level supervisors, articulating intent, managing risk, and documenting progress, while the AI handles most of the code generation, debugging, and validation tasks. Mastery of programming syntax becomes less critical than the ability to express precise intent and reason about architecture.

Reference: https://arxiv.org/pdf/2604.00436v1