Why AI‑Assisted Development Still Falls Short: Lessons from a Year of Experiments

1. Short‑term expectations are unrealistic

Commercial vendors often claim 10%‑30% efficiency boosts from AI‑assisted development, but independent measurements show only 2%‑13% improvement, typically under 5% for most projects, due to legacy knowledge gaps, rigid processes, and immature tools.

Nevertheless, AI can reduce repetitive tasks, lower knowledge‑search costs, improve developer happiness, and accelerate learning of new technologies, especially when paired with a 70B+ model and suitable plugins.

2. Deterministic inputs reduce hallucinations

When designing an AI‑assisted unit‑test generation solution for a bank, the team combined remote test‑case agents with AutoDev’s precise testing capability. Deterministic generation based on known test frameworks, function signatures, and input/output specifications markedly lowered hallucinations compared with generic chat models.

Key deterministic inputs include:

Tool context such as testing frameworks, development stacks, and programming languages.

Business knowledge from requirement and design documents.

Domain knowledge specific to industries like finance or e‑commerce.

Existing normative artifacts such as previously written test cases and data.

3. Manual context injection remains essential

Tools like Cursor, GitHub Copilot, and Continue allow users to add explicit context (e.g., files or annotations) to improve determinism and avoid hallucinations. Effective manual injection methods include:

Embedding context via comments or special markers, e.g., $selection or language.

Interactive UI feedback loops that let users select relevant files or add extra inputs.

Template‑based workflows for commit messages, tests, etc.

Custom prompt engineering to inject team‑specific context.

These approaches incur learning overhead but are crucial for a good user experience.

4. Building a unified domain language is vital for complex AI assistance

The team introduced two VSCode chat strategies: one inspired by Bloop’s hypothesis‑driven retrieval and another mirroring Copilot’s keyword extraction. Consistent naming and shared terminology are required for AI to retrieve relevant knowledge effectively.

Applying Domain‑Driven Design (DDD) principles—defining domain models, creating a common language, and establishing bounded contexts—helps convey clear boundaries to AI tools and prevents context drift.

5. Knowledge retrieval can follow many paths

AI‑assisted tools fall into two categories: those that augment existing development environments (e.g., SourceGraph, JetBrains) and those that are standalone AI‑first products (e.g., Cursor). Retrieval strategies differ:

Structured knowledge extraction (e.g., converting PDFs/Word to markdown, then cleaning with AI).

Leveraging existing code‑search engines like SourceGraph Codey.

Choosing appropriate vector models: local models for simple code search, cloud models for cross‑team queries.

6. Agent planning should stem from pattern decomposition

Experiments with AutoCRUD combined AI with classic CRUD patterns, showing that agents need to understand layered architectures (Controller‑Service‑Repository) and adapt prompts to language‑specific patterns. Clear task descriptions and IDE integration are essential for reliable planning.

7. Knowledge reconstruction for AI is inevitable

Tools such as Unblocked aggregate knowledge from GitHub, Slack, Confluence, and Linear, exposing it via APIs so AI agents like AutoDev can consume up‑to‑date internal knowledge bases.

8. Interaction optimizations cannot overcome model limits

Current models have reached a performance plateau; relying solely on interaction (e.g., patch‑style code generation like StreamDiff) cannot solve inherent uncertainties, especially for non‑generic problems, numerical accuracy, or import‑related errors.

9. Master software engineering before AI assistance

AI cannot compensate for poor software‑engineering practices. Teams need solid requirement documentation, code comments, review processes, automated operations, and monitoring before AI tools can deliver meaningful productivity gains.

In summary, AI‑assisted development offers modest gains when combined with deterministic inputs, manual context injection, unified domain language, and robust knowledge retrieval, but it is not a substitute for disciplined engineering practices.