Can AI Write Perfect Unit Tests? Inside AutoDev’s Prompt‑Fine‑Tune Pipeline

Background

AutoDev is an open‑source IDE plugin that aims to provide a complete end‑to‑end AI‑assisted programming workflow. The project combines IDE‑side model fine‑tuning of open large language models, builds corresponding model and dataset assets, and creates a dedicated data‑engineering pipeline called Unit Eval for test generation.

Integrated “Write‑Eval” Pipeline

The core idea is an integrated “write‑evaluate” loop: AI tool → model fine‑tuning → model evaluation. This loop produces test code that matches the specific context of different organizations.

What Makes a Good AI Test Context?

A useful test context must contain class constructor information, input and output signatures of interfaces/functions, details about the test framework (e.g., JUnit 4 vs JUnit 5, mock framework), and coding conventions such as naming rules.

Typical problems when the context is missing include using the wrong JUnit version, selecting the wrong mock library, constructing incorrect objects, calling private methods directly, and violating naming conventions.

Prompt Engineering for Test Generation

To help the model understand the required code, a concise prompt template is used. The template injects the test framework, core framework, test specification, and related model information into the prompt.

Write unit test for following code.
${context.testFramework}
${context.coreFramework}
${context.testSpec}
${context.related_model}
```{context.language}
${context.selection}
```

Experiments showed that open models often struggle to interpret complex prompts, so a focused dataset around prompt context is required.

Dataset Construction and Model Fine‑Tuning

The team fine‑tuned the DeepSeek 6.7B model, chosen for its high‑throughput code completion and test‑generation capabilities. The dataset includes three layers of context:

Technical stack context

Test‑stack context

Code block input/output information

The dataset is built from the Unit Eval project and released at https://github.com/unit-mesh/unit-eval/releases/tag/v0.2.0.

Quality Control with ArchGuard Rules

Before adding generated tests to the dataset, ArchGuard rules scan for test‑code “bad smells” such as missing assertions, sleep calls, excessive debug prints, and overly many assert statements. Only tests that pass these quality checks are kept.

No assertions in test

Tests containing Thread.sleep Excessive debug output

Too many assert statements

Examples

A typical generated test case looks like:

@Test
public void testCreateBlog() {
    BlogPost blogDto = new BlogPost("title", "content", "author");
    when(blogRepository.save(blogDto)).thenReturn(blogDto);
    BlogPost blog = blogService.createBlog(blogDto);
    assertEquals("title", blog.getTitle());
    assertEquals("content", blog.getContent());
    assertEquals("author", blog.getAuthor());
}

The generated method names sometimes violate naming conventions (e.g., not following the should_return_…_when_… pattern), highlighting the need for further fine‑tuning.

Prompt Consistency

During the writing of this article, inconsistencies in prompts were discovered and corrected, such as ensuring test class names use snake_case and accurately reflecting the project's Spring Boot, JUnit, AssertJ, and Mockito stack.

Conclusion

Building an AI‑assisted coding tool like AutoDev requires continuous evolution of the architecture, prompt engineering, dataset quality, and model fine‑tuning to reliably generate usable unit tests that adhere to project‑specific conventions.