Can LLMs Revolutionize Code Review? Inside AutoDev’s AI‑Powered Approach

Motivation for AI‑Assisted Code Review

As code volume grows, the number of changes that need review increases, and AI‑generated code from tools such as GitHub Copilot or ChatGPT often bypasses human scrutiny. Reducing the manual effort of code review while preserving quality therefore becomes critical.

Commit‑to‑Requirement Linking

Effective review starts with well‑structured commit messages that reference a user story or requirement. Common conventions are: feat(devops): init first review command #8 In JIRA the issue ID is embedded, while on GitHub the pattern #<issue‑id> is used. When developers omit proper messages, an AIGC model can generate them automatically.

Why AIGC Cannot Replace Static Analysis

Traditional quality tools (SonarQube, JavaLint, ArchGuard, Feakin) provide deterministic analysis of code and architecture. AIGC output depends heavily on prompt quality and can drift with long inputs, making it less reliable for systematic checks.

Stakeholder Concerns in Code Review

Junior developers focus on coding standards and peer suggestions.

Mid‑level developers care about productivity and work‑life balance.

Senior developers evaluate architectural soundness and business alignment.

Technical leads assess impact on project timelines.

Consequently, routine checks should be delegated to static tools, while higher‑level judgment remains with humans.

Design of an AIGC‑Assisted Review System

The core difficulty is handling context and filtering irrelevant changes (e.g., documentation updates). The system follows a configuration‑driven prompt evolution:

Collect business context from the DevOps platform.

Filter out non‑essential files.

Exclude irrelevant sections within patches.

Generate concise, high‑quality review results.

Implementation Workflow (AutoDev + DevOpsGenius)

The same logic is applied on both IDE and platform sides. The six‑step pipeline is:

Retrieve a list of commit messages from the version‑control system.

Parse each message into a structured representation (type, scope, description, references).

Select commits that require review based on configurable criteria (e.g., exclude chore, ci, docs).

Extract related user‑story titles using identifiers found in the messages.

Generate the diff (patch) for the selected commits.

Send the diff together with business context to the LLM review engine and await the result.

Prompt Template

BusinessContext: ${context.businessContext}
CommitMessage: ${context.fullMessage}
CodeChanges: ${context.changes}

Providing the raw patch yields better results because most LLM training data is based on diffs.

Commit Filtering Example (Kotlin)

// Read configuration and filter out commits that do not need review
val filterCommits = parsedMsgs.filter {
    if (it.meta.containsKey("type")) {
        val type = it.meta["type"] as String
        project.commitLog?.isIgnoreType(type) ?: true
    } else {
        true
    }
}

This reduces prompt length by presenting only essential commits.

IDE vs. Platform Prompt Differences

In IDE scenarios reviewers often need to understand a series of commits that implement a feature, which can exceed token limits; therefore additional summarisation or chunking is required. On the platform side the prompt targets a single pull‑request or commit, allowing a more focused analysis while still applying the same patch‑optimisation tactics.

Current Limitations

Different teams adopt varied review styles and focus points. Integrating these diverse concerns into a unified prompt remains an open challenge.

Future Evolution

The community is invited to contribute to the open‑source repository:

https://github.com/unit-mesh/devops-genius

Further details, including configuration files and additional prompt templates, are available in the project documentation.