How to Measure and Improve Front‑End Code Review Quality: Metrics, Insights, and Best Practices

DEF (Alibaba Front‑End R&D Platform) built a CodeReview system on Aone and KAITIAN, aiming to provide a better code review experience for front‑end developers and establish a quality evaluation framework. The “CR Quality Thinking” series consists of three parts: metrics and status, solutions, and phased review practice.

CR Quality Thinking: Metrics and Status

CR Quality Thinking: Solutions

CR Quality Thinking: Phased Review Practice

This article introduces the history of code review, quality evaluation indicators, and analyzes code review quality data using CodeCollaborator reports, comparing with DEF’s current data to understand the present state of front‑end code review quality.

Background

CodeReview (CR) is a widely used step in software development that aims to:

Knowledge transfer: reviewers and authors share knowledge during the review.

Code style consistency.

Architectural and maintainability checks.

Problem detection: uncover logical bugs and defects.

Traditional CR was conducted offline in meetings, which is heavyweight and time‑consuming. Modern CR is tool‑enabled and asynchronous, making the process lightweight. The typical online CR workflow includes:

Create a review after code development.

Pre‑review: developers preview changes and run static scans.

Reviewer comments on the changes.

Developers address comments.

Review approval.

Tool‑enabled CR solves many traditional problems, but formalization introduces new challenges such as large change sizes, tight release windows, and limited tool customizability, causing reviewers to skim rather than deeply evaluate.

CR Quality Metrics

To build a measurement system, we first look at raw data that can be collected during a review:

Change lines (LOC): total lines changed, including documentation and whitespace. Comments often reveal defects related to documentation mismatches.

Although we agree that sLOC (source code ignoring whitespace) is better correlated with “executable code” than is LOC, in our experience the comments often have as much to do with the review as the source code. Many defects are found in document/code mismatch or just in lack of proper documentation. Therefore we believe LOC is a better measure of the amount of work necessary for the reviewer.

Inspection Time: total time the reviewer spends reading the change.

Defect Count: number of comments marked as defects (including static analysis findings).

From these raw metrics we derive:

Inspection Rate = LOC / Inspection Time (review speed).

Defect Rate = Defect Count / Inspection Time (defect discovery speed).

Defect Density = Defect Count / LOC (defects per KLOC).

CodeCollaborator Analysis Report

In 2006 SmartBear Software conducted a 10‑month study using CodeCollaborator, covering 2,500 reviews, 3.2 million lines of code, and 50 developers—the largest lightweight review study to date. The workflow for a review in CodeCollaborator is:

Developer submits changes via CLI or GUI.

Developer selects reviewers, who receive email notifications.

Reviewers open the CR web page (often a single reviewer).

Reviewers view diffs and comment on problematic lines.

Comments can be marked as defects; developers submit fixes until no new defects remain.

Reviewer approves and code is merged.

Data Preparation

CodeCollaborator’s data distribution (left) shows change size vs. review speed on a log‑log plot. After removing outliers (review speed >1500 LOC/h, review time <30 s, or change size >2000 LOC, about 21% of data), the cleaned data (right) reveals most reviews involve ≤200 lines and speeds ≤500 LOC/h.

After obtaining reasonable review time and line count data, we collected the key quality metric: defect count. CodeCollaborator allows marking comments as defects and assigning severity. Because many developers only comment without marking defects, we also manually labeled 300 sample reviews to obtain a more realistic defect dataset.

CR Effectiveness – Defect Density Analysis

Assuming a constant theoretical defect density, the observed defect density reflects review quality. CodeCollaborator analyzed the relationship between defect density and change size, review speed, and pre‑review activity, as shown below:

Key conclusions:

Reviews should stay under 200 lines (max 400) to keep reviewers in control and discover defects.

Optimal inspection speed is around 300 LOC/h; speeds >500 LOC/h lead to many missed defects.

Authors who pre‑prepare with comments and documentation dramatically reduce defect density, often to zero.

CR Efficiency – Defect Discovery Speed Analysis

Defect density correlates with change size and speed, but defect discovery speed is largely constant (~15 defects per hour) across most reviews, except for a small subset of very small changes (≈6%) that achieve higher rates.

DEF Data Status

DEF’s CR records detailed metrics (LOC, inspection time, comment count, etc.). As of 2021‑06‑30, the overall review quality has significant improvement space:

Nearly 20% of reviews exceed the 400‑line threshold; ~10% exceed 2,000 lines.

Review time is short, causing speeds above the 500 LOC/h quality threshold.

Feedback is limited; overall comment rate is low.

For comparison, Google’s CR data shows:

Median change size lies between typical open‑source projects (11–32 lines) and internal projects (often <263 lines), closer to open‑source.

20% of reviews involve multiple defect cycles.

[1] The median change size in OSS projects varies from 11 to 32 lines changed, depending on the project. At companies, this change size is typically larger, sometimes as high as 263 lines. We find that change size at Google more closely matches OSS: most changes are small. [2] We attempt to quantify how “lightweight” reviews are (CP1). We measure how much back‑and‑forth there is in a review, by examining how many times a change author mails out a set of comments that resolve previously unresolved comments. We make the assumption that one iteration corresponds to one instance of the author resolving some comments; zero iterations means that the author can commit immediately. We find that over 80% of all changes involve at most one iteration of resolving comments.

Conclusion

DEF’s current CR quality is average. The existing process often forces reviews to occur just before release, compressing reviewer time. Single‑change reviews and lack of multi‑change integration lead to large review sizes. There is substantial room to improve review speed and overall quality, making the next phase of optimization a valuable challenge.

References:

Best Kept Secrets of Peer Code Review – https://static1.smartbear.co/smartbear/media/pdfs/best-kept-secrets-of-peer-code-review_redirected.pdf

Modern Code Review: A Case Study at Google – https://research.google/pubs/pub47025/