How We Built a Scalable Quality Metrics Dashboard for Software Teams

Background and Goal

Quality measurement collects product development and post‑release quality data, aggregates it, and presents it via charts and quality scores. The motivation was that quality data were scattered across systems, making it hard to view by dimensions such as department, application, and individual.

Product Design

Aggregation Dimensions

Data are aggregated by three dimensions: department, application, and person.

Evaluation Methods

Horizontal comparison with peers (e.g., top‑3 bug rates).

Vertical comparison over time (e.g., month‑by‑month bug rate trends).

Achievement rate against predefined targets (e.g., unit‑test coverage reaching 80%).

Composite quality score calculated from weighted sub‑dimension scores.

Improvement Suggestions

Providing actionable advice is complex because it depends on data quality, completeness, and relevance. Currently only key trend comparisons and vertical analysis prompts are offered; deeper recommendations require human analysis.

Detailed Design

Aggregation Details

Department : Bug data linked to the tester’s department; application‑related data linked via application‑department mapping.

Application : Some data (e.g., test bugs) lack an application tag, and forcing selection proved costly, so it was omitted.

Metric Items

Metric items are quantitative or percentage values such as effective bug count or primary bug rate.

Selection was based on data maturity, stakeholder interest, and implementation cost.

Presentation

Metrics are grouped into pages. Users first see a quality overview page showing the overall quality score and key indicator trends, then can drill down into specific metric group pages that provide horizontal and vertical details.

Technical Implementation

The system consists of an offline data‑processing layer and a data‑presentation layer, providing fast chart rendering and high flexibility for changing requirements.

Offline calculations are performed with Hive, allowing quick updates without full system releases.

Real‑time freshness is sacrificed because most quality data do not require instant updates; manual data refresh is available when needed.

Data Processing Flow

Data Import

Data that can be loaded directly into the data warehouse are imported via configuration.

API‑only data are synchronized to a database via scheduled jobs before being imported.

Detail Processing

This step prepares data for later aggregation, e.g., determining whether a bug was reopened by analyzing operation logs and marking it for downstream statistics.

Statistical Calculation

Granularity: Daily granularity is used; weekly/monthly aggregates are derived on‑demand.

Full vs. incremental computation: Full recomputation is used for volatile data (e.g., bug lifecycle), while incremental computation is applied to relatively static data such as release counts.

Challenges Encountered

Too many metric items caused excessive table joins; the solution was to merge tables with identical aggregation conditions and progressively join them.

Defining the effective‑bug timestamp (creation vs. resolution) required a consistent approach; both were acceptable, with clear labeling for users.

Conclusion

After three iterative releases, the quality‑measurement system was fully launched in September. Lessons learned include clarifying target users and scenarios during product design, improving proactive user outreach, and enhancing data‑presentation speed. Future work will integrate quality‑operation processes, refine metric items, improve data quality, and provide more concrete improvement suggestions to drive overall software quality.