Boost Data Reliability: Automated Testing for Big Data Metrics

Background

The chart shows that last year 33% of online defects in the data center were caused by errors in big‑data scheduling tasks, leading to abnormal data metrics. Causes include upstream data anomalies, Flink updates that lost data in ClickHouse, and system‑level task failures. Visualizing data and establishing alert mechanisms are essential for stabilizing big‑data indicators, motivating the research on an automated data‑metric testing system.

Feasibility Analysis

The goal of automation is to achieve the greatest certainty about software quality with minimal cost. Before automation, checking data metrics required manual login, navigation, and inspection. After automation, interface test scripts can perform these steps, eliminating repetitive manual effort.

ROI can be expressed as (n·t) / (d + m), where t is the time of a manual test, n the number of automated runs, d the development time of the script, and m its maintenance time. Because core metrics change rarely, n is large, and optimizing the framework reduces d and m, yielding an ROI far greater than 1.

Design and Implementation

1. Interface Automation for Periodic Metric Retrieval

HTTP interfaces are captured using Charles to export HAR files containing all request parameters. A scheduling module triggers the automation, parses responses to extract metric values, and stores them in a MySQL table.

2. Visualization

Metrics are queried from the database, aggregated by module, and rendered as visual dashboards using PyEcharts on the front end, with Flask providing the back‑end API.

3. Data Monitoring and Alerting

A corporate‑WeChat robot evaluates the metrics against timeliness, accuracy, and completeness rules. When thresholds are exceeded, the robot sends alerts to responsible product, development, and testing staff, and the visual interface assists in pinpointing issues.

Improving ROI

By introducing a custom ORM layer that maps YAML‑defined objects to database tables and visualization components, adding a new automated test case only requires updating the YAML file, dramatically reducing development effort.

Landing Effects

Detected online issues fall into three categories:

Data issues: interface switches causing cached, stale data.

Code issues: incorrect data processing or parameter errors at month start.

Task issues: data sync delays or failed scheduling tasks.

Upstream issues: changes in upstream system fields affecting downstream data.

These findings demonstrate the practical benefits of the automated testing framework.