How to Perform Quality Control for Temperature Data

Current Situation and Challenges

Temperature is the primary information displayed by any weather app, and occasional abnormal values can cause serious user dissatisfaction and economic loss. The production and transmission of meteorological data involve a long pipeline, so detecting and intercepting anomalous values quickly is essential. The industry currently relies on a single‑threshold filter, e.g., assuming Chinese temperatures lie between -55°C and 50°C and discarding values outside this range. While this works for variables with hard limits (e.g., humidity), it fails for temperature because many out‑of‑range anomalies remain within the bounds (e.g., -25°C in Haikou in August or 45°C in Mohe in December). Narrowing the threshold would exclude legitimate extreme weather, and widening it (as in the national standard QX/T 118‑2020, -80°C ~ 60°C) makes the filter ineffective. Moreover, temperature varies strongly with season and geography, so a single static threshold cannot capture the true acceptable range.

Solution Idea

To address the spatial‑temporal variability, the authors prioritize a statistical approach. Machine learning is considered but deemed costly; instead, a probability distribution function (PDF) based on historical data is used. By aggregating all hourly temperatures for a fixed location over a year, the distribution appears bimodal due to unequal lengths of hot and cold seasons. Resampling techniques remove seasonal bias, allowing the data to approximate a normal distribution. The mean and standard deviation of this normal distribution are then used to assess whether a new observation lies within a configurable tolerance (e.g., within ± k σ).

Data Processing

Historical Dataset and Pre‑processing

More than 2,000 national weather stations (including 35 in Taiwan) provided hourly temperature records from 2010 to 2020. ERA5 reanalysis data were selected for their hourly resolution, but a validation showed noticeable gaps between ERA5 and ground observations. Therefore, a fusion correction was applied: Gaussian radial interpolation maps station temperatures onto the ERA5 grid; station weights (initially 1) are also interpolated, decreasing with distance. The weighted average of the interpolated station data and ERA5 yields a higher‑quality gridded historical dataset for subsequent sampling.

Statistical Sampling Scheme

Because a single year provides only ten samples per hour, the authors employ a two‑layer cross‑period sampling strategy. For a target hour, they collect the hour itself and the two adjacent hours (hour‑level group). Then, for the three days before and after the target day, they gather the same hour‑level groups (day‑level group). Repeating this across all ten years produces 21 samples per day‑level group, i.e., 210 samples for the target hour. Spatial smoothing adds the eight neighboring grid points, expanding the sample size to 1,890 per point, which is sufficient for fitting a meaningful distribution.

Compute Statistical Parameters

Using the sampled data, the mean and standard deviation are calculated for each of the 8,760 hourly slots in a typical year, resulting in a grid of mean and σ values that represent the expected temperature distribution.

Temperature Quality‑Control System

The system is exposed as a language‑agnostic RESTful API with four design goals:

Language‑independent access via HTTP.

Fine‑grained validation by latitude/longitude and timestamp, returning a clear True/False verdict.

Adjustable tolerance parameter to control strictness.

Provision of reference mean, minimum, and maximum values for user insight.

Example request for Beijing (lon = 116.400244, lat = 39.912943) at 08:00 UTC on 2023‑10‑10:

curl "http://localhost:8000/v1/rationality/latlon/tmp/?lon=116.400244&lat=39.912943&value=26&timestamp=1696896000"

Response (rejected):

{
    "accepted": false,
    "varname": "tmp",
    "longitude": 116.400244,
    "latitude": 39.912943,
    "datetime": "2023-10-10T00:00:00+00:00",
    "input_value": 26.0,
    "tolerance": 0.9999,
    "max": 24.62,
    "min": -0.2,
    "mean": 12.21
}

The same temperature at 15:00 yields an accepted result with a broader acceptable range (6.45 °C ~ 31.79 °C, mean = 19.12 °C). Seasonal checks show 26 °C is accepted on 2023‑08‑10 08:00 but rejected on 2023‑12‑10 08:00, matching expectations. Extreme‑location tests (Mohe and Haikou) confirm that the system correctly distinguishes plausible values for each region and flags mismatched values when swapped.

The tolerance parameter tol can be increased (but must stay < 1) to widen the acceptable interval, allowing the system to adapt to different application requirements.

Limitations and Future Outlook

While the hourly‑level, location‑aware QC meets most business needs, the spatial resolution limits accuracy in mountainous terrain, leading to occasional misclassifications. Future improvements include incorporating denser ground observations and adding altitude‑aware temperature models.