Cross-Contrastive Learning Cuts Flink Anomaly Detection Errors by 12%

01 Opening

Recently, the Alibaba Cloud big‑data engineering team together with the School of Data Science and Engineering at East China Normal University published the paper Noise Matters: Cross Contrastive Learning for Flink Anomaly Detection . The work was accepted at the VLDB 2025 conference and demonstrates a neural‑network‑based hotspot anomaly detector for Flink clusters that improves the F1 score by 12.1% compared with existing state‑of‑the‑art methods.

02 Background

Flink clusters frequently encounter hotspot problems: monitored jobs experience increasing latency and CPU usage remains high for extended periods. Detecting anomalous time‑series is required to locate the problematic machines. Existing unsupervised time‑series anomaly detection (UTAD) methods fail in this scenario for two main reasons. First, Flink‑specific anomalies such as slow‑rising trends or sustained high‑level spikes are not well captured by reconstruction‑based or correlation‑based approaches. Second, real‑world Flink logs contain substantial noise, violating the clean‑data assumption of many traditional methods.

03 Challenges

Limitation 1: Beyond point‑level anomalies, Flink workloads need detection of its unique patterns—e.g., multiple jobs on a node showing continuously rising delay or staying at a high level, which most SOTA detectors miss because the reconstruction error between normal and abnormal samples is small.

Limitation 2: The massive scale of time‑series collected from production Flink clusters introduces abundant noise and outliers. Traditional unsupervised detectors assume relatively clean training data and thus degrade when faced with noisy, label‑free datasets.

04 Breakthrough

Instead of computing reconstruction errors or view‑wise differences at each timestamp, we propose a new cross‑contrastive learning framework that explicitly focuses on Flink‑specific anomalies. An attention mechanism learns representations from both global and local perspectives, and cross‑contrastive learning encourages similar representations for adjacent normal timestamps while enlarging the distance for timestamps containing Flink‑specific anomalies such as slow‑rising trends.

We also introduce a novel loss function that injects prior knowledge via an “anomaly boundary” for each timestamp. Normal timestamps have small boundaries and their normalized scores stay close to the observation, so their loss is minimized. Abnormal timestamps receive larger boundaries, limiting how much their loss can be reduced and assigning them higher anomaly scores. This design makes the model robust to noisy training data and improves detection accuracy.

05 Application

The Noise Matters technique has been integrated into the Flink cluster intelligent inspection system, enabling operations teams to assess cluster health proactively and identify potential risk points before they impact production.