Detect Concept Drift Without Prior: Simple Supervised Method Using LSTM or LightGBM

Problem Statement

When outsourcing image‑classification labeling, the most informative samples must be selected. This requires both class diversity and representativeness. Detecting new or anomalous samples—known as concept drift—without assuming a specific data distribution is a key technical challenge.

Supervised Prior‑Free Drift Detection

The method treats drift detection as a binary classification problem that does not rely on any prior distribution assumptions.

Assign label 0 to all historical (already labeled) examples and label 1 to the batch of new examples that need to be inspected.

Concatenate the two datasets into a single DataFrame.

Train a binary classifier on the combined data. For sequential or image data an LSTM‑based model works well; for structured/tabular data LightGBM is recommended. Use 5‑fold cross‑validation to obtain robust probability estimates.

For each sample compute the predicted probability of belonging to the “new” class (label 1). Choose a confidence threshold (e.g., 0.90). Samples with probability above the threshold are considered high‑confidence drift candidates.

When LightGBM is used, extract the feature‑importance scores to identify which features contribute most to the drift.

This approach is fully supervised, requires no distributional priors, and works for both unstructured (LSTM) and structured (LightGBM) data.

Practical Example – PAKDD AutoML Competition

In the PAKDD AutoML competition the same pipeline was applied to detect drifted features. The workflow was:

Train the binary classifier and evaluate the validation AUC.

If AUC > 0.65, treat the identified high‑confidence samples as drift signals.

Refresh the affected features using a sliding time window and drop features that exhibit excessive volatility.

This resulted in a production‑ready drift‑monitoring component that improved model stability.

Code Implementation

import pandas as pd

# 1. Label the datasets
df_history['label'] = 0   # historical data
df_now['label'] = 1       # data to be inspected

# 2. Concatenate
df_all = pd.concat([df_history, df_now]).reset_index(drop=True)

# 3. Train a binary classifier with 5‑fold CV
#    `model` can be an LSTM wrapper or LightGBM; the `fit` method returns
#    a list of trained sub‑models and the predicted probability of class 1.
model_list, prob = model.fit(
    X=df_all['feature'],
    y=df_all['label'],
    kfold=True
)

# 4. Select high‑confidence drift samples
threshold = 0.90
df_drift = df_all[prob > threshold]

After understanding these steps, the entire pipeline can be reproduced without further code review.

Conclusion

The supervised, prior‑free drift detection method provides a simple yet effective way to identify distribution inconsistencies in both unstructured and structured datasets. By leveraging high‑confidence predictions and optional feature‑importance rankings, it can be integrated into continuous data‑quality monitoring pipelines for production systems.