Winning Kaggle’s Jigsaw Toxicity Challenge with Transfer Learning and Zero‑Shot Classification

Kaggle has hosted four editions of the Jigsaw toxic comment classification competition. The first edition (2018) pre‑dated BERT, the second saw BERT dominate, the third introduced multilingual toxic comment detection, and the fourth removed the training set entirely, providing only a small validation set of paired sentences with leak data.

Solution Overview

The winning approach consists of three main steps:

Historical Model Training : Use the 2018 and 2019 Jigsaw datasets, which contain fine‑grained toxicity labels (six labels in the first edition, seven in the second), to train separate models.

Genetic‑Algorithm Weight Optimization : Apply the models from step 1 to predict the fourth‑edition validation pairs. Then run a genetic algorithm to search for optimal integer weights (0‑4) for the six label predictions, maximizing the validation score. The core code is shown below.

Final Inference and Ensembling : Combine the trained model with the optimized weights to generate scores for the test data, and finally ensemble fifteen such models to produce the final submission.

Genetic Algorithm Code

from geneticalgorithm import geneticalgorithm as ga

def loss_func(wt):
    less_toxic_score = np.zeros((len(less_toxic_pred),), dtype=np.float16)
    more_toxic_score = np.zeros((len(more_toxic_pred),), dtype=np.float16)
    for i in range(6):
        less_toxic_score += less_toxic_pred[:, i] * wt[i]
        more_toxic_score += more_toxic_pred[:, i] * wt[i]
    return 1.0 - np.mean(less_toxic_score < more_toxic_score)

varbound = np.array([[0, 4]] * 6)
model = ga(function=loss_func, dimension=6, variable_type='int', variable_boundaries=varbound)
model.run()
best_wt = model.output_dict['variable']

# validate result again
less_toxic_score = np.zeros((len(less_toxic_pred),), dtype=np.float16)
more_toxic_score = np.zeros((len(more_toxic_pred),), dtype=np.float16)
for i in range(6):
    less_toxic_score += less_toxic_pred[:, i] * best_wt[i]
    more_toxic_score += more_toxic_pred[:, i] * best_wt[i]
print(np.mean(less_toxic_score < more_toxic_score))

This pipeline ensures that the cross‑validation performance is reliable and that the model generalizes well to unseen data by leveraging prior knowledge from earlier competitions.

Additional references include earlier posts on bias mitigation in machine learning, a comprehensive review of NLP and computer‑vision pre‑training trends, and a guide to industrial text‑classification pitfalls.