Why Adding Toxic Data Can Make Language Models Safer and More Capable

Large language models (LLMs) traditionally filter out toxic data during pre‑training to avoid harmful outputs. A new paper titled When Bad Data Leads to Good Models challenges this practice by demonstrating that modestly injecting toxic data into the pre‑training corpus can improve the model’s internal toxicity representation, making subsequent detoxification more efficient without sacrificing overall performance.

Research Motivation

Strict filtering reduces data diversity and can cause the toxicity concept to become entangled with other features, making it hard to control later. The authors hypothesize that increasing the proportion of toxic examples during pre‑training enhances the model’s “alignment ability,” allowing post‑training techniques to steer the model toward harmless behavior more easily.

If a model cannot truly “forget” toxicity, it may be better to let it understand toxicity so that later interventions can be more precise.

Theoretical Framework: Feature Entanglement

The paper adopts the “superposition hypothesis” (Elhage et al.) which suggests that when the number of features exceeds the number of neurons, multiple features are compressed into the same dimensions, causing entanglement. A mathematical metric is introduced to quantify this entanglement:

Symbol : u – unit direction vector of a feature.

Symbol : c – maximum absolute cosine similarity between this feature and any other feature.

Interpretation : Values close to 1 indicate high entanglement; values near 0 indicate independence.

Intuitively, this is like a crowded room where many voices overlap; a high entanglement score means a feature’s “voice” is hard to isolate.

Toy Experiment

A 4‑layer Transformer is trained on synthetic sequences generated by Markov chains, with some features deliberately under‑represented. By varying the proportion of these under‑represented features, the authors observe changes in the entanglement metric.

The results show that increasing the proportion of under‑represented features significantly reduces entanglement.

When toxic content is scarce in pre‑training, its representation becomes highly entangled with other features, causing post‑training interventions to unintentionally damage general abilities. Adding toxic data makes the toxicity representation more independent and easier to control.

Real‑World Experiments with Olmo‑1B

The authors train several Olmo‑1B models with varying ratios of clean data (C4) and toxic data (4chan), ranging from 0% to 25% toxic content while keeping the total amount of clean data constant.

Evaluation Metrics

General Ability : Measured with MMLU benchmark.

Toxicity Detection : Measured with the Toxigen dataset.

Findings:

Adding a moderate amount of toxic data has little impact on MMLU scores and can even yield slight improvements.

Toxicity detection performance improves markedly as the toxic data proportion rises.

Bad data (toxic content) in pre‑training helps the model build clearer toxicity representations, making post‑training interventions more effective.

Linear Probe Experiments

Linear classifiers are trained on attention heads across model layers to predict toxicity. Models trained with toxic data exhibit higher probe accuracy and contain more “high‑accuracy heads,” acting as dedicated toxicity sensors for precise post‑training control.

Logit Lens Analysis

Using Logit Lens, the top 50 words most aligned with the toxicity direction are identified. Models trained with toxic data surface more genuine toxic terms (e.g., “stupid”, “Jew”, “hate”), whereas models trained only on clean data associate the toxicity direction with neutral words.

Toxic data helps the model construct a more accurate, linear toxicity representation.

Inference‑Time Intervention (ITI)

ITI adjusts model activations during generation by pushing away from identified toxic directions. Three intervention strengths are tested (weak = 4, medium = 8, strong = 12) on the top 30 high‑accuracy heads.

Without intervention, higher toxic data ratios increase baseline toxicity.

With ITI, a toxic data ratio around 10% yields the lowest toxicity, forming a “smile curve”.

Comparison with Baselines

The “pre‑train‑plus‑toxic” approach is compared against several baselines:

Prompt engineering with harmless prompts.

MEDA/INST – adding toxicity annotations to data.

SFT/DPO – supervised fine‑tuning and direct preference optimization.

Results show that with 10% toxic data and weak ITI, the model achieves the lowest toxicity on Toxigen and Real Toxicity Prompts while incurring minimal cross‑entropy loss, and it resists red‑team attacks best.

This method reduces toxicity while preserving general ability and requires no manual labeling.

Conclusion

The study overturns the conventional belief that “bad data must be filtered out.” Moderate inclusion of toxic data clarifies the model’s toxicity representation, enabling more effective post‑training detoxification and preserving strong overall performance. The authors advocate for viewing pre‑training and post‑training as a unified system and for empirically driven data‑selection strategies.