Machine Learning Algorithms & Natural Language Processing

Oct 26, 2026 · Artificial Intelligence

On-Policy Distillation Explained: White-Box, Self-Distillation & Black-Box Methods

This article provides a comprehensive analysis of On-Policy Distillation (OPD), covering white-box distillation with token-level probability matching, self-distillation using privileged context, black-box distillation via reward modeling, and industrial applications from MiMo, DeepSeek, and Qwen, while discussing OPD's limitations and generalization capabilities.