DeepKD: Double‑Layer Decoupling and Adaptive Denoising Set New ImageNet SOTA

Problem Statement

Knowledge distillation (KD) suffers from two fundamental issues: (1) inherent conflict between target‑class and non‑target‑class knowledge streams during optimization, leading to sub‑optimal trajectories, and (2) low‑confidence logits in the non‑target set inject noisy signals that hinder effective knowledge transfer.

Core Innovations

DeepKD framework : integrates a double‑layer decoupling strategy with an adaptive denoising mechanism.

GSNR‑based momentum allocation : derives independent momentum buffers for task‑oriented gradient (TOG), target‑class gradient (TCG) and non‑target‑class gradient (NCG) based on their gradient signal‑to‑noise ratio (GSNR).

Dynamic top‑K mask (DTM) : progressively filters low‑confidence logits following a curriculum‑learning schedule, preserving semantically related dark knowledge while removing noise.

Theoretical Analysis of GSNR and Momentum Allocation

The authors first analyze the gradient GSNR of each component. By sampling gradients every 200 iterations, they estimate the expectation and variance of the stochastic gradient vector. Empirically, NCG and TOG exhibit higher GSNR than TCG, suggesting that larger momentum should be assigned to components with higher GSNR. Consequently, they propose a momentum update: v_{t+1}=\beta v_t + (1-\beta) g_t where the base momentum \(\beta\) is modulated per component according to its GSNR, yielding a positive correlation between optimal momentum coefficient and GSNR (see Fig. 2 in the original paper).

Dynamic Top‑K Mask Mechanism

Two limitations of prior logit‑based KD methods are identified: (1) teacher logits for target classes have very high confidence (often >92%), while non‑target logits are low‑confidence but contain valuable dark knowledge; (2) only non‑target classes that are semantically close to the target provide useful signals, whereas distant classes add noise.

To address this, the authors first design a static top‑K mask that permanently discards logits with the largest semantic distance. Building on this, the dynamic top‑K mask gradually expands the retained set from 5 % of classes to the full class set in three curriculum phases:

Easy‑learning phase: linearly increase K from 5 % to the optimal static K.

Transition phase: keep K at the optimal static value.

Hard‑learning phase: linearly expand K to cover all classes.

The mask for iteration t is computed as:

Mask_t = \mathbf{1}_{\text{rank}(logits) \le K_t}

where rank denotes the ascending order of logits. The masked distillation loss multiplies the standard KD loss element‑wise by Mask_t, effectively suppressing noisy logits while retaining semantically relevant dark knowledge.

DeepKD Framework

Combining the GSNR‑driven momentum buffers and the dynamic top‑K mask, DeepKD decomposes the overall loss into three parallel streams (TOG, TCG, NCG). Each stream has its own momentum buffer and is optimized according to its GSNR‑derived coefficient. The final loss is:

L = \alpha L_{CE} + \beta L_{TCG} + \gamma L_{NCG}^{\text{DTM}}

where L_{CE} is the standard cross‑entropy loss, L_{TCG} the target‑class KD loss, and L_{NCG}^{\text{DTM}} the masked non‑target KD loss.

Experiments

DeepKD is evaluated on three benchmarks:

CIFAR‑100 (100 classes, 32×32 images)

ImageNet‑1K (1 000 classes, 224×224 images)

MS‑COCO object detection (80 classes)

Key results include:

On CIFAR‑100, DeepKD improves top‑1 accuracy by +0.61 % to +3.70 % over same‑architecture baselines; adding the dynamic mask yields an additional +1.86 % (up to 79.15 %).

On ImageNet‑1K, ResNet‑50 and MobileNet‑V1 see top‑1 gains of +4.15 % to +74.65 % (the latter figure appears to be a typo in the source but is retained as reported). The CRLD+DeepKD (with top‑k) reaches 74.23 % for ResNet34→ResNet18 and 75.75 % for RegNetY‑16GF→DeiT‑Tiny.

For MS‑COCO detection, the dynamic mask raises AP by +1.93 % to 32.16 % and achieves a peak AP of 36.59 %, surpassing feature‑based methods such as LSKD.

All experiments use SGD with momentum 0.9, weight decay 5×10⁻⁴ (CIFAR) or 1×10⁻⁴ (ImageNet), batch sizes 64 (CIFAR) and 512 (ImageNet), and are run on RTX 4090 GPUs.

Ablation Studies

Separate ablations on CIFAR‑100 demonstrate that:

Removing the GSNR‑driven momentum allocation degrades performance, confirming the importance of component‑wise momentum.

Omitting the dynamic top‑K mask reduces accuracy gains, showing its complementary effect.

The method introduces only one hyper‑parameter Δ (momentum difference), which remains robust across datasets (Δ=0.075 for KD+DeepKD on CIFAR, Δ=0.05 for other variants).

Further ablations vary Δ, the static K value, and the phase lengths of the curriculum, all confirming that each component contributes positively and that their combination yields the best results.

Limitations and Future Work

Current work focuses on logit‑based distillation; extending the GSNR‑driven momentum decoupling to feature‑based distillation is a promising direction. Future research will explore multi‑teacher scenarios, cross‑modal knowledge transfer, and automated tuning of the dynamic mask schedule for diverse architectures and datasets.