Why 90% of DETR Queries Stay Idle and How PaQ‑DETR Boosts mAP by 4.2%

Problem: Query activation imbalance in DETR

DETR-style detectors use hundreds of object queries, but analysis of Deformable‑DETR, DINO and other variants shows that a tiny fraction of queries receive most gradient updates while the majority remain idle. The Gini coefficient of query activation reaches 0.97, indicating a severe long‑tail distribution that wastes model capacity.

Why prior solutions fall short

Dynamic‑query generation improves query expressiveness but does not change the supervision distribution. One‑to‑many matching increases the number of positive samples but still leaves most queries under‑supervised. Consequently the activation imbalance persists.

PaQ‑DETR design

PaQ‑DETR jointly optimizes query representation and supervision allocation through two cooperating components:

Pattern‑Sharing Dynamic Query Generation : Learn a small set of shared semantic patterns (e.g., 50–150). For each image, a lightweight two‑layer MLP produces a softmax‑normalized weight vector that mixes these patterns into the full set of queries, enabling gradient sharing across queries.

Quality‑Aware Adaptive Assignment : Define a quality score q = IoU - \lambda \times cls\_confidence for each prediction–ground‑truth pair. For each ground‑truth object, select the top‑k predictions by q and allocate a dynamic number of positives proportional to the summed quality, giving high‑quality but low‑confidence predictions more supervision.

Dynamic query generation

Feature extraction & fusion : Multi‑scale Transformer encoder features are enlarged with dilated convolutions and fused to obtain a global descriptor.

Weight generation : The descriptor is spatially pooled and passed through a two‑layer MLP; a Softmax ensures the weights sum to one, forming a mixing recipe.

Query synthesis : Each final query q_i = \sum_j w_{ij} \times p_j, where p_j is a learned pattern and w_{ij} is the generated weight. Gradients from any pattern flow to all queries that use it, achieving true gradient sharing.

Quality‑aware assignment

The quality score combines IoU and a weighted classification confidence (parameter \lambda balances the two terms). For each ground‑truth object, the top‑k predictions with highest scores are considered; the number of positives N = \max(N_{min}, \text{round}(\sum_{i=1}^k q_i / \tau)), where \tau is a scaling factor. This adaptive mechanism concentrates supervision on the most informative samples.

Experimental results

Evaluated on COCO 2017 and additional benchmarks without adding extra inference modules. Compared with strong baselines (Deformable‑DETR, DAB‑DETR, DN‑DETR, DINO), PaQ‑DETR consistently improves mAP. Example: on the DINO++ baseline (50.3 mAP) PaQ‑DETR reaches 51.9 mAP (+1.6 pts). With a Swin‑L backbone, PaQ‑DETR attains 57.8 mAP, surpassing all compared methods.

Ablation study

Dynamic query learning alone: +1.1 mAP, especially for large objects.

Quality‑aware assignment alone: +0.8 mAP.

Both combined: +1.6 mAP and Gini coefficient reduced from 0.97 to 0.89.

Pattern count: 150 patterns give the best trade‑off; even 50 patterns still provide noticeable gains.

Diversity‑loss weight: a moderate weight prevents pattern collapse and improves performance.

Model analysis

Training curves show faster convergence than Deformable‑DETR, DN‑DETR and DINO. Visualizations of dynamic weights reveal category‑specific pattern combinations and shared patterns across categories, indicating semantic reuse. t‑SNE of weight vectors clusters images by semantic content (animals, vehicles, aircraft, etc.). Computational overhead is minimal: <5 % more FLOPs, 0.5 GB extra memory, and a 0.2 FPS drop at inference.

Limitations and future work

Optimal pattern count may vary across datasets; automatic tuning is an open direction.

Fixed query count can bottleneck extremely dense scenes; integrating with dynamic query‑count prediction (e.g., DQ‑DETR) is a possible extension.

Extending the pattern‑sharing idea to tasks beyond detection (video detection, panoptic segmentation) remains to be explored.

Key takeaways

Balancing query utilization is more effective than merely increasing model size.

Joint optimization of query representation and supervision yields synergistic improvements.

Lightweight architectural changes can deliver large performance boosts with negligible cost.