How HINT’s Hierarchical Multi‑Head Attention Boosts Image Restoration

Paper Information

Title: Devil is in the Uniformity: Exploring Diverse Learners within Transformer for Image Restoration

Affiliations: Nankai University Computer Science School, VCIP & TMCC & DISSec, Nankai International Advanced Institute (Shenzhen·Futian), Nanjing University of Science and Technology Computer Science & Engineering School

ArXiv PDF: https://arxiv.org/pdf/2503.20174v1

Research Background

Transformer‑based image restoration models achieve strong performance, but the standard multi‑head attention (MHA) allocates identical‑size sub‑spaces to each head. This leads to redundant attention maps where multiple heads focus on the same regions and ignore degraded areas, limiting restoration quality.

Contributions

Hierarchical Multi‑Head Attention (HMHA): Partitions the channel dimension into a hierarchy of sub‑spaces C = [C₁, C₂, …, C_h] with C₁ ≤ C₂ ≤ … ≤ C_h. Each head attends within a distinct sub‑space, encouraging diverse contextual feature learning.

Query‑Key Cache Updating (QKCU) module: Maintains intra‑ and inter‑layer caches of query‑key pairs. A gating mechanism selectively updates the cache, allowing heads to share the most informative pairs and reducing redundancy.

HINT model: Integrates HMHA and QKCU into a four‑level encoder‑decoder architecture with a bottleneck at level 4 and a refinement stage composed of four basic blocks.

Method Details

HMHA first re‑orders channels based on similarity, then splits the channel space into hierarchical sub‑spaces. Within each sub‑space, a head performs scaled dot‑product attention, ensuring that each head extracts distinct features. QKCU stores query‑key pairs in a cache; a learned gate decides whether to replace cached entries, enabling efficient information flow across layers.

Experimental Setup

Datasets: Twelve benchmark datasets covering five restoration tasks – low‑light enhancement (LOL‑v2), dehazing (SOTS), desnowing (Snow100K), denoising, and deraining.

Metrics: PSNR, SSIM, and the non‑reference metric MANIQA for real‑world images.

Training: Optimizer – AdamW; learning‑rate hyper‑parameter α = 0.9. The HINT encoder‑decoder has four hierarchical levels, a bottleneck at level 4, and a refinement stage with four basic blocks.

Results and Analysis

Low‑light enhancement (LOL‑v2): Improves average PSNR by 0.9 dB over Retinexformer and at least 1.74 dB over other baselines.

Desnowing (Snow100K): Achieves the highest PSNR, 1.64 dB above the recent AST pipeline.

Dehazing (SOTS): Gains ≥0.35 dB PSNR over competing methods.

Model efficiency: Delivers the best PSNR while using fewer parameters than CNN‑based MIRNet and Transformer‑based IPT/Restormer.

Qualitative results show more vivid colors, fewer artifacts, and better detail preservation across all tasks.

Conclusion

By introducing hierarchical sub‑space partitioning and a query‑key cache, HINT mitigates the redundancy inherent in standard MHA, yielding superior performance on diverse low‑level vision tasks while keeping model size modest.

Future Work

Extreme low‑light scenarios remain challenging. Future research will collect larger real‑world datasets and explore scaling the architecture to handle such conditions.

Appendix

Code, pretrained models, and additional resources are available at https://github.com/AIFengheshu/Plug-play-modules