Why the Scale‑Aware Modulation Transformer Outperforms CNNs and Vision Transformers with Fewer Parameters

01 Introduction

In recent years, vision foundation models based on Transformers and CNNs have achieved great success. Many works combine Transformer structures with CNN architectures to create more efficient hybrid CNN‑Transformer networks, yet their accuracy often remains unsatisfactory. This article presents a new foundation model, the Scale‑Aware Modulation Transformer (SMT), which delivers substantial performance gains with lower parameter count and FLOPs.

02 Motivation

Shallow stages of hierarchical networks suffer from the quadratic complexity of self‑attention due to high‑resolution feature maps, making efficient attention design crucial.

Previous hierarchical models such as Swin, CvT, PvT, etc., mainly focus on designing more efficient attention units (e.g., local attention, lightweight convolutional attention).

ViT shows that shallow layers capture local information while deep layers capture global dependencies; this transition also appears in multi‑stage architectures.

Modeling and simulating this dependency transition is considered important and effective.

03 SMT Framework

The overall SMT architecture consists of four stages with down‑sampling rates of {4, 8, 16, 32}. Unlike FocalNet, SMT employs the proposed Scale‑Aware Modulation (SAM) in the first two stages, then stacks a SAM block followed by a Multi‑Head Self‑Attention (MSA) block in the penultimate stage, and finally uses only an MSA block in the last stage to capture long‑range dependencies.

3.1 Scale‑Aware Modulation Module

SMT introduces a novel lightweight Scale‑Aware Modulation (SAM) unit that captures multi‑scale features while expanding the receptive field, enhancing convolutional modulation capability.

Multi‑Head Mixed Convolution (MHMC)

MHMC incorporates multiple convolutional layers with different kernel sizes across heads, allowing each head to capture spatial features at various scales. Larger heads use larger kernels to enlarge the receptive field and model long‑range dependencies.

Scale‑Aware Aggregation (SAA)

SAA is a lightweight aggregation module that reorganizes and groups features from different MHMC heads, then performs up‑down fusion within each group and cross‑group fusion via 1×1 convolutions, achieving efficient multi‑scale feature integration.

Scale‑Aware Modulation (SAM) Operator

After MHMC and SAA produce an output feature map (the modulator), SAM uses a scalar multiplication to modulate the value tensor V.

3.2 Evolutionary Hybrid Network (EHN)

EHN reallocates computation modules according to the changing dependency capture patterns across stages. Two hybrid stacking strategies are evaluated for the penultimate stage: (i) sequentially stacking a SAM block and an MSA block, and (ii) placing SAM blocks in the first half and MSA blocks in the second half. Experiments on ImageNet‑1K show strategy (i) performs better.

Analysis of the relative receptive field of MSA blocks in the penultimate stage reveals a slight decline in shallow layers, attributed to SAM’s influence, followed by a steady increase as depth grows, confirming that EHN effectively models the transition from local to global dependency capture.

04 Experiments

4.1 Image Classification

SMT consistently outperforms larger models on ImageNet‑1K with fewer parameters and FLOPs. Notably, SMT‑B achieves 84.3% top‑1 accuracy with only 32.0 M parameters and 7.7 G FLOPs, surpassing many 80 M‑parameter models. Pre‑training on ImageNet‑22K further boosts SMT‑L to 87.1%/88.1% accuracy, beating InternImage‑XL with 4× fewer parameters and 3× fewer FLOPs.

4.2 Object Detection

Across multiple detection frameworks (Mask R‑CNN, Cascade R‑CNN, RetinaNet, Sparse R‑CNN, ATSS, DINO), SMT delivers higher mAP while using significantly fewer parameters. For example, SMT‑B improves Mask R‑CNN by 2.1 mAP (1× schedule) and 1.3 mAP (3× schedule) with half the parameters of Swin‑B.

4.3 Semantic Segmentation

Using the UperNet framework on ADE20K, SMT achieves better segmentation accuracy with lower parameter and FLOP budgets across all scales.

4.4 Ablation Studies

Ablation results confirm the effectiveness of the SAM module, the MHMC design, and the evolutionary hybrid stacking strategy.

05 Conclusion and Outlook

SMT demonstrates strong scalability and superior performance for vision foundation models. Future work includes exploring even more efficient computation modules to replace SAM/MSA in shallow stages and further integrating CNN and Transformer strengths for next‑generation hybrid architectures.

References

Scale‑Aware Modulation Meet Transformer, arXiv:2307.08579

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, arXiv:2010.11929

Focal Modulation Network, arXiv:2203.11926

MixConv: Mixed Depthwise Convolutional Kernels, arXiv:1907.09595

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, arXiv:2103.14030

InternImage: Exploring Large‑Scale Vision Foundation Models with Deformable Convolutions, arXiv:2211.05778