TinySR: Lightweight Diffusion for Real-World Super-Resolution, 5.68x Faster

Real-world image super-resolution (Real-ISR) must handle noise, blur, compression artifacts, lens defects, and ISP-induced degradations simultaneously, making it far more challenging than synthetic degradation settings. Diffusion models have raised the perceptual quality ceiling for Real-ISR, but their iterative denoising and heavy backbone networks hinder deployment on mobile devices where latency, power, memory, and model size are constrained.

Why This Matters

Mobile imaging increasingly demands instant results: photo-album restoration, post-capture enhancement, AIGC image refinement, and continuous video-frame super-resolution. Users will not tolerate seconds of spinning wheels, heating, or frame drops. The real need is a model that delivers stable quality within hardware budgets, requiring joint optimization of parameters, MACs, operator types, memory access patterns, module coupling, and end-to-end inference efficiency.

What TinySR Does

TinySR, from Zhejiang University CAD&CG State Key Lab (AiXM group), vivo BlueImage Lab, and Zhijiang Lab, rethinks the entire diffusion super-resolution pipeline for deployment. Instead of merely reducing sampling steps, it systematically lightens the backbone, VAE, conditioning branches, and runtime flow.

Technical Core 1: Learnable Pruning Decisions

Traditional pruning relies on hand-crafted rules (uniform layer removal, static importance scores) that ignore functional asymmetry across layers—some layers specialize in structure recovery, others in texture synthesis. TinySR introduces learnable probability masks for structural units, letting the model discover which blocks retain strong recovery ability under compression constraints. This shifts the goal from “most important now” to “most recoverable after pruning.”

Dynamic Inter-Block Activation

Building on block-wise pruning, Dynamic Inter-Block Activation lets the training process dynamically adjust activation states across blocks, avoiding static keep/discard decisions. This flexible cross-block exploration improves pruning effectiveness without significantly expanding search complexity.

Expansion-Corrosion Strategy

To stabilize the pruning trajectory, an expansion-corrosion mechanism gradually adjusts masks rather than abruptly collapsing structure. This progressive adaptation preserves exploration space and promotes stable convergence, yielding a network optimized for the task rather than merely shrunk.

Technical Core 2: VAE as a Deployment Bottleneck

VAE overhead is often overlooked but contributes heavily to latency and compute. TinySR identifies three VAE bottlenecks: excessive channel width, costly attention modules, and standard convolutions. It applies channel pruning to reduce width, removes attention, and replaces encoder convolutions with depthwise separable convolutions (SepConv). Ablation shows the lightweight VAE achieves ~10x inference speedup and ~22x MACs reduction while preserving reconstruction quality close to the teacher.

Technical Core 3: Removing Task-Irrelevant Modules

TinySR discards time-step embeddings, text-prompt embeddings, and associated conditioning branches. For a one-step Real-ISR model targeting real-time deployment, these general-purpose generation controls add parameters, branches, and inference complexity with negligible quality gain. Ablation confirms minimal metric change but significant reductions in parameters, MACs, and latency, illustrating that specialized models deploy more easily than general ones.

Pre-Caching for Runtime Efficiency

Modulation parameters stabilize after training and become input-independent. TinySR pre-computes and caches them, eliminating redundant computation, data movement, and module switching at inference. This engineering-oriented optimization narrows the gap between theoretical speedup and user-perceived latency.

Experimental Results

Compared to teacher model TSD-SR, TinySR achieves:

Up to 5.68x inference acceleration

83% parameter reduction

84% MACs reduction

On DIV2K-Val and other benchmarks, TinySR matches or exceeds baselines on LPIPS, DISTS, FID, NIQE, MUSIQ, and CLIPIQA. It avoids over-generated artifacts common in multi-step diffusion and blur typical of some one-step methods, striking a balanced trade-off among natural texture, structural integrity, and perceptual quality. Direct comparisons with OSEDiff, SinSR, and AdcSR show TinySR leads in speed, compute, and model size while retaining competitive quality.

Significance for Mobile Deployment

Instant response for phone imaging enhancement – aligns with real-world user experience goals.

Continuous video-frame processing – lower per-frame cost enables sustained throughput without thermal throttling.

Edge-cloud tiered enhancement – TinySR serves as a lightweight on-device first stage, improving initial quality before cloud refinement.

The work underscores that mobile deployment depends not only on FLOPs but also operator support, cache reuse, quantization readiness, module coupling, and hardware mapping efficiency—TinySR’s design reflects this holistic view.

Conclusion and Outlook

TinySR answers a pragmatic question: when can diffusion super-resolution leave the lab and enter lightweight, real-time, mobile product pipelines? Its contribution is a complete, deployment-driven redesign spanning depth pruning, VAE compression, conditioning removal, and runtime caching. For Real-ISR, this step is critical: only models that preserve detail while fitting mobile budgets can become next-generation imaging primitives. The paper (arXiv:2508.17434) and code (GitHub: Microtreei/TinySR) are publicly available.

Code example

来源：PaperWeekly
本文
约4500字
，建议阅读
9
分钟
本文介绍了 TinySR 轻量扩散模型，实现高质且高效的真实图像超分。