Rethinking Diffusion‑Based Video Super‑Resolution with Dense Feature‑Guided Alignment (DGAF‑VSR)

Background

CVPR (IEEE/CVF Conference on Computer Vision and Pattern Recognition) is a top venue for computer‑vision research. This work, authored by the Taotian Video Technology team together with academic collaborators, addresses the long‑standing trade‑off in video super‑resolution (VSR) between perceptual quality, reconstruction fidelity, and temporal consistency.

Key Observations

Two observations motivate the redesign of diffusion‑based VSR:

Observation 1: Features in the latent space exhibit stronger spatial and temporal correlations than raw pixels, making the feature domain a more reliable carrier for temporal guidance.

Observation 2: Performing warping at a higher resolution preserves high‑frequency details better, but the benefit saturates beyond an optimal scale.

Core Modules

Optical‑Guided Warping Module (OGWM)

OGWM aligns up‑sampled features of the previous frame using optical flow estimated by RAFT. The three‑step pipeline is:

Input preparation: Upsample the previous‑frame features to a higher resolution to reduce detail loss during warping.

Feature alignment: Apply the RAFT flow to deform the upsampled features, achieving precise spatial correspondence.

Down‑sampling and integration: Downsample the aligned high‑resolution features back to the original scale and feed them to the temporal condition module.

This “up‑scale → align → down‑scale” strategy mitigates the detail loss and error accumulation typical of low‑resolution pixel‑level alignment.

Feature‑wise Temporal Condition Module (FTCM)

FTCM injects densely aligned neighboring‑frame features into each diffusion step. Unlike previous diffusion‑based VSR methods that use a lightweight U‑Net encoder as a temporal condition, FTCM adopts a full U‑Net (inspired by BrushNet) to provide stronger feature extraction and reconstruction capability. At each diffusion timestep, the denoising process depends on both the current frame’s latent feature and the OGWM‑aligned neighbor features, forming a dense feature‑guided diffusion.

Experimental Validation

Extensive experiments were conducted on synthetic datasets (REDS4, Vid4) and a real‑world dataset (VideoLQ). DGAF‑VSR was compared against non‑diffusion baselines (EDVR, BasicVSR, BasicVSR++, RVRT) and diffusion‑based baselines (StableVSR, MGLD‑VSR, UAV, STAR, DOVE, SeedVR2).

DGAF‑VSR achieved dominant scores on perceptual metrics (LPIPS, DISTS, MUSIQ, CLIP‑IQA, NIQE) and the best PSNR/SSIM among diffusion‑based methods. On the real‑world VideoLQ dataset, it also obtained the highest performance, demonstrating strong practical potential.

Ablation Study

Removing either OGWM or FTCM degrades performance, confirming that both dense feature alignment and temporal conditioning are essential for the observed gains.

Conclusion and Outlook

DGAF‑VSR demonstrates that careful analysis of feature‑level alignment and dense temporal guidance can overcome the perceptual‑fidelity‑temporal trade‑off in diffusion‑based VSR. While the method sets new SOTA results, inference cost remains high. Future work may explore more efficient alignment strategies, lightweight dense‑guidance mechanisms, or reduced‑step diffusion pipelines to further advance generative video super‑resolution for live‑streaming and short‑video applications.