RVM: Real-Time High-Resolution Video Matting

Introduction

Video matting separates foreground people from background in video streams and is widely used in video conferencing and media production. This article summarizes the paper Robust High-Resolution Video Matting with Temporal Guidance , which proposes a real‑time algorithm for high‑resolution video.

Main Contributions

The proposed method processes 4K video at 76 FPS and 1080p video at 104 FPS on an NVIDIA 1080Ti GPU. It introduces temporal guidance through a recurrent neural network (GRU) to overcome the frame‑wise processing of previous methods, reducing flicker, learning richer background cues as objects move, and improving model robustness. The authors also observe that jointly training semantic segmentation benefits the matting task, so the network is built as a multi‑task model.

Network Architecture

The network consists of a feature‑extraction encoder, a recurrent decoder, and a depth‑guided up‑sampling filter. The encoder uses MobileNetV3‑Large as the backbone and incorporates an LR‑ASPP module for semantic segmentation, extracting features at 1/2, 1/4, 1/8 and 1/16 scales.

The recurrent decoder aggregates temporal information with a multi‑scale ConvGRU. For each level, the feature map is split along the channel dimension; one part follows a traditional decoding path while the other passes through ConvGRU, allowing the model to combine current‑frame and past‑frame information.

The decoder comprises three blocks:

Bottleneck block – receives 1/16‑scale features from LR‑ASPP.

Upsampling block – reused for 1/8, 1/4 and 1/2 scales, merging bilinearly up‑sampled output, encoder features of the same scale, and the down‑sampled input image via convolution, batch‑normalization and ReLU.

Output block – predicts a single‑channel alpha matte, a three‑channel foreground, and a one‑channel segmentation map used for the auxiliary segmentation task.

After decoding, a depth‑guided filter restores the high‑resolution result.

Experimental Results

The paper reports both accuracy and speed metrics, illustrated in the accompanying figures. Accuracy results are shown in the first image, while speed benchmarks (4K @ 76 FPS, 1080p @ 104 FPS) are displayed in the second image.

Conclusion

By integrating temporal guidance via ConvGRU and a multi‑task learning framework, the method achieves real‑time, high‑quality matting for high‑resolution video, addressing the limitations of prior frame‑wise approaches.