Exploring the Three Key Research Directions in Video Object Segmentation

Video Object Segmentation (VOS) extracts the full region of a target object in every frame of a video. It underpins downstream tasks such as 3‑D video effects, content remixing, and interactive media.

Semi‑Supervised VOS (One‑Shot VOS)

In semi‑supervised VOS a ground‑truth mask is provided for the target object in the first frame. The algorithm must propagate this mask to all subsequent frames despite object motion, illumination changes, occlusions, and background similarity.

Two main pipeline families exist:

Online learning : The first‑frame mask is used to fine‑tune a segmentation network for the specific video. Representative methods include Lucid datadreaming , OSVOS , and PReMVOS . This yields high accuracy but requires substantial GPU time for per‑video adaptation.

Offline (no online learning) : A pre‑trained model directly predicts masks for all frames. Notable examples are FEELVOS and the Space‑time Memory Network . These methods run in real time with a modest drop in precision.

Performance is measured by the average Jaccard index (J) and the boundary F‑measure (F). Because a first‑frame mask is mandatory, pure semi‑supervised approaches cannot be deployed in fully automatic scenarios.

Interactive VOS

Interactive VOS replaces the first‑frame mask with sparse user inputs (e.g., bounding boxes, scribbles, extreme points) on any frame. The typical workflow consists of five steps:

User provides an interaction on a chosen frame.

An interactive image‑segmentation model generates a mask for that frame.

The mask is propagated to the remaining frames using a semi‑supervised VOS algorithm.

If propagation errors appear, the user adds new interactions on problematic frames.

Steps 3–4 repeat until the segmentation meets the desired quality.

Evaluation follows the DAVIS Challenge metrics: J&F@60s (the Jaccard‑F score interpolated at 60 seconds) and the AUC of the accuracy‑over‑time curve. Low latency is critical, so online‑learning based semi‑supervised methods are rarely used in interactive pipelines.

Unsupervised VOS

Unsupervised VOS requires no user input beyond the raw RGB video. The goal is to automatically segment salient objects. Since object identity is ambiguous, evaluation matches predicted masks to ground‑truth objects based on overlap; extra predicted objects are not penalised. The same average J and F metrics are used.

Research Highlights from Alibaba MoKu Lab

Introduced a “VOS with robust tracking” strategy that raised interactive J&F@60s on the DAVIS 2017 validation set from 0.353 (March 2019) to 0.761 (May 2019).

Achieved a semi‑supervised Jaccard of 0.763, comparable to leading industry results.

Secured 4th place in the interactive track of the 2019 DAVIS Challenge.

Future Directions

To handle complex scenes (tiny objects, fast motion, severe occlusion), the lab plans to explore:

More effective online‑learning schemes.

Enhanced space‑time memory networks.

Region‑proposal and verification pipelines.

Deeper integration with state‑of‑the‑art image segmentation and multi‑object tracking.

References

Caelles et al., “The 2019 DAVIS Challenge on VOS: Unsupervised Multi‑Object Segmentation,” arXiv:1905.00737, 2019.

Khoreva et al., “Lucid Data Dreaming for Object Tracking,” arXiv:1703.09554, 2017.

Caelles et al., “One‑Shot Video Object Segmentation,” CVPR, 2017.

Luiten et al., “PReMVOS: Proposal‑generation, Refinement and Merging for VOS,” arXiv:1807.09190, 2018.

Voigtlaender et al., “FEELVOS: Fast End‑to‑End Embedding Learning for VOS,” CVPR, 2019.

Oh et al., “Fast User‑Guided VOS by Interaction‑and‑Propagation Networks,” CVPR, 2019.

Wang et al., “Fast Online Object Tracking and Segmentation: A Unifying Approach,” CVPR, 2019.

Ren et al., “Robust Multiple Object Mask Propagation with Efficient Object Tracking,” CVPR Workshops, 2019.

Code example

[1] The 2019 DAVIS Challenge on VOS: Unsupervised Multi-ObjectSegmentation. S. Caelles, J. Pont-Tuset, F. Perazzi, A. Montes, K.-K. Maninis,and L. Van Gool .arXiv:1905.00737, 2019
[2] A. Khoreva, R. Benenson, E. Ilg, T. Brox, and B. Schiele. Lucid datadreaming for object tracking. In arXiv preprint arXiv: 1703.09554, 2017. 2
[3] S. Caelles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taix´e,D. Cremers,and L. Van Gool. One-shot video object segmentation. CVPR, 2017
[4] J. Luiten, P. Voigtlaender, and B. Leibe. PReMVOS:Proposal-generation, refinement and merging for video object segmentation.arXiv preprint arXiv:1807.09190, 2018.
[5] Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, BastianLeibe, Liang-Chieh Chen. FEELVOS: Fast End-to-End Embedding Learning for VideoObject Segmentation. CVPR 2019
[6]. Seoung Wug Oh, Joon-Young Lee, Ning Xu, Seon Joo Kim.Fast User-GuidedVideo Object Segmentation by Interaction-and-Propagation Networks. CVPR2019
[7]. Wang, Qiang，Zhang, Li，Luca Bertinetto, Weiming Hu, Philip H.S. Torr.Fast Online ObjectTracking and Segmentation: A Unifying Approach. CVPR2019