Enhancing Multimodal Video Classification with Improved Image Features and Category System

The presentation introduces Alibaba Entertainment's category system, a hierarchical taxonomy designed to support video operations, recommendation cold‑start, and search relevance, emphasizing its business value for operations, recommendation, and search.

It describes the construction process: defining category standards, iterative refinement with reviewers, automated model training, and continuous optimization, resulting in a three‑level taxonomy that improves operational efficiency, recommendation PVCTR (+43%), and search accuracy.

The talk then explains program (show) classification, also known as short‑long association, highlighting its utility for locating related short clips and improving fine‑grained classification across over 60,000 programs.

The core of the session focuses on the multimodal video classification pipeline, which embeds video, text, and audio modalities, fuses them via a NeXtVLAD‑based network, applies a gating mechanism, and finally classifies using a Mixture‑of‑Experts (MoE) model.

Several feature‑level improvements are detailed:

Backbone upgrade to EfficientNet, chosen for its superior accuracy‑efficiency trade‑off and strong fine‑grained representation.

Data‑augmentation strategies such as Attention Dropping and focused Cropping to force the network to learn diverse visual cues.

Training methodology shifts from purely supervised classification to unsupervised instance discrimination (MoCo), leveraging momentum encoders and contrastive loss to obtain richer embeddings.

Experimental results show that these enhancements raise classification performance (e.g., +20% on targeted data augmentation, +13 points when scaling classes to 27k) and demonstrate the potential of better backbones and unsupervised learning for large‑scale categorization.

Case analyses illustrate how the improved model corrects misclassifications (e.g., distinguishing similar medical or historical characters) and how pitfalls such as model “shortcut” learning of subtitles, logos, or black borders can be mitigated by targeted augmentations.

The speaker concludes with reflections on the importance of robust feature extraction, the limitations of pure instance discrimination for video frames, and outlines future work including exploring SimCLR‑style methods and more effective multimodal fusion architectures.

A brief Q&A addresses modality fusion, training strategies, sample balancing, hierarchical model updates, and challenges of few‑shot learning.