Multimodal Video Scene Classification for Adaptive Video Processing

The paper introduces a multimodal video scene classification algorithm, a hot research topic in computer vision, which serves as a pre‑processing step for complex multimedia tasks such as content‑adaptive transcoding, intelligent video restoration, and video quality assessment (VQA). By automatically selecting appropriate models for different scene types, the algorithm improves the effectiveness of downstream video processing.

Background: User‑generated content (UGC) on platforms like Bilibili contains diverse scenes with varying visual quality. Applying a single enhancement method to all scenes is sub‑optimal, thus a front‑end algorithm that can recognize and classify video scenes is needed. Scene classification can be achieved by analyzing frame‑level features.

Existing techniques: Traditional image classification relied on handcrafted features (SIFT, HOG) and classic classifiers (SVM, decision trees, random forests, K‑NN, Naïve Bayes). Deep learning, especially Convolutional Neural Networks (CNNs) such as LeNet5, GoogLeNet, VGG, ResNet, and EfficientNet, provides end‑to‑end feature learning but requires large labeled datasets. Multimodal algorithms that fuse visual, textual, and audio modalities have emerged, improving generalization and enabling cross‑modal classification.

Common public datasets (CIFAR‑10, CIFAR‑100, ImageNet) are not tailored to specific business scenarios, prompting the need for custom data collection and annotation.

Algorithm design:

1. Data labeling – Prompt engineering for CLIP is used to generate pseudo‑labels for a small unlabeled image set (zero‑shot). The prompts are designed according to business needs, and similarity between image and text embeddings determines the initial class.

2. Model training – The pre‑trained CLIP image encoder replaces traditional feature extractors. A fully‑connected layer is added, and the cleaned pseudo‑labeled dataset is used to fine‑tune the model. Iterative training improves classification accuracy.

3. Experiments – ResNet‑50 is used as a baseline for a three‑class task (nature, animation/game, document) with 5,920 manually labeled images (90% training, 10% validation). Training ResNet‑50 for 300 epochs takes ~180 minutes on an RTX 3090, while the CLIP‑based model with a fully‑connected head converges in 600 epochs within ~125 seconds. Evaluation on accuracy, precision, recall, and F1 score shows the CLIP‑based solution outperforms both ResNet‑50 and CLIP zero‑shot approaches.

4. Video scene classification pipeline – Videos are first segmented into scene clips using frame‑difference based scene‑change detection. Key frames from each clip are extracted and classified. Classification scores are normalized and aggregated to determine the final scene label for each segment.

Conclusion and outlook: The rapid development of multimodal models has accelerated the evolution of classification algorithms, enhancing training efficiency and accuracy. The proposed multimodal scene classification system can be integrated into Bilibili’s multimedia projects (adaptive transcoding, intelligent restoration, VQA) to enable scene‑aware quality control and bitrate optimization. Future work will explore larger multimodal models and further improve system performance.

References: [1] He et al., “Deep residual learning for image recognition,” CVPR 2016. [2] Radford et al., “Learning transferable visual models from natural language supervision,” ICML 2021. [3] Li et al., “Blip: Bootstrapping language‑image pre‑training for unified vision‑language understanding and generation,” ICML 2022.