video deduplication

High Availability Architecture

Apr 27, 2023 · Artificial Intelligence

Design and Optimization of Bilibili's Large‑Scale Video Duplicate Detection System

This article describes the design, algorithmic improvements, and engineering performance optimizations of Bilibili's massive video duplicate detection (collision) system, covering challenges of low‑edit‑degree reposts, two‑stage retrieval, self‑supervised feature extraction, GPU‑accelerated preprocessing, and the resulting gains in accuracy and throughput.

BilibiliFeature ExtractionVector Search

0 likes · 17 min read

Design and Optimization of Bilibili's Large‑Scale Video Duplicate Detection System

Bilibili Tech

Apr 21, 2023 · Artificial Intelligence

Design and Optimization of Bilibili's Large-Scale Video Duplicate Detection System

Bilibili built a massive video‑duplicate detection platform that trains a self‑supervised ResNet‑50 feature extractor, removes black borders, and uses a two‑stage ANN‑plus‑segment‑level matching pipeline accelerated by custom GPU decoding and inference, boosting duplicate rejection 7.5×, recall 3.75×, and cutting manual misses from 65 to 5 per day.

Feature ExtractionGPU Accelerationdeep learning

0 likes · 19 min read

HomeTech

Aug 7, 2019 · Artificial Intelligence

Near-Duplicate Video Retrieval: Framework, Feature Extraction, Metric Learning, and Model Optimization

This article presents a comprehensive study of near‑duplicate video retrieval, covering the definition of near‑duplicate videos, motivations for deduplication, challenges, a two‑stage offline/online processing framework, keyframe and VGG16‑based feature extraction, metric‑learning loss functions, training procedures, dataset preparation, evaluation metrics, and model enhancements using LSTM and attention mechanisms.

AttentionLSTMVGG16

0 likes · 12 min read

Near-Duplicate Video Retrieval: Framework, Feature Extraction, Metric Learning, and Model Optimization

Xianyu Technology

Sep 7, 2018 · Artificial Intelligence

Video Deduplication on Xianyu Using High‑Dimensional Vector Retrieval

The Xianyu platform combats video plagiarism by extracting key frames, converting them into 1024‑dimensional vectors, and using product quantization‑based high‑dimensional vector retrieval to achieve over 95% recall with ~100 ms latency and more than 1000 QPS, enabling scalable video, image, and product deduplication.

PQVector Retrievalfaiss

0 likes · 12 min read

Video Deduplication on Xianyu Using High‑Dimensional Vector Retrieval