Understanding DeepSort: A Classic Multi-Object Tracking Algorithm

Object tracking is a key task in computer vision that aims to locate and follow specific targets across video frames, enabling applications such as surveillance, traffic monitoring, and security.

The basic pipeline includes detection, feature extraction, matching, position update, and motion estimation. Classic methods include Kalman filter, particle filter, CAMShift, optical flow, and deep learning approaches.

DeepSort, an extension of the SORT algorithm, incorporates a CNN‑based appearance descriptor trained on person re‑identification datasets, combined with a matching cascade and track confirmation mechanism to improve multi‑object tracking, especially under occlusion.

SORT relies on Kalman filtering for motion prediction and the Hungarian algorithm for assignment based on IoU cost matrices. It works well when targets are not occluded but may lose IDs during occlusions.

DeepSort adds a deep appearance model (a lightweight ResNet‑inspired network outputting a 128‑dimensional feature vector) and uses a cascade matching strategy that first matches confirmed tracks with detections using both motion and appearance cues.

The workflow proceeds as: initialize tracks from detections, predict next positions with Kalman filters, compute IoU cost, solve assignment with Hungarian algorithm, update matched tracks, create new tracks for unmatched detections, and delete unmatched tracks after a timeout. Confirmed tracks undergo additional appearance‑based matching.

References include the original DeepSort paper (arXiv:1703.07402), the official GitHub repository, and several tutorial videos and blog posts.