Accelerating Computer Vision Pipelines with CV-CUDA: Reducing Complexity and Performance Bottlenecks

John Ousterhout's principle that software design aims to reduce complexity also applies to low‑level hardware‑adapted software such as visual model pipelines, where preprocessing and post‑processing become performance bottlenecks when model inference is accelerated.

Traditional CV libraries like OpenCV and TorchVision rely on CPU for most preprocessing, leading to 50‑90% of workload and causing inconsistencies between CPU and GPU versions.

CV‑CUDA, an open‑source GPU‑based image preprocessing library co‑developed by NVIDIA and ByteDance, moves the entire preprocessing pipeline to GPU, achieving up to 30× speedup and 70% overall pipeline efficiency gains, while supporting batch and variable‑shape processing.

The library provides asynchronous, stream‑aware operators, memory pre‑allocation, kernel fusion, and optimized memory access, reducing CPU‑GPU data transfers and resource contention.

Real‑world case studies at NVIDIA, ByteDance and Sina Weibo demonstrate significant throughput improvements (e.g., 20× over OpenCV CPU, 2× over OpenCV GPU) in image classification, OCR, and video processing tasks.

Future work includes expanding the operator set from 20 in the alpha release to over 50 in the upcoming beta, covering more complex algorithms such as ConvexHull and FindContours.