How an Automatic Pipeline Framework Supercharges AI Inference in Retail Surveillance

Background

With deep learning booming worldwide, many algorithm models emerge, but optimizing them for production often sacrifices accuracy. In retail stores, high‑resolution cameras (1080p+) and dozens of streams require heavy AI models (person detection, re‑identification, action recognition) that strain GPU resources.

Manual vs. Automatic Pipelines

Manual pipelines require developers to split algorithms, write separate processes, and tightly couple data channels, leading to low model‑iteration efficiency, tangled scheduling logic, and poor flexibility. Automatic pipelines generate the data flow and scheduling automatically from the training code, keeping algorithm logic independent of execution logic.

System Framework

The framework is built in Python to match TensorFlow model definitions, allowing algorithm engineers to reuse training code directly. It consists of a task parser, task scheduler manager, disaster recovery, task monitor, and cross‑process data pipeline, all abstracted to resemble a factory production line.

Streaming Programming

AI tasks are divided into stages (pre‑process, inference, post‑process) and described with input/output data structures. The framework builds a task graph, shares image data via a multi‑pipeline shared buffer, and schedules tasks across processes without copying image data.

Task Scheduling System

Because Python’s GIL limits multithreading, the system uses multiple processes. Each pipeline mostly uses a single resource type (CPU or GPU), enabling efficient resource allocation. A lightweight wake‑up mechanism based on data‑shared pipelines replaces heavyweight semaphores.

High‑Performance Cross‑Process Shared Pipeline

Custom shared memory (SHA) supports complex object serialization, outperforming standard multiprocessing.Queue reads by 400× and matching multiprocessing.Array write speed while handling arbitrary Python objects.

Performance Evaluation

Tests on an i5‑6‑core CPU with a GTX‑1070 GPU showed that the automatic pipeline version supports 18 camera streams, processes a frame in 11.9 ms, and raises CPU utilization from 155 % to 282 % and GPU utilization from 36‑40 % to 70‑80 % compared with the serial and manual pipeline versions.

Business Impact

The deployed system has been running for months, providing retailers with heat‑maps and flow‑maps that guide store layout and product placement, improving shopping experience and sales.

Conclusion and Outlook

The automatic pipeline dramatically reduces development complexity, boosts model iteration speed, and improves inference efficiency. Future work includes extending the framework to training pipelines and further optimizing stability and performance.