C++ Multithreaded Service Architecture for High‑Throughput AI Inference

With the rapid adoption of artificial intelligence, deploying deep‑learning models in production requires low latency and high throughput; the article discusses using C++ together with TensorRT, OpenMP, and CUDA to meet these demands.

It first describes a naïve serial implementation for a face‑recognition access‑control system, where each request passes through detection, quality assessment, feature extraction, and liveness checks sequentially, causing a single request to occupy the entire pipeline and leading to poor scalability under heavy load.

Two performance‑improvement strategies are presented: (1) compress the processing time of a single request, and (2) keep the request time unchanged but handle multiple requests concurrently; both can be combined.

The core solution is a Pthreads‑based multithreaded framework. The main thread only initializes sub‑threads, manages their state, and handles client I/O, while each sub‑thread (module) performs a specific computation. Modules communicate via channel queues, enabling a pipeline where multiple requests are processed in parallel, similar to an assembly line.

Each module extracts data from its input channel, runs the assigned algorithm, and pushes the result to the next channel. The article provides a representative pseudo‑code example for a face‑detect module:

void faceDetectModule::runConsumer() {
    waitStatus(SATUS::RUN);
    while (true) {
        if (checkStatus()) break;
        Frame *curframe = system_in_queue_->pop();
        if (checkStatus()) break;
        detector_->processFrame(curFrame);
        system_out_queue_->push(curframe);
    }
    LOG(INFO) << "faceDetectModule job finish!";
}

The channel implementation uses a thread‑safe FIFO queue; the pop operation acquires a mutex, waits for data, retrieves the front element, releases the lock, and notifies other threads:

Frame* FrameQueue::pop() {
    std::unique_lock<std::mutex> lock(mutex_);
    while (queue_.empty())
        condition_.wait(lock);
    Frame* frame = queue_.front();
    queue_.pop();
    lock.unlock();
    condition_.notify_all();
    return frame;
}

Further enhancements include optimizing the most time‑consuming modules with OpenMP/CUDA and parallelizing independent modules, which can yield additional latency reductions.

In conclusion, the presented C++ Pthreads multithreaded service significantly improves throughput for complex AI services, and combined with other acceleration techniques it offers a flexible path to meet stringent performance requirements.