How Baidu App Achieves Real‑Time Mobile Super‑Resolution with Deep Learning

Background

With the proliferation of mobile devices, content creation and consumption on smartphones have become ubiquitous. Baidu App, a major content distribution platform, serves massive amounts of PGC and UGC images and videos. As 2K screen resolutions become mainstream, users expect high‑definition media, but many resources suffer from low resolution due to capture, transmission, and storage constraints, degrading the viewing experience.

Improving Resolution

Resolution measures the pixel density of an image on the imaging plane and directly influences perceived detail. Higher resolution yields richer detail and better visual quality, though it also increases file size.

Traditional up‑sampling methods such as interpolation often introduce artifacts like mosaicking, jagged edges, and blurring. Recent advances in deep learning, especially convolutional neural networks (CNNs) that mimic human visual perception, enable more accurate reconstruction by learning image features.

Baidu App Super‑Resolution Model

The deployed model builds on the VDSR (Very Deep Super‑Resolution) residual learning framework. To meet mobile constraints, the network is pruned and incorporates Depthwise Separable Convolutions, reducing computational load. The model accepts a Y‑channel image that has already been up‑sampled to the target resolution and supports variable input sizes.

Challenges of Real‑Time Mobile Super‑Resolution

Running deep‑learning super‑resolution on mobile devices faces strict latency, memory, and power budgets. Maintaining high frame rates while preserving visual fidelity requires careful algorithmic and system‑level optimizations.

Strategies and Optimizations

Application‑layer optimizations:

Memory management for large images: split images into tiles and process them in parallel queues, dynamically limiting peak memory usage.

Video super‑resolution stability: a strategy module provides two modes—maximum quality and safe‑frame‑rate—to ensure smooth playback.

Compute resource scheduling: offload CPU‑bound pre‑ and post‑processing to GPU operators, consolidating the entire pipeline on the GPU.

Inference‑engine optimizations:

Prediction latency reduced to less than 50% of the original time; on iOS, CoreML inference time is cut to one‑quarter.

Achieved 25 ms per 480p frame on iPhone XR and 23 ms on Snapdragon 845 Android devices.

GPU memory consumption for image and video super‑resolution lowered to under 50% of the baseline.

Business Impact and Results

Both image and video super‑resolution have been deployed across multiple Baidu mobile products. Tens of millions of images and videos are processed daily on‑device, delivering enhanced visual quality without server‑side intervention, thereby reducing server compute, storage, and bandwidth costs.

End‑to‑End Integration Example

The following code snippets illustrate how to invoke the super‑resolution API on iOS and Android.

// iOS
/**
 * Execute super‑resolution on an image.
 * @param image   Image to be processed.
 * @param scaleType   Desired up‑sampling factor.
 * @param block   Callback with result or error.
 */
- (void)executeSuperResolutionWithImage:(UIImage *)aImage
                                   scale:(MMLImageSuperResolutionScaleType)scaleType
                               completion:(void (^)(UIImage *srImage, NSError *error))block API_AVAILABLE(ios(9.0));

// Android
/**
 * Execute image super‑resolution.
 * @param inputBitmap   Bitmap to be processed.
 * @param scale         SR factor.
 * @param onSrResultListener   Callback for the result.
 */
void sr(Bitmap inputBitmap, float scale, OnSrResultListener onSrResultListener);

References

https://en.wikipedia.org/wiki/Image_resolution

https://arxiv.org/abs/1511.04587