How Constrained Spatial‑Temporal Modeling Boosts Low‑Light Video Calls in Real Time

Application Background

Most smartphones can record video, but limited lens size and cost result in low photon flux per pixel, especially in indoor or low‑light scenes, causing under‑exposed, dark video that degrades real‑time mobile video calls. Low‑lighting video enhancement modifies pixel values to improve visual quality and expand application scenarios.

Related Technologies

Existing low‑light video enhancement methods borrow from image enhancement techniques such as histogram equalization, contrast stretching, gamma correction, homomorphic filtering, tone mapping, and Retinex. These approaches either over‑enhance, amplify noise, lack adaptability, or are computationally too expensive for real‑time mobile video.

Design Motivation

Current video enhancement solutions mainly adapt single‑frame image methods, ignoring inter‑frame correlation, leading to inconsistent enhancement, flicker, and grain. Real‑time video chat also demands low computational and memory overhead, which many sophisticated algorithms cannot meet.

The desired algorithm should: (1) enhance low‑light video frames, (2) avoid flicker across consecutive frames, (3) refrain from over‑enhancing normally lit frames, and (4) meet real‑time mobile constraints.

Constrained Spatial‑Temporal Low‑Light Video Enhancement

The proposed method introduces spatial brightness and contrast constraints together with a temporal brightness consistency constraint, forming a convex optimization problem with a closed‑form solution. Processing is performed in YUV420 space, adjusting only the Y (luma) component while preserving UV chroma.

Brightness enhancement uses a family of learned functions F_I derived from paired low‑ and normal‑light video samples. To avoid contrast imbalance, an adaptive contrast enhancement function F_C is generated by excluding extreme d% of pixel values and applying a soft‑threshold adjustment.

Temporal consistency is enforced by constructing a cost function G that penalizes brightness differences between the current frame and the average of neighboring frames, suppressing flicker.

The overall optimization problem combines three quadratic terms and is solved via least‑squares, yielding an efficient solution suitable for real‑time execution.

Algorithm Contributions

Offline training of brightness enhancement functions and adaptive contrast functions to achieve balanced enhancement without over‑processing.

Integration of spatial and temporal constraints into a unified regularized optimization framework, meeting the four design goals.

Low computational complexity and strong robustness, demonstrated by extensive testing and online deployment without noticeable artifacts.

Experimental Results

Subjective comparisons on various scenes (Bonsai, Face, Street, Kimino) show that the proposed method produces brighter, flicker‑free video with minimal over‑enhancement compared to four reference methods.

Objective evaluation using MATLAB‑based runtime analysis shows the method achieves the best real‑time performance with minimal CPU cost.

NR‑CDIQA quality assessment indicates superior visual quality, especially on normally lit video where the method preserves original appearance.

References

[1] K. He, J. Sun, and X. Tang, “Single Image Haze Removal Using Dark Channel Prior,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1956‑1963, 2009.

[2] Y. Li, R. T. Tan, and M. S. Brown, “Nighttime Haze Removal with Glow and Multiple Light Colors,” Proc. IEEE Int. Conf. Computer Vision, pp. 226‑234, 2015.

[3] X. Dong et al., “Fast efficient algorithm for enhancement of low lighting video,” Proc. IEEE Int. Conf. Multimedia and Expo, 2011.

[4] W. Shi et al., “Group‑based sparse representation for low lighting image enhancement,” Proc. IEEE Int. Conf. Image Processing, pp. 4082‑4086, 2016.

[5] Y. Fang et al., “No‑Reference Quality Assessment of Contrast‑Distorted Images Based on Natural Scene Statistics,” IEEE Signal Processing Lett., vol. 22, no. 7, pp. 838‑842, 2015.

Conclusion

The paper introduces a regularized optimization framework for low‑light video sequences, incorporating brightness enhancement, contrast enhancement, and temporal consistency constraints, and provides a real‑time solution with low complexity and strong robustness. The method has been patented in China and presented at a major international video coding conference.