An Efficient and Integrated Algorithm for Video Enhancement in Challenging Lighting Conditions.

Publications:

Xuan Dong, Yi (Amy) Pang, Jiangtao (Gene) Wen, "Fast Efficient Algorithm for Enhancement of Low Lighting Video", SIGGRAPH 2010 Poster, Los Angeles, CA, July 27-29, 2010 (PDF).

Xuan Dong, Guan Wang, Yi (Amy) Pang, Weixin Li, Jiangtao(Gene) Wen, " Fast Efficient Algorithm for Enhancement of Low Lighting Video", the 2011 IEEE International Conference on Multimedia and Expo (ICME 2011), Barcelona, Spain, July 11-15, 2011 (PDF).

Xuan Dong, Jiangtao (Gene) Wen, Weixin Li, Yi (Amy) Pang, Guan Wang, Yao Lu, Wei Meng, "An Efficient and Integrated Algorithm for Video
Enhancement in Challenging Lighting Conditions", arxiv (PDF, MP4).

A demo can be seen below:

Introduction:

As video surveillance equipments and mobile devices such as digital cameras, smart phones and netbooks are increasingly widely deployed, cameras are expected to acquire, record and sometimes compress and transmit video content in all lighting and weather conditions. The majority of cameras, however, are not specifically designed to be all-purpose and weather-proof, rendering the video footage unusable for critical applications under many circumstances.

Image and video processing and enhancement including gamma correction, de-hazing, de-bluring and etc. are well-studied areas with many successful algorithms proposed over the years. Although different algorithms perform well for different lighting impairments, they often require tedious and sometimes manual input-dependent fine-tuning of algorithm parameters. In addition, different specific types of impairments often require different specific algorithms.

Take the enhancement of videos acquired under low lighting conditions as an example. To mitigate the problem, far and near infrared based techniques are used in many systems, and at the same time, various image processing based approaches have also been proposed. Although far and near infrared systems are useful for detecting objects such as pedestrians and animals in low lighting environments, especially in ``professional'' video surveillance systems, they suffer from the common disadvantage that detectable objects must have a temperature that is higher than their surroundings. In many cases where the critical object has a temperature similar to its surroundings, e.g. a big hole in the road, the infrared systems are not as helpful. Furthermore, infrared systems are usually more expensive, harder to maintain, with a relatively shorter life-span than conventional systems. They also introduce extra, and often times considerable power consumption. In many consumer applications such as video capture and communications on smart phones, it is usually not feasible to deploy infrared systems due to such cost and power consumption issues. Conventional low lighting image and video processing enhancement algorithms often work by reducing noise in the input low lighting video followed by contrast enhancement techniques such as tone-mapping, histogram stretching and equalization, and gamma correction to recover visual information in low lighting images and videos. Although these algorithms can lead to very visually pleasing enhancement results, they are usually too complicated for practical real-time applications, especially on mobile devices.

In this project, we describe a novel integrated video enhancement algorithm applicable to a wide range of input impairments. It has low computational and memory complexities that are both within the realm of reasonable availability of many mobile devices. In our system, a low complexity automatic module first determines the pre-dominate source of impairment in the input video. The input is then pre-processed based on the particular source of impairment, followed by processing by the core enhancement module. Finally, post-processing is applied to produce the enhanced output. In addition, spatial and temporal correlations were utilized to improve the speed of the algorithm and visual quality of the output, enabling it to be embedded into video encoders or decoders to share temporal and spatial prediction modules in the video codec to further lower complexity. The processing speed of our system can achieve real-time (more than 25 frames per second) on common PC while other popular algorithms are far more complicated, for example Bennett et al.'s [1] algorithm needs more than one minute to process each frame.

Enhancement examples:

Haze removal:

input enhancement result

Low lighting enhancement:

input enhancement result

High dynamic range:

input enhancement result

Reference:

[1] E. P. Bennett, L. McMillan. “Video Enhancement Using Per-pixel Virtual Exposures,” in Proc. SIGGRAPH ’05, Los Angeles, CA, Jul. 2005, pp. 845-852.