Robust Video Content Alignment and Compensation for Clear Vision Through the Rain

Outdoor vision-based systems suffer from atmospheric turbulences, and rain is one of the worst factors for vision degradation. Current rain removal methods show limitations either for complex dynamic scenes, or under torrential rain with opaque occlusions. We propose a novel derain framework which applies superpixel (SP) segmentation to decompose the scene into depth consistent units. Alignment of scene contents are done at the SP level, which proves to be robust against rain occlusion interferences and fast camera motion. Two alignment output tensors, i.e., optimal temporal match tensor and sorted spatial-temporal match tensor, provide informative clues for the location of rain streaks and the occluded background contents. Different classical and novel methods such as Robust Principle Component Analysis and Convolutional Neural Networks are applied and compared for their respective advantages in efficiently exploiting the rich spatial-temporal features provided by the two tensors. Extensive evaluations show that advantage of up to 5dB is achieved on the scene restoration PSNR over state-of-the-art methods, and the advantage is especially obvious with highly complex and dynamic scenes. Visual evaluations show that the proposed framework is not only able to suppress heavy and opaque occluding rain streaks, but also large semi-transparent regional fluctuations and distortions.