Large Displacement 3D Scene Flow With Occlusion Reasoning

3D motion estimation is a fundamental problem with many computer vision applications. With the emergence of modern, affordable and increasingly accurate RGB-D sensors, single view approaches for estimating 3D motion, also known as scene flow, are becoming popular. In this paper we propose a novel coarse to fine correspondence-based scene flow approach to account for the effects of large displacements and to model occlusion, based on explicit geometric reasoning. Our methodology enforces piecewise motion rigidity at the level of the depth point cloud without explicitly smoothing the parameters of adjacent neighborhoods. By integrating all geometric and photometric components in a single, consistent, occlusion-aware energy model our method is able to deal with fast motions and large occlusions areas, as present in challenging datasets like MPI Sintel Flow Dataset, which have recently been augmented with depth information. By explicitly modeling large displacements and occlusion, we can now more successfully work with difficult sequences which cannot be currently processed by state of the art scene flow methods that rely on small inter-frame motion assumptions. We also show that by leveraging depth information, we can obtain superior correspondence fields compared to the best state of the art large-displacement (2D) optical flow methods.