Learning Ordinal Relationships for Mid-Level Vision

We propose a framework that infers mid-level visual properties of an image by learning about ordinal relation- ships. Instead of estimating metric quantities directly, the system proposes pairwise relationship estimates for points in the input image. These sparse probabilistic ordinal mea- surements are globalized to create a dense output map of continuous metric measurements. Estimating order rela- tionships between pairs of points has several advantages over metric estimation: it solves a simpler problem than metric regression; humans are better at relative judgements, so data collection is easier; ordinal relationships are invari- ant to monotonic transformations of the data, thereby in- creasing the robustness of the system and providing qualitatively different information. We demonstrate that this frame- work works well on two important mid-level vision tasks: intrinsic image decomposition and depth from an RGB im- age. We train two systems with the same architecture on data from these two modalities. We provide an analysis of the resulting models, showing that they learn a number of simple rules to make ordinal decisions. We apply our algo-rithm to depth estimation, with good results, and intrinsic image decomposition, with state-of-the-art results.