Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation

Monocular depth estimation is a fundamental computer vision task. Recovering 3D depth from a single image is geometrically ill-posed and requires scene understanding, so it is not surprising that the rise of deep learning has led to a breakthrough. The impressive progress of monocular depth estimators has mirrored the growth in model capacity, from relatively modest CNNs to large Transformer architectures. Still, monocular depth estimators tend to struggle when presented with images with unfamiliar content and layout, since their knowledge of the visual world is restricted by the data seen during training, and challenged by zero-shot generalization to new domains. This motivates us to explore whether the extensive priors captured in recent generative diffusion models can enable better, more generalizable depth estimation. We introduce Marigold, a method for affine-invariant monocular depth estimation that is derived from Stable Diffusion and retains its rich prior knowledge. The estimator can be fine-tuned in a couple of days on a single GPU using only synthetic training data. It delivers state-of-the-art performance across a wide range of datasets, including over 20% performance gains in specific cases. Project page:

PDF Abstract

Results from the Paper

Ranked #3 on Monocular Depth Estimation on NYU-Depth V2 (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Monocular Depth Estimation KITTI Eigen split Marigold absolute relative error 0.099 # 44
RMSE 3.304 # 33
RMSE log 0.138 # 29
Delta < 1.25 0.916 # 30
Delta < 1.25^2 0.987 # 28
Delta < 1.25^3 0.996 # 29
Monocular Depth Estimation NYU-Depth V2 Marigold RMSE 0.224 # 3
absolute relative error 0.055 # 1
Delta < 1.25 0.964 # 6
Delta < 1.25^2 0.991 # 17
Delta < 1.25^3 0.998 # 14
log 10 0.024 # 1