ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation

19 Nov 2021  ·  Laurynas Karazija, Iro Laina, Christian Rupprecht ·

There has been a recent surge in methods that aim to decompose and segment scenes into multiple objects in an unsupervised manner, i.e., unsupervised multi-object segmentation. Performing such a task is a long-standing goal of computer vision, offering to unlock object-level reasoning without requiring dense annotations to train segmentation models. Despite significant progress, current models are developed and trained on visually simple scenes depicting mono-colored objects on plain backgrounds. The natural world, however, is visually complex with confounding aspects such as diverse textures and complicated lighting effects. In this study, we present a new benchmark called ClevrTex, designed as the next challenge to compare, evaluate and analyze algorithms. ClevrTex features synthetic scenes with diverse shapes, textures and photo-mapped materials, created using physically based rendering techniques. It includes 50k examples depicting 3-10 objects arranged on a background, created using a catalog of 60 materials, and a further test set featuring 10k images created using 25 different materials. We benchmark a large set of recent unsupervised multi-object segmentation models on ClevrTex and find all state-of-the-art approaches fail to learn good representations in the textured setting, despite impressive performance on simpler data. We also create variants of the ClevrTex dataset, controlling for different aspects of scene complexity, and probe current approaches for individual shortcomings. Dataset and code are available at https://www.robots.ox.ac.uk/~vgg/research/clevrtex.

PDF Abstract

Datasets


Introduced in the Paper:

ClevrTex

Used in the Paper:

MNIST CLEVR ShapeStacks
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Unsupervised Object Segmentation ClevrTex GNM mIoU 42.25± 0.18 # 3
MSE 383± 2 # 10
Unsupervised Object Segmentation ClevrTex SPAIR mIoU 0.0 ± 0.0 # 12
MSE 1101± 2 # 12
Unsupervised Object Segmentation ClevrTex SPACE mIoU 9.14± 3.46 # 10
MSE 298± 80 # 5
Unsupervised Object Segmentation ClevrTex MN mIoU 10.46± 0.10 # 9
MSE 335± 1 # 7
Unsupervised Object Segmentation ClevrTex DTI mIoU 33.79± 1.30 # 4
MSE 438± 22 # 11
Unsupervised Object Segmentation ClevrTex GenV2 mIoU 7.93± 1.53 # 11
MSE 315±106 # 6
Unsupervised Object Segmentation ClevrTex eMORL mIoU 30.17± 2.60 # 5
MSE 347± 20 # 9
Unsupervised Object Segmentation ClevrTex MONet mIoU 19.78± 1.02 # 8
MSE 146± 7 # 2
Unsupervised Object Segmentation ClevrTex SA mIoU 22.58± 2.07 # 7
MSE 254± 8 # 4
Unsupervised Object Segmentation ClevrTex IODINE mIoU 29.16± 0.75 # 6
MSE 340± 3 # 8

Methods


No methods listed for this paper. Add relevant methods here