Sonata: Self-Supervised Learning of Reliable Point Representations

In this paper, we question whether we have a reliable self-supervised point cloud model that can be used for diverse 3D tasks via simple linear probing, even with limited data and minimal computation. We find that existing 3D self-supervised learning approaches fall short when evaluated on representation quality through linear probing. We hypothesize that this is due to what we term the "geometric shortcut", which causes representations to collapse to low-level spatial features. This challenge is unique to 3D and arises from the sparse nature of point cloud data. We address it through two key strategies: obscuring spatial information and enhancing the reliance on input features, ultimately composing a Sonata of 140k point clouds through self-distillation. Sonata is simple and intuitive, yet its learned representations are strong and reliable: zero-shot visualizations demonstrate semantic grouping, alongside strong spatial reasoning through nearest-neighbor relationships. Sonata demonstrates exceptional parameter and data efficiency, tripling linear probing accuracy (from 21.8% to 72.5%) on ScanNet and nearly doubling performance with only 1% of the data compared to previous approaches. Full fine-tuning further advances SOTA across both 3D indoor and outdoor perception tasks.

PDF Abstract

Results from the Paper


 Ranked #1 on Semantic Segmentation on S3DIS (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Semantic Segmentation S3DIS Sonata + PTv3 Mean IoU 82.3 # 1
mAcc 89.9 # 1
oAcc 93.3 # 1
Number of params 128M # 54
Semantic Segmentation S3DIS Area5 Sonata + PTv3 mIoU 76.0 # 1
oAcc 93.0 # 1
mAcc 81.6 # 1
Semantic Segmentation ScanNet Sonata + PTv3 val mIoU 79.4 # 2
3D Semantic Segmentation ScanNet200 Sonata + PTv3 val mIoU 36.8 # 5

Methods


No methods listed for this paper. Add relevant methods here