Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining

5 Feb 2023  ·  Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, Li Yi ·

Mainstream 3D representation learning approaches are built upon contrastive or generative modeling pretext tasks, where great improvements in performance on various downstream tasks have been achieved. However, we find these two paradigms have different characteristics: (i) contrastive models are data-hungry that suffer from a representation over-fitting issue; (ii) generative models have a data filling issue that shows inferior data scaling capacity compared to contrastive models. This motivates us to learn 3D representations by sharing the merits of both paradigms, which is non-trivial due to the pattern difference between the two paradigms. In this paper, we propose Contrast with Reconstruct (ReCon) that unifies these two paradigms. ReCon is trained to learn from both generative modeling teachers and single/cross-modal contrastive teachers through ensemble distillation, where the generative student guides the contrastive student. An encoder-decoder style ReCon-block is proposed that transfers knowledge through cross attention with stop-gradient, which avoids pretraining over-fitting and pattern difference issues. ReCon achieves a new state-of-the-art in 3D representation learning, e.g., 91.26% accuracy on ScanObjectNN. Codes have been released at https://github.com/qizekun/ReCon.

PDF Abstract

Results from the Paper


 Ranked #1 on 3D Point Cloud Linear Classification on ModelNet40 (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Zero-Shot Transfer 3D Point Cloud Classification ModelNet10 ReCon Accuracy (%) 75.6 # 1
3D Point Cloud Classification ModelNet40 ReCon Overall Accuracy 94.7 # 5
3D Point Cloud Linear Classification ModelNet40 ReCon Overall Accuracy 93.4 # 1
Zero-Shot Transfer 3D Point Cloud Classification ModelNet40 ReCon Accuracy (%) 61.7 # 4
Few-Shot 3D Point Cloud Classification ModelNet40 10-way (10-shot) ReCon Overall Accuracy 93.3 # 3
Standard Deviation 3.9 # 8
Few-Shot 3D Point Cloud Classification ModelNet40 10-way (20-shot) ReCon Overall Accuracy 95.8 # 2
Standard Deviation 3.0 # 10
Few-Shot 3D Point Cloud Classification ModelNet40 5-way (10-shot) ReCon Overall Accuracy 97.3 # 2
Standard Deviation 1.9 # 5
Few-Shot 3D Point Cloud Classification ModelNet40 5-way (20-shot) ReCon Overall Accuracy 98.9 # 2
Standard Deviation 1.2 # 3
3D Point Cloud Classification ScanObjectNN ReCon Overall Accuracy 91.26 # 3
OBJ-BG (OA) 95.35 # 2
OBJ-ONLY (OA) 93.80 # 2
Zero-Shot Transfer 3D Point Cloud Classification ScanObjectNN ReCon PB_T50_RS Accuracy (%) 30.5 # 2
OBJ_BG Accuracy(%) 40.4 # 2
OBJ_ONLY Accuracy(%) 43.7 # 2
3D Point Cloud Classification ScanObjectNN ReCon (no voting) Overall Accuracy 90.63 # 5
OBJ-BG (OA) 95.18 # 3
OBJ-ONLY (OA) 93.29 # 3

Methods


No methods listed for this paper. Add relevant methods here