Frustum PointNets for 3D Object Detection from RGB-D Data

CVPR 2018  ·  Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, Leonidas J. Guibas ·

In this work, we study 3D object detection from RGB-D data in both indoor and outdoor scenes. While previous methods focus on images or 3D voxels, often obscuring natural 3D patterns and invariances of 3D data, we directly operate on raw point clouds by popping up RGB-D scans. However, a key challenge of this approach is how to efficiently localize objects in point clouds of large-scale scenes (region proposal). Instead of solely relying on 3D proposals, our method leverages both mature 2D object detectors and advanced 3D deep learning for object localization, achieving efficiency as well as high recall for even small objects. Benefited from learning directly in raw point clouds, our method is also able to precisely estimate 3D bounding boxes even under strong occlusion or with very sparse points. Evaluated on KITTI and SUN RGB-D 3D detection benchmarks, our method outperforms the state of the art by remarkable margins while having real-time capability.

PDF Abstract CVPR 2018 PDF CVPR 2018 Abstract

Code
Add Remove Mark official

Datasets

KITTI SUN RGB-D
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Object Localization KITTI Cars Easy Frustum PointNets AP 88.7% # 2
Compare
3D Object Detection KITTI Cars Easy Frustum PointNets AP 81.2% # 21
Compare
3D Object Detection KITTI Cars Easy val F-PointNet [Qi:2018fd] AP 83.26 # 8
Compare
3D Object Detection KITTI Cars Hard Frustum PointNets AP 62.19% # 21
Compare
Object Detection KITTI Cars Hard F-PointNet AP 62.19 # 4
Compare
Object Localization KITTI Cars Hard Frustum PointNets AP 75.33% # 2
Compare
3D Object Detection KITTI Cars Hard val F-PointNet [Qi:2018fd] AP 62.56 # 8
Compare
Object Localization KITTI Cars Moderate Frustum PointNets AP 84.0% # 1
Compare
3D Object Detection KITTI Cars Moderate Frustum PointNets AP 70.39% # 28
Compare
3D Object Detection KITTI Cars Moderate val F-PointNet [Qi:2018fd] AP 69.28 # 9
Compare
3D Object Detection KITTI Cyclist Easy val F-PointNet [Qi:2018fd] AP 74.54 # 4
Compare
3D Object Detection KITTI Cyclist Easy val F-PointNet++ [Qi:2018fd] AP 77.15 # 3
Compare
3D Object Detection KITTI Cyclist Hard val F-PointNet++ [Qi:2018fd] AP 53.37 # 3
Compare
3D Object Detection KITTI Cyclist Hard val F-PointNet [Qi:2018fd] AP 52.65 # 4
Compare
3D Object Detection KITTI Cyclist Moderate val F-PointNet++ [Qi:2018fd] AP 56.49 # 3
Compare
3D Object Detection KITTI Cyclist Moderate val F-PointNet [Qi:2018fd] AP 55.95 # 4
Compare
Object Localization KITTI Cyclists Easy Frustum PointNets AP 75.38% # 1
Compare
3D Object Detection KITTI Cyclists Easy Frustum PointNets AP 71.96% # 9
Compare
Object Localization KITTI Cyclists Hard Frustum PointNets AP 54.68% # 1
Compare
3D Object Detection KITTI Cyclists Hard Frustum PointNets AP 50.39% # 9
Compare
Object Localization KITTI Cyclists Moderate Frustum PointNets AP 61.96% # 1
Compare
Birds Eye View Object Detection KITTI Cyclists Moderate F-PointNet AP 61.96% # 4
Compare
3D Object Detection KITTI Cyclists Moderate Frustum PointNets AP 56.77% # 9
Compare
3D Object Detection KITTI Pedestrian Easy val F-PointNet++ [Qi:2018fd] AP 70.00 # 2
Compare
3D Object Detection KITTI Pedestrian Easy val F-PointNet [Qi:2018fd] AP 65.08 # 4
Compare
3D Object Detection KITTI Pedestrian Hard val F-PointNet++ [Qi:2018fd] AP 53.59 # 3
Compare
3D Object Detection KITTI Pedestrian Hard val F-PointNet [Qi:2018fd] AP 49.28 # 4
Compare
3D Object Detection KITTI Pedestrian Moderate val F-PointNet [Qi:2018fd] AP 55.85 # 4
Compare
3D Object Detection KITTI Pedestrian Moderate val F-PointNet++ [Qi:2018fd] AP 61.32 # 2
Compare
Object Localization KITTI Pedestrians Easy Frustum PointNets AP 58.09% # 1
Compare
3D Object Detection KITTI Pedestrians Easy Frustum PointNets AP 51.21% # 6
Compare
Object Localization KITTI Pedestrians Hard Frustum PointNets AP 47.2% # 2
Compare
3D Object Detection KITTI Pedestrians Hard Frustum PointNets AP 40.23% # 6
Compare
Object Localization KITTI Pedestrians Moderate Frustum PointNets AP 50.22% # 2
Compare
Birds Eye View Object Detection KITTI Pedestrians Moderate F-PointNet AP 50.22% # 5
Compare
3D Object Detection KITTI Pedestrians Moderate Frustum PointNets AP 42.15% # 9
Compare
Object Detection In Indoor Scenes SUN RGB-D Frustum Pointnet (RGB) AP 0.5 56.8 # 2
Compare
3D Object Detection SUN-RGBD Frustum PointNets mAP@0.25 54.0 # 6
Compare
3D Object Detection SUN-RGBD val F-PointNet mAP@0.25 54.0 # 23
Compare
Inference Speed (s) 0.12 # 2
Compare

Methods
Add Remove

No methods listed for this paper. Add relevant methods here
Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy