GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervision

We present a novel end-to-end framework named as GSNet (Geometric and Scene-aware Network), which jointly estimates 6DoF poses and reconstructs detailed 3D car shapes from single urban street view. GSNet utilizes a unique four-way feature extraction and fusion scheme and directly regresses 6DoF poses and shapes in a single forward pass... Extensive experiments show that our diverse feature extraction and fusion scheme can greatly improve model performance. Based on a divide-and-conquer 3D shape representation strategy, GSNet reconstructs 3D vehicle shape with great detail (1352 vertices and 2700 faces). This dense mesh representation further leads us to consider geometrical consistency and scene context, and inspires a new multi-objective loss function to regularize network training, which in turn improves the accuracy of 6D pose estimation and validates the merit of jointly performing both tasks. We evaluate GSNet on the largest multi-task ApolloCar3D benchmark and achieve state-of-the-art performance both quantitatively and qualitatively. Project page is available at https://lkeab.github.io/gsnet/. read more

PDF Abstract ECCV 2020 PDF ECCV 2020 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Vehicle Key-Point and Orientation Estimation ApolloCar3D GSNet A3DP 20.21 # 1
Pose Estimation ApolloCar3D GSNet A3DP 20.21 # 1
Keypoint Detection ApolloCar3D GSNet A3DP 20.21 # 1
3D Pose Estimation ApolloCar3D GSNet A3DP 20.21 # 1
3D Reconstruction ApolloCar3D GSNet A3DP 20.21 # 1
3D Shape Reconstruction From A Single 2D Image ApolloCar3D GSNet A3DP 20.21 # 1
6D Pose Estimation using RGB ApolloCar3D GSNet A3DP 20.21 # 1
6D Pose Estimation ApolloCar3D GSNet A3DP 20.21 # 1
Autonomous Driving ApolloCar3D GSNet A3DP 20.21 # 1
Autonomous Vehicles ApolloCar3D GSNet A3DP 20.21 # 1
Vehicle Pose Estimation ApolloCar3D GSNet A3DP 20.21 # 1
3D Shape Reconstruction ApolloCar3D GSNet A3DP 20.21 # 1
3D Car Instance Understanding ApolloCar3D GSNet A3DP 20.21 # 1

Methods


No methods listed for this paper. Add relevant methods here