MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World

no code implementations • 16 Jan 2024 • Yining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, Chuang Gan

Human beings possess the capability to multiply a melange of multisensory cues while actively exploring and interacting with the 3D world.

Language Modelling Large Language Model

Disentangling Orthogonal Planes for Indoor Panoramic Room Layout Estimation with Cross-Scale Distortion Awareness

1 code implementation • CVPR 2023 • Zhijie Shen, Zishuo Zheng, Chunyu Lin, Lang Nie, Kang Liao, Shuai Zheng, Yao Zhao

Based on the Manhattan World assumption, most existing indoor layout estimation schemes focus on recovering layouts from vertically compressed 1D sequences.

Room Layout Estimation Segmentation

RecRecNet: Rectangling Rectified Wide-Angle Images by Thin-Plate Spline Model and DoF-based Curriculum Learning

1 code implementation • ICCV 2023 • Kang Liao, Lang Nie, Chunyu Lin, Zishuo Zheng, Yao Zhao

In this work, we explore constructing a win-win representation on both content and boundary by contributing a new learning model, i. e., Rectangling Rectification Network (RecRecNet).

Complementary Bi-directional Feature Compression for Indoor 360° Semantic Segmentation with Self-distillation

no code implementations • 6 Jul 2022 • Zishuo Zheng, Chunyu Lin, Lang Nie, Kang Liao, Zhijie Shen, Yao Zhao

In this paper, we combine the two different representations and propose a novel 360{\deg} semantic segmentation solution from a complementary perspective.

Feature Compression Semantic Segmentation

PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation

1 code implementation • 17 Mar 2022 • Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, Yao Zhao

In particular, we divide patches on the spherical tangent domain into tokens to reduce the negative effect of panoramic distortions.

Depth Estimation Semantic Segmentation