no code implementations • 26 Apr 2024 • Shangzhan Zhang, Sida Peng, Tao Xu, Yuanbo Yang, Tianrun Chen, Nan Xue, Yujun Shen, Hujun Bao, Ruizhen Hu, Xiaowei Zhou
Instead of relying on extensive paired data, i. e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs.
no code implementations • 25 Apr 2024 • Junting Dong, Qi Fang, Zehuan Huang, Xudong Xu, Jingbo Wang, Sida Peng, Bo Dai
Previous works usually encode the human body and clothes as a holistic model and generate the whole model in a single-stage optimization, which makes them struggle for clothing editing and meanwhile lose fine-grained control over the whole generation process.
no code implementations • 17 Apr 2024 • Xi Chen, Sida Peng, Dongchen Yang, YuAn Liu, Bowen Pan, Chengfei Lv, Xiaowei Zhou
This paper aims to recover object materials from posed images captured under an unknown static lighting condition.
no code implementations • 5 Apr 2024 • Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, Xiaowei Zhou
Recovering dense and long-range pixel motion in videos is a challenging problem.
1 code implementation • 21 Mar 2024 • Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, Gordon Wetzstein
We introduce GRM, a large-scale reconstructor capable of recovering a 3D asset from sparse-view images in around 0. 1s.
no code implementations • 19 Mar 2024 • Xianglong He, Junyi Chen, Sida Peng, Di Huang, Yangguang Li, Xiaoshui Huang, Chun Yuan, Wanli Ouyang, Tong He
To simplify the generation of GaussianVolume and empower the model to generate instances with detailed 3D geometry, we propose a coarse-to-fine pipeline.
1 code implementation • 7 Mar 2024 • Yifan Wang, Xingyi He, Sida Peng, Dongli Tan, Xiaowei Zhou
Furthermore, we find spatial variance exists in LoFTR's fine correlation module, which is adverse to matching accuracy.
no code implementations • 19 Jan 2024 • Sida Peng, Wojciech Swiatek, Allen Gao, Paul Cullivan, Haoge Chang
In recent years, generative AI has undergone major advancements, demonstrating significant promise in augmenting human productivity.
no code implementations • 2 Jan 2024 • Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, Sida Peng
We introduce Street Gaussians, a new explicit scene representation that tackles all these limitations.
no code implementations • 14 Dec 2023 • Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xiaoshui Huang, Ding Liang, Wanli Ouyang
Recent advancements in text-to-3D generation technology have significantly advanced the conversion of textual descriptions into imaginative well-geometrical and finely textured 3D objects.
no code implementations • 13 Dec 2023 • Haoyu Guo, He Zhu, Sida Peng, Yuang Wang, Yujun Shen, Ruizhen Hu, Xiaowei Zhou
Experimental results on the ScanNet, ScanNet++ and KITTI-360 datasets demonstrate that our method achieves robust segmentation performance and can generalize across different types of scenes.
1 code implementation • 11 Dec 2023 • Zhen Xu, Tao Xie, Sida Peng, Haotong Lin, Qing Shuai, Zhiyuan Yu, Guangzhao He, Jiaming Sun, Hujun Bao, Xiaowei Zhou
Volumetric video is a technology that digitally records dynamic events such as artistic performances, sporting events, and remote conversations.
no code implementations • 17 Oct 2023 • Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, Xiaowei Zhou
Experiments show that our representation can be rendered at over 400 FPS on the DNA-Rendering dataset at 1080p resolution and 80 FPS on the ENeRF-Outdoor dataset at 4K resolution using an RTX 4090 GPU, which is 30x faster than previous methods and achieves the state-of-the-art rendering quality.
no code implementations • 12 Oct 2023 • Haotong Lin, Sida Peng, Zhen Xu, Tao Xie, Xingyi He, Hujun Bao, Xiaowei Zhou
This paper aims to tackle the challenge of dynamic view synthesis from multi-view videos.
no code implementations • ICCV 2023 • Huaijin Pi, Sida Peng, Minghui Yang, Xiaowei Zhou, Hujun Bao
This paper presents a novel approach to generating the 3D motion of a human interacting with a target object, with a focus on solving the challenge of synthesizing long-range and diverse motions, which could not be fulfilled by existing auto-regressive models or path planning-based methods.
no code implementations • 15 Aug 2023 • Zhen Xu, Sida Peng, Chen Geng, Linzhan Mou, Zihan Yan, Jiaming Sun, Hujun Bao, Xiaowei Zhou
Based on the HDQ algorithm, we leverage sphere tracing to efficiently estimate the surface intersection and light visibility.
no code implementations • 24 Jul 2023 • Shangzhan Zhang, Sida Peng, Yinji ShenTu, Qing Shuai, Tianrun Chen, Kaicheng Yu, Hujun Bao, Xiaowei Zhou
We extensively evaluate our approach on various scenes and show that our approach achieves spatially and temporally consistent editing results.
1 code implementation • 27 Jun 2023 • Xingyi He, Jiaming Sun, Yifan Wang, Sida Peng, QiXing Huang, Hujun Bao, Xiaowei Zhou
We propose a new detector-free SfM framework to draw benefits from the recent success of detector-free matchers to avoid the early determination of keypoints, while solving the multi-view inconsistency issue of detector-free matchers.
1 code implementation • CVPR 2023 • Haotong Lin, Qianqian Wang, Ruojin Cai, Sida Peng, Hadar Averbuch-Elor, Xiaowei Zhou, Noah Snavely
Specifically, we represent the scene as a space-time radiance field with a per-image illumination embedding, where temporally-varying scene changes are encoded using a set of learned step functions.
no code implementations • CVPR 2023 • Zehong Shen, Zhi Cen, Sida Peng, Qing Shuai, Hujun Bao, Xiaowei Zhou
We present a novel method for recovering the absolute pose and shape of a human in a pre-scanned scene given a single image.
no code implementations • CVPR 2023 • Yuang Wang, Xingyi He, Sida Peng, Haotong Lin, Hujun Bao, Xiaowei Zhou
A fully automated object reconstruction pipeline is crucial for digital content creation.
no code implementations • CVPR 2023 • Sida Peng, Yunzhi Yan, Qing Shuai, Hujun Bao, Xiaowei Zhou
This paper introduces a novel representation of volumetric videos for real-time view synthesis of dynamic scenes.
no code implementations • CVPR 2023 • Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, Dahua Lin
An alternative solution is to use a feature grid representation, which is computationally efficient and can naturally scale to a large scene with increased grid resolutions.
1 code implementation • CVPR 2023 • Chen Geng, Sida Peng, Zhen Xu, Hujun Bao, Xiaowei Zhou
In this paper, we propose a novel method for learning neural volumetric videos of dynamic humans from sparse view videos in minutes with competitive visual quality.
no code implementations • CVPR 2023 • Shangzhan Zhang, Sida Peng, Tianrun Chen, Linzhan Mou, Haotong Lin, Kaicheng Yu, Yiyi Liao, Xiaowei Zhou
We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single images from the Internet.
no code implementations • CVPR 2023 • Zifan Shi, Yujun Shen, Yinghao Xu, Sida Peng, Yiyi Liao, Sheng Guo, Qifeng Chen, Dit-yan Yeung
Existing methods for 3D-aware image synthesis largely depend on the 3D pose distribution pre-estimated on the training set.
no code implementations • ICCV 2023 • Junting Dong, Qi Fang, Tianshuo Yang, Qing Shuai, Chengyu Qiao, Sida Peng
However, these methods usually rely on limited multi-view images typically collected in the studio or commercial high-quality 3D scans for training, which heavily prohibits their generalization capability for in-the-wild images.
no code implementations • ICCV 2023 • Di Huang, Sida Peng, Tong He, Honghui Yang, Xiaowei Zhou, Wanli Ouyang
We propose a novel approach to self-supervised learning of point cloud representations by differentiable neural rendering.
no code implementations • CVPR 2023 • Yinghao Xu, Menglei Chai, Zifan Shi, Sida Peng, Ivan Skorokhodov, Aliaksandr Siarohin, Ceyuan Yang, Yujun Shen, Hsin-Ying Lee, Bolei Zhou, Sergey Tulyakov
Existing 3D-aware image synthesis approaches mainly focus on generating a single canonical object and show limited capacity in composing a complex scene containing a variety of objects.
1 code implementation • 27 Oct 2022 • Zifan Shi, Sida Peng, Yinghao Xu, Andreas Geiger, Yiyi Liao, Yujun Shen
In this survey, we thoroughly review the ongoing developments of 3D generative models, including methods that employ 2D and 3D supervision.
1 code implementation • CVPR 2022 • Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, Xiaowei Zhou
Based on the Manhattan-world assumption, planar constraints are employed to regularize the geometry in floor and wall regions predicted by a 2D semantic segmentation network.
1 code implementation • 22 Apr 2022 • YuAn Liu, Yilin Wen, Sida Peng, Cheng Lin, Xiaoxiao Long, Taku Komura, Wenping Wang
In this paper, we present a generalizable model-free 6-DoF object pose estimator called Gen6D.
no code implementations • 31 Mar 2022 • Xiangjun Gao, Jiaolong Yang, Jongyoo Kim, Sida Peng, Zicheng Liu, Xin Tong
For this task, we propose a simple yet effective method to train a generalizable NeRF with multiview images as conditional input.
1 code implementation • 15 Mar 2022 • Sida Peng, Zhen Xu, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Hujun Bao, Xiaowei Zhou
Some recent works have proposed to decompose a non-rigidly deforming scene into a canonical neural radiance field and a set of deformation fields that map observation-space points to the canonical space, thereby enabling them to learn the dynamic scene from images.
1 code implementation • CVPR 2022 • Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, Bolei Zhou
The feature field is further accumulated into a 2D feature map as the textural representation, followed by a neural renderer for appearance synthesis.
no code implementations • 2 Dec 2021 • Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, Xiaowei Zhou
We propose a novel scene representation, called ENeRF, for the fast creation of interactive free-viewpoint videos.
1 code implementation • CVPR 2022 • YuAn Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, Wenping Wang
On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction.
1 code implementation • ICCV 2021 • Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, Hujun Bao
Moreover, the learned blend weight fields can be combined with input skeletal motions to generate new deformation fields to animate the human model.
3 code implementations • CVPR 2021 • Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou
To this end, we propose Neural Body, a new human body representation which assumes that the learned neural representations at different frames share the same set of latent codes anchored to a deformable mesh, so that the observations across frames can be naturally integrated.
no code implementations • 7 Sep 2020 • Yang Ning, Sida Peng, Jing Tao
This paper proposes a doubly robust two-stage semiparametric difference-in-difference estimator for estimating heterogeneous treatment effects with high-dimensional data.
1 code implementation • CVPR 2020 • Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, Xiaowei Zhou
Based on deep snake, we develop a two-stage pipeline for instance segmentation: initial contour proposal and contour deformation, which can handle errors in object localization.
Ranked #2 on Semantic Contour Prediction on Sbd val
1 code implementation • NeurIPS 2019 • Yuan Liu, Zehong Shen, Zhixuan Lin, Sida Peng, Hujun Bao, Xiaowei Zhou
Instead of feature pooling, we use group convolutions to exploit underlying structures of the extracted features on the group, resulting in descriptors that are both discriminative and provably invariant to the group of transformations.
4 code implementations • CVPR 2019 • Sida Peng, Yu-An Liu, Qi-Xing Huang, Hujun Bao, Xiaowei Zhou
We further create a Truncation LINEMOD dataset to validate the robustness of our approach against truncation.
Ranked #2 on 6D Pose Estimation using RGB on YCB-Video (Mean AUC metric)
no code implementations • 20 Dec 2018 • Yang Ning, Sida Peng, Kosuke Imai
We first use a class of penalized M-estimators for the propensity score and outcome models.