Search Results for author: Lan Xu

Found 55 papers, 15 papers with code

RobustFusion: Human Volumetric Capture with Data-driven Visual Cues using a RGBD Camera

no code implementations ECCV 2020 Zhuo Su, Lan Xu, Zerong Zheng, Tao Yu, Yebin Liu, Lu Fang

To enable robust tracking, we embrace both the initial model and the various visual cues into a novel performance capture scheme with hybrid motion optimization and semantic volumetric fusion, which can successfully capture challenging human motions under the monocular setting without pre-scanned detailed template and owns the reinitialization ability to recover from tracking failures and the disappear-reoccur scenarios.

4D reconstruction

HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting

no code implementations6 Dec 2023 Yuheng Jiang, Zhehao Shen, Penghao Wang, Zhuo Su, Yu Hong, Yingliang Zhang, Jingyi Yu, Lan Xu

Then, we utilize a 4D Gaussian optimization scheme with adaptive spatial-temporal regularizers to effectively balance the non-rigid prior and Gaussian updating.

VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams

no code implementations3 Dec 2023 Liao Wang, Kaixin Yao, Chengcheng Guo, Zhirui Zhang, Qiang Hu, Jingyi Yu, Lan Xu, Minye Wu

In this paper, we introduce VideoRF, the first approach to enable real-time streaming and rendering of dynamic radiance fields on mobile platforms.

Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator

1 code implementation NeurIPS 2023 Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, Sibei Yang

Text-to-video is a rapidly growing research area that aims to generate a semantic, identical, and temporal coherence sequence of frames that accurately align with the input text prompt.

Text-to-Video Generation Video Generation +1

InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions

1 code implementation12 Apr 2023 Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, Lan Xu

Then, we propose a novel representation for motion input in our interaction diffusion model, which explicitly formulates the global relations between the two performers in the world frame.


Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos

no code implementations CVPR 2023 Liao Wang, Qiang Hu, Qihan He, Ziyu Wang, Jingyi Yu, Tinne Tuytelaars, Lan Xu, Minye Wu

The success of the Neural Radiance Fields (NeRFs) for modeling and free-view rendering static objects has inspired numerous attempts on dynamic scenes.

Neural Rendering

CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions

no code implementations CVPR 2023 Ming Yan, Xin Wang, Yudi Dai, Siqi Shen, Chenglu Wen, Lan Xu, Yuexin Ma, Cheng Wang

The core of this dataset is a blending optimization process, which corrects for the pose as it drifts and is affected by the magnetic conditions.

Pose Prediction

CryoFormer: Continuous Heterogeneous Cryo-EM Reconstruction using Transformer-based Neural Representations

no code implementations28 Mar 2023 Xinhang Liu, Yan Zeng, Yifan Qin, Hao Li, Jiakai Zhang, Lan Xu, Jingyi Yu

Cryo-electron microscopy (cryo-EM) allows for the high-resolution reconstruction of 3D structures of proteins and other biomolecules.

SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environments

1 code implementation CVPR 2023 Yudi Dai, Yitai Lin, Xiping Lin, Chenglu Wen, Lan Xu, Hongwei Yi, Siqi Shen, Yuexin Ma, Cheng Wang

We present SLOPER4D, a novel scene-aware dataset collected in large urban environments to facilitate the research of global human pose estimation (GHPE) with human-scene interaction in the wild.

3D Human Pose Estimation Camera Calibration +1

NEPHELE: A Neural Platform for Highly Realistic Cloud Radiance Rendering

no code implementations7 Mar 2023 Haimin Luo, Siyuan Zhang, Fuqiang Zhao, Haotian Jing, Penghao Wang, Zhenxiao Yu, Dongxue Yan, Junran Ding, Boyuan Zhang, Qiang Hu, Shu Yin, Lan Xu, Jingyi Yu

Using such a cloud platform compatible with neural rendering, we further showcase the capabilities of our cloud radiance rendering through a series of applications, ranging from cloud VR/AR rendering.

Neural Rendering

IKOL: Inverse kinematics optimization layer for 3D human pose and shape estimation via Gauss-Newton differentiation

1 code implementation2 Feb 2023 Juze Zhang, Ye Shi, Yuexin Ma, Lan Xu, Jingyi Yu, Jingya Wang

This paper presents an inverse kinematic optimization layer (IKOL) for 3D human pose and shape estimation that leverages the strength of both optimization- and regression-based methods within an end-to-end framework.

3D human pose and shape estimation regression

NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions

no code implementations CVPR 2023 Juze Zhang, Haimin Luo, Hongdi Yang, Xinru Xu, Qianyang Wu, Ye Shi, Jingyi Yu, Lan Xu, Jingya Wang

We construct a dense multi-view dome to acquire a complex human object interaction dataset, named HODome, that consists of $\sim$75M frames on 10 subjects interacting with 23 objects.

Human-Object Interaction Detection

Relightable Neural Human Assets from Multi-view Gradient Illuminations

no code implementations CVPR 2023 Taotao Zhou, Kai He, Di wu, Teng Xu, Qixuan Zhang, Kuixiang Shao, Wenzheng Chen, Lan Xu, Jingyi Yu

UltraStage will be publicly available to the community to stimulate significant future developments in various human modeling and rendering tasks.

Image Relighting Novel View Synthesis

HumanGen: Generating Human Radiance Fields with Explicit Priors

no code implementations CVPR 2023 Suyi Jiang, Haoran Jiang, Ziyu Wang, Haimin Luo, Wenzheng Chen, Lan Xu

With the aid of the anchor image, we adapt a 3D reconstructor for fine-grained details synthesis and propose a two-stage blending scheme to boost appearance generation.

Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDAR

no code implementations30 Nov 2022 Peishan Cong, Yiteng Xu, Yiming Ren, Juze Zhang, Lan Xu, Jingya Wang, Jingyi Yu, Yuexin Ma

Motivated by this, we propose a monocular camera and single LiDAR-based method for 3D multi-person pose estimation in large-scale scenes, which is easy to deploy and insensitive to light.

3D Multi-Person Pose Estimation 3D Pose Estimation +2

LiCamGait: Gait Recognition in the Wild by Using LiDAR and Camera Multi-modal Visual Sensors

no code implementations22 Nov 2022 Xiao Han, Peishan Cong, Lan Xu, Jingya Wang, Jingyi Yu, Yuexin Ma

LiDAR can capture accurate depth information in large-scale scenarios without the effect of light conditions, and the captured point cloud contains gait-related 3D geometric properties and dynamic motion characteristics.

Gait Recognition in the Wild

Learning Variational Motion Prior for Video-based Motion Capture

no code implementations27 Oct 2022 Xin Chen, Zhuo Su, Lingbo Yang, Pei Cheng, Lan Xu, Bin Fu, Gang Yu

To improve the generalization capacity of prior space, we propose a transformer-based variational autoencoder pretrained over marker-based 3D mocap data, with a novel style-mapping block to boost the generation quality.

Pose Estimation

Human Performance Modeling and Rendering via Neural Animated Mesh

1 code implementation18 Sep 2022 Fuqiang Zhao, Yuheng Jiang, Kaixin Yao, Jiakai Zhang, Liao Wang, Haizhao Dai, Yuhui Zhong, Yingliang Zhang, Minye Wu, Lan Xu, Jingyi Yu

In this paper, we present a comprehensive neural approach for high-quality reconstruction, compression, and rendering of human performances from dense multi-view videos.

SCULPTOR: Skeleton-Consistent Face Creation Using a Learned Parametric Generator

no code implementations14 Sep 2022 Zesong Qiu, Yuwei Li, Dongming He, Qixuan Zhang, Longwen Zhang, Yinghao Zhang, Jingya Wang, Lan Xu, Xudong Wang, Yuyao Zhang, Jingyi Yu

Named after the fossils of one of the oldest known human ancestors, our LUCY dataset contains high-quality Computed Tomography (CT) scans of the complete human head before and after orthognathic surgeries, critical for evaluating surgery results.

Computed Tomography (CT)

Mutual Adaptive Reasoning for Monocular 3D Multi-Person Pose Estimation

no code implementations16 Jul 2022 Juze Zhang, Jingya Wang, Ye Shi, Fei Gao, Lan Xu, Jingyi Yu

This method first uses 2. 5D pose and geometry information to infer camera-centric root depths in a forward pass, and then exploits the root depths to further improve representation learning of 2. 5D pose estimation in a backward pass.

3D Multi-Person Pose Estimation Depth Estimation +2

NARRATE: A Normal Assisted Free-View Portrait Stylizer

no code implementations3 Jul 2022 Youjia Wang, Teng Xu, Yiwen Wu, Minzhang Li, Wenzheng Chen, Lan Xu, Jingyi Yu

We extend Total Relighting to fix this problem by unifying its multi-view input normal maps with the physical face model.

Face Model Neural Rendering +1

LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors

no code implementations30 May 2022 Yiming Ren, Chengfeng Zhao, Yannan He, Peishan Cong, Han Liang, Jingyi Yu, Lan Xu, Yuexin Ma

We propose a multi-sensor fusion method for capturing challenging 3D human motions with accurate consecutive local poses and global trajectories in large-scale scenarios, only using single LiDAR and 4 IMUs, which are set up conveniently and worn lightly.

Sensor Fusion Translation

STCrowd: A Multimodal Dataset for Pedestrian Perception in Crowded Scenes

1 code implementation CVPR 2022 Peishan Cong, Xinge Zhu, Feng Qiao, Yiming Ren, Xidong Peng, Yuenan Hou, Lan Xu, Ruigang Yang, Dinesh Manocha, Yuexin Ma

In addition, considering the property of sparse global distribution and density-varying local distribution of pedestrians, we further propose a novel method, Density-aware Hierarchical heatmap Aggregation (DHA), to enhance pedestrian perception in crowded scenes.

Pedestrian Detection Sensor Fusion

HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR

1 code implementation CVPR 2022 Yudi Dai, Yitai Lin, Chenglu Wen, Siqi Shen, Lan Xu, Jingyi Yu, Yuexin Ma, Cheng Wang

We propose Human-centered 4D Scene Capture (HSC4D) to accurately and efficiently create a dynamic digital world, containing large-scale indoor-outdoor scenes, diverse human motions, and rich interactions between humans and environments.

3D Human Pose Estimation Autonomous Driving

NeReF: Neural Refractive Field for Fluid Surface Reconstruction and Implicit Representation

no code implementations8 Mar 2022 Ziyu Wang, Wei Yang, Junming Cao, Lan Xu, Junqing Yu, Jingyi Yu

We present a novel neural refractive field(NeReF) to recover wavefront of transparent fluids by simultaneously estimating the surface position and normal of the fluid front.

Surface Reconstruction

Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time

no code implementations CVPR 2022 Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Lan Xu, Jingyi Yu

In this paper, we present a novel Fourier PlenOctree (FPO) technique to tackle efficient neural modeling and real-time rendering of dynamic scenes captured under the free-view video (FVV) setting.

NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing

no code implementations12 Feb 2022 Jiakai Zhang, Liao Wang, Xinhang Liu, Fuqiang Zhao, Minzhang Li, Haizhao Dai, Boyuan Zhang, Wei Yang, Lan Xu, Jingyi Yu

We further develop a hybrid neural-rasterization rendering framework to support consumer-level VR headsets so that the aforementioned volumetric video viewing and editing, for the first time, can be conducted immersively in virtual 3D space.

3D Reconstruction

Video-driven Neural Physically-based Facial Asset for Production

no code implementations11 Feb 2022 Longwen Zhang, Chuxiao Zeng, Qixuan Zhang, Hongyang Lin, Ruixiang Cao, Wei Yang, Lan Xu, Jingyi Yu

In this paper, we present a new learning-based, video-driven approach for generating dynamic facial geometries with high-quality physically-based assets.

motion retargeting Texture Synthesis

Artemis: Articulated Neural Pets with Appearance and Motion synthesis

1 code implementation11 Feb 2022 Haimin Luo, Teng Xu, Yuheng Jiang, Chenglin Zhou, QIwei Qiu, Yingliang Zhang, Wei Yang, Lan Xu, Jingyi Yu

Our ARTEMIS enables interactive motion control, real-time animation, and photo-realistic rendering of furry animals.

Motion Synthesis

NIMBLE: A Non-rigid Hand Model with Bones and Muscles

no code implementations9 Feb 2022 Yuwei Li, Longwen Zhang, Zesong Qiu, Yingwenqi Jiang, Nianyi Li, Yuexin Ma, Yuyao Zhang, Lan Xu, Jingyi Yu

Emerging Metaverse applications demand reliable, accurate, and photorealistic reproductions of human hands to perform sophisticated operations as if in the physical world.

HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs

no code implementations CVPR 2022 Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, Lan Xu

The raw HumanNeRF can already produce reasonable rendering on sparse video inputs of unseen subjects and camera settings.

iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering

no code implementations12 Aug 2021 Liao Wang, Ziyu Wang, Pei Lin, Yuheng Jiang, Xin Suo, Minye Wu, Lan Xu, Jingyi Yu

To fill this gap, in this paper we propose a neural interactive bullet-time generator (iButter) for photo-realistic human free-viewpoint rendering from dense RGB streams, which enables flexible and interactive design for human bullet-time visual effects.

Video Generation

Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

no code implementations1 Aug 2021 Guoxing Sun, Xin Chen, Yizhang Chen, Anqi Pang, Pei Lin, Yuheng Jiang, Lan Xu, Jingya Wang, Jingyi Yu

In this paper, we propose a neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of both human and objects under challenging interaction scenarios in arbitrary novel views, from only sparse RGB streams.

4D reconstruction Dynamic Reconstruction +4

Neural Relighting and Expression Transfer On Video Portraits

no code implementations30 Jul 2021 Youjia Wang, Taotao Zhou, Minzhang Li, Teng Xu, Minye Wu, Lan Xu, Jingyi Yu

We present a neural relighting and expression transfer technique to transfer the facial expressions from a source performer to a portrait video of a target performer while enabling dynamic relighting.

Multi-Task Learning Neural Rendering

Few-shot Neural Human Performance Rendering from Sparse RGBD Videos

no code implementations14 Jul 2021 Anqi Pang, Xin Chen, Haimin Luo, Minye Wu, Jingyi Yu, Lan Xu

To fill this gap, in this paper we propose a few-shot neural human rendering approach (FNHR) from only sparse RGBD inputs, which exploits the temporal and spatial redundancy to generate photo-realistic free-view output of human activities.

Neural Rendering

IREM: High-Resolution Magnetic Resonance (MR) Image Reconstruction via Implicit Neural Representation

no code implementations29 Jun 2021 Qing Wu, Yuwei Li, Lan Xu, Ruiming Feng, Hongjiang Wei, Qing Yang, Boliang Yu, Xiaozhao Liu, Jingyi Yu, Yuyao Zhang

For collecting high-quality high-resolution (HR) MR image, we propose a novel image reconstruction network named IREM, which is trained on multiple low-resolution (LR) MR images and achieve an arbitrary up-sampling rate for HR image reconstruction.

Anatomy Image Reconstruction +1

PIANO: A Parametric Hand Bone Model from Magnetic Resonance Imaging

1 code implementation21 Jun 2021 Yuwei Li, Minye Wu, Yuyao Zhang, Lan Xu, Jingyi Yu

Hand modeling is critical for immersive VR/AR, action understanding, or human healthcare.

Action Understanding

RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream

no code implementations30 Apr 2021 Zhuo Su, Lan Xu, Dawei Zhong, Zhong Li, Fan Deng, Shuxue Quan, Lu Fang

To fill this gap, in this paper, we propose RobustFusion, a robust volumetric performance reconstruction system for human-object interaction scenarios using only a single RGBD sensor, which combines various data-driven visual and interaction cues to handle the complex interaction patterns and severe occlusions.

4D reconstruction Disentanglement +4

Editable Free-viewpoint Video Using a Layered Neural Representation

1 code implementation30 Apr 2021 Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, Jingyi Yu

Such layered representation supports fully perception and realistic manipulation of the dynamic scene whilst still supporting a free viewing experience in a wide range.

Disentanglement Scene Parsing +1

SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos

1 code implementation23 Apr 2021 Xin Chen, Anqi Pang, Wei Yang, Yuexin Ma, Lan Xu, Jingyi Yu

In this paper, we propose SportsCap -- the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input.

Action Assessment Markerless Motion Capture

MirrorNeRF: One-shot Neural Portrait Radiance Field from Multi-mirror Catadioptric Imaging

no code implementations6 Apr 2021 Ziyu Wang, Liao Wang, Fuqiang Zhao, Minye Wu, Lan Xu, Jingyi Yu

In this paper, we propose MirrorNeRF - a one-shot neural portrait free-viewpoint rendering approach using a catadioptric imaging system with multiple sphere mirrors and a single high-resolution digital camera, which is the first to combine neural radiance field with catadioptric imaging so as to enable one-shot photo-realistic human portrait reconstruction and rendering, in a low-cost and casual capture setting.

Convolutional Neural Opacity Radiance Fields

1 code implementation5 Apr 2021 Haimin Luo, Anpei Chen, Qixuan Zhang, Bai Pang, Minye Wu, Lan Xu, Jingyi Yu

In this paper, we propose a novel scheme to generate opacity radiance fields with a convolutional neural renderer for fuzzy objects, which is the first to combine both explicit opacity supervision and convolutional mechanism into the neural radiance field framework so as to enable high-quality appearance and global consistent alpha mattes generation in arbitrary novel views.

Neural Video Portrait Relighting in Real-time via Consistency Modeling

1 code implementation ICCV 2021 Longwen Zhang, Qixuan Zhang, Minye Wu, Jingyi Yu, Lan Xu

In this paper, we propose a neural approach for real-time, high-quality and coherent video portrait relighting, which jointly models the semantic, temporal and lighting consistency using a new dynamic OLAT dataset.

Disentanglement Single-Image Portrait Relighting

GNeRF: GAN-based Neural Radiance Field without Posed Camera

1 code implementation ICCV 2021 Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, Jingyi Yu

We introduce GNeRF, a framework to marry Generative Adversarial Networks (GAN) with Neural Radiance Field (NeRF) reconstruction for the complex scenarios with unknown and even randomly initialized camera poses.

Novel View Synthesis

NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras

no code implementations CVPR 2021 Xin Suo, Yuheng Jiang, Pei Lin, Yingliang Zhang, Kaiwen Guo, Minye Wu, Lan Xu

4D reconstruction and rendering of human activities is critical for immersive VR/AR experience. Recent advances still fail to recover fine geometry and texture results with the level of detail present in the input images from sparse multi-view RGB cameras.

4D reconstruction Multi-Task Learning

ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References

2 code implementations CVPR 2021 Yannan He, Anqi Pang, Xin Chen, Han Liang, Minye Wu, Yuexin Ma, Lan Xu

We propose a hybrid motion inference stage with a generation network, which utilizes a temporal encoder-decoder to extract the motion details from the pair-wise sparse-view reference, as well as a motion discriminator to utilize the unpaired marker-based references to extract specific challenging motion characteristics in a data-driven manner.

OccuSeg: Occupancy-aware 3D Instance Segmentation

no code implementations CVPR 2020 Lei Han, Tian Zheng, Lan Xu, Lu Fang

3D instance segmentation, with a variety of applications in robotics and augmented reality, is in large demands these days.

3D Instance Segmentation Clustering +3

Beyond SIFT using Binary features for Loop Closure Detection

no code implementations18 Sep 2017 Lei Han, Guyue Zhou, Lan Xu, Lu Fang

The proposed system originates from our previous work Multi-Index hashing for Loop closure Detection (MILD), which employs Multi-Index Hashing (MIH)~\cite{greene1994multi} for Approximate Nearest Neighbor (ANN) search of binary features.

Loop Closure Detection

FlyCap: Markerless Motion Capture Using Multiple Autonomous Flying Cameras

no code implementations29 Oct 2016 Lan Xu, Lu Fang, Wei Cheng, Kaiwen Guo, Guyue Zhou, Qionghai Dai, Yebin Liu

We propose a novel non-rigid surface registration method to track and fuse the depth of the three flying cameras for surface motion tracking of the moving target, and simultaneously calculate the pose of each flying camera.

Markerless Motion Capture Visual Odometry

Cannot find the paper you are looking for? You can Submit a new open access paper.