Search Results for author:
Found 92 papers, 34 papers with code
RobustFusion: Human Volumetric Capture with Data-driven Visual Cues using a RGBD Camera
To enable robust tracking, we embrace both the initial model and the various visual cues into a novel performance capture scheme with hybrid motion optimization and semantic volumetric fusion, which can successfully capture challenging human motions under the monocular setting without pre-scanned detailed template and owns the reinitialization ability to recover from tracking failures and the disappear-reoccur scenarios.
TexVocab: Texture Vocabulary-conditioned Human Avatars
To adequately utilize the available image evidence in multi-view video-based avatar modeling, we propose TexVocab, a novel avatar representation that constructs a texture vocabulary and associates body poses with texture maps for animation.
Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives
In contrast, the second-stage is the local diffusion, which parallelly generates detailed motion sequences under the guidance of the dance primitives and choreographic rules.
Ranked #1 on
Motion Synthesis
on FineDance
TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding
Humans commonly work with multiple objects in daily life and can intuitively transfer manipulation skills to novel objects by understanding object functional regularities.
Ins-HOI: Instance Aware Human-Object Interactions Recovery
To address this, we further propose a complementary training strategy that leverages synthetic data to introduce instance-level shape priors, enabling the disentanglement of occupancy fields for different instances.
GMTalker: Gaussian Mixture based Emotional talking video Portraits
Specifically, we propose a Gaussian Mixture based Expression Generator (GMEG) which can construct a continuous and multi-modal latent space, achieving more flexible emotion manipulation.
Layered 3D Human Generation via Semantic-Aware Diffusion Model
To keep the generated clothing consistent with the target text, we propose a semantic-confidence strategy for clothing that can eliminate the non-clothing content generated by the model.
MonoGaussianAvatar: Monocular Gaussian Point-based Head Avatar
In response to these challenges, we propose MonoGaussianAvatar (Monocular Gaussian Point-based Head Avatar), a novel approach that harnesses 3D Gaussian point representation coupled with a Gaussian deformation field to learn explicit head avatars from monocular portrait videos.
Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians
Creating high-fidelity 3D head avatars has always been a research hotspot, but there remains a great challenge under lightweight sparse view setups.
GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis
We present a new approach, termed GPS-Gaussian, for synthesizing novel views of a character in a real-time manner.
InvertAvatar: Incremental GAN Inversion for Generalized Head Avatars
While high fidelity and efficiency are central to the creation of digital head avatars, recent methods relying on 2D or 3D generative models often experience limitations such as shape distortion, expression inaccuracy, and identity flickering.
SpeechAct: Towards Generating Whole-body Motion from Speech
Then, we regress the motion representation from the audio signal by a translation model employing our contrastive motion learning method.
Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling
Overall, our method can create lifelike avatars with dynamic, realistic and generalized appearances.
Human as Points: Explicit Point-based 3D Human Reconstruction from Single-view RGB Images
The latest trends in the research field of single-view human reconstruction devote to learning deep implicit functions constrained by explicit body shape priors.
DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene.
High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field
One crucial aspect of 3D head avatar reconstruction lies in the details of facial expressions.
HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation
The main idea is to enhance the model's 2D perception of 3D geometry by learning a normal-adapted diffusion model and a normal-aligned diffusion model.
HAvatar: High-fidelity Head Avatar via Facial Model Conditioned Neural Radiance Field
The problem of modeling an animatable 3D human head avatar under light-weight setups is of significant importance but has not been well solved.
Leveraging Intrinsic Properties for Non-Rigid Garment Alignment
To achieve wrinkle-level as well as texture-level alignment, we present a novel coarse-to-fine two-stage method that leverages intrinsic manifold properties with two neural deformation fields, in the 3D space and the intrinsic space, respectively.
CaPhy: Capturing Physical Properties for Animatable Human Avatars
In contrast to previous work on 3D avatar reconstruction, our method is able to generalize to novel poses with realistic dynamic cloth deformations.
ProxyCap: Real-time Monocular Full-body Capture in World Space via Human-Centric Proxy-to-Motion Learning
For more accurate and physically plausible predictions in world space, our network is designed to learn human motions from a human-centric perspective, which enables the understanding of the same motion captured with different camera trajectories.
Ranked #208 on
3D Human Pose Estimation
on Human3.6M
Control4D: Efficient 4D Portrait Editing with Text
We introduce Control4D, an innovative framework for editing dynamic 4D portraits using text instructions.
Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images
In the second part, we introduce a novel method to diffuse estimated contact states from the hand mesh surface to nearby 3D space and leverage diffused contact probabilities to construct the implicit neural representation for the manipulated object.
AvatarReX: Real-time Expressive Full-body Avatars
We present AvatarReX, a new method for learning NeRF-based full-body avatars from video data.
LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatar
Existing approaches to animatable NeRF-based head avatars are either built upon face templates or use the expression coefficients of templates as the driving signal.
StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Video
Results and experiments demonstrate the superiority of our method in terms of image quality, full portrait video generation, and real-time re-animation compared to existing facial reenactment methods.
PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling
To this end, we present PoseVocab, a novel pose encoding method that encourages the network to discover the optimal pose embeddings for learning the dynamic human appearance.
CloSET: Modeling Clothed Humans on Continuous Surface with Explicit Template Decomposition
In this way, the clothing deformations are disentangled such that the pose-dependent wrinkles can be better learned and applied to unseen poses.
Narrator: Towards Natural Control of Human-Scene Interaction Generation via Relationship Reasoning
Also, we model global and local spatial relationships in a 3D scene and a textual description respectively based on the scene graph, and introduce a partlevel action mechanism to represent interactions as atomic body part states.
Delving Deep into Pixel Alignment Feature for Accurate Multi-view Human Mesh Recovery
The key components of a typical regressor lie in the feature extraction of input views and the fusion of multi-view features.
Tensor4D: Efficient Neural 4D Decomposition for High-Fidelity Dynamic Reconstruction and Rendering
The key of our solution is an efficient 4D tensor decomposition method so that the dynamic scene can be directly represented as a 4D spatio-temporal tensor.
AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels
AvatarMAV is the first to model both the canonical appearance and the decoupled expression motion by neural voxels for head avatar.
Tensor4D : Efficient Neural 4D Decomposition for High-fidelity Dynamic Reconstruction and Rendering
The key of our solution is an efficient 4D tensor decomposition method so that the dynamic scene can be directly represented as a 4D spatio-temporal tensor.
Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars
We propose a novel 3D GAN framework for unsupervised learning of generative, high-quality and 3D-consistent facial avatars from unstructured 2D images.
DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras
At its core is a novel diffusion-based stereo module, which introduces diffusion models, a type of powerful generative models, into the iterative stereo matching network.
Learning Implicit Templates for Point-Based Clothed Human Modeling
We present FITE, a First-Implicit-Then-Explicit framework for modeling human avatars in clothing.
PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images
To address these issues, we propose a Pyramidal Mesh Alignment Feedback (PyMAF) loop in our regression network for well-aligned human mesh recovery and extend it as PyMAF-X for the recovery of expressive full-body models.
Ranked #6 on
3D Human Pose Estimation
on AGORA
(using extra training data)
Geometry-aware Single-image Full-body Human Relighting
For the relighting, we introduce a ray tracing-based per-pixel lighting representation that explicitly models high-frequency shadows and propose a learning-based shading refinement module to restore realistic shadows (including hard cast shadows) from the ray-traced shading maps.
AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture
Then given a monocular RGB video of this subject, our method integrates information from both the image observation and the avatar prior, and accordingly recon-structs high-fidelity 3D textured models with dynamic details regardless of the visibility.
Geo-NI: Geometry-aware Neural Interpolation for Light Field Rendering
In this paper, we present a Geometry-aware Neural Interpolation (Geo-NI) framework for light field rendering.
FOF: Learning Fourier Occupancy Field for Monocular Real-time Human Reconstruction
Based on FOF, we design the first 30+FPS high-fidelity real-time monocular human reconstruction framework.
IDE-3D: Interactive Disentangled Editing for High-Resolution 3D-aware Portrait Synthesis
Existing 3D-aware facial generation methods face a dilemma in quality versus editability: they either generate editable results in low resolution or high-quality ones with no editing flexibility.
GIMO: Gaze-Informed Human Motion Prediction in Context
We perform an extensive study of the benefits of leveraging the eye gaze for ego-centric human motion prediction with various state-of-the-art architectures.
ProbNVS: Fast Novel View Synthesis with Learned Probability-Guided Sampling
Existing state-of-the-art novel view synthesis methods rely on either fairly accurate 3D geometry estimation or sampling of the entire space for neural volumetric rendering, which limit the overall efficiency.
Structured Local Radiance Fields for Human Avatar Modeling
These local radiance fields not only leverage the flexibility of implicit representation in shape and appearance modeling, but also factorize cloth deformations into skeleton motions, node residual translations and the dynamic detail variations inside each individual radiance field.
FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset
In the coarse module, we generate a base parametric model from large-scale RGB-D images, which is able to predict accurate rough 3D face models in different genders, ages, etc.
Interacting Attention Graph for Single Image Two-Hand Reconstruction
To solve occlusion and interaction challenges of two-hand reconstruction, we introduce two novel attention based modules in each upsampling step of the original GCN.
Ranked #4 on
3D Interacting Hand Pose Estimation
on InterHand2.6M
3D Interacting Hand Pose Estimation
Vocal Bursts Valence Prediction
Recovering 3D Human Mesh from Monocular Images: A Survey
Since the release of statistical body models, 3D human mesh recovery has been drawing broader attention.
High-Fidelity Human Avatars From a Single RGB Camera
To cope with the complexity of textures and generate photo-realistic results, we propose a reference-based neural rendering network and exploit a bottom-up sharpening-guided fine-tuning strategy to obtain detailed textures.
HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars
To handle complicated motions (e. g., self-occlusions), we then leverage the encoded information on the UV manifold to construct a 3D volumetric representation based on a dynamic pose-conditioned neural radiance field.
FENeRF: Face Editing in Neural Radiance Fields
2D GANs can generate high fidelity portraits but with low view consistency.
Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras
Overall, we propose the first light-weight total capture system and achieves fast, robust and accurate multi-person total motion capture performance.
Ranked #2 on
3D Multi-Person Pose Estimation
on Shelf
LocalTrans: A Multiscale Local Transformer Network for Cross-Resolution Homography Estimation
By combining the local transformer with the multiscale structure, the network is able to capture long-short range correspondences efficiently and accurately.
DoubleField: Bridging the Neural Surface and Radiance Fields for High-fidelity Human Reconstruction and Rendering
We introduce DoubleField, a novel framework combining the merits of both surface field and radiance field for high-fidelity human reconstruction and rendering.
Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors
Human volumetric capture is a long-standing topic in computer vision and computer graphics.
DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras
We propose DeepMultiCap, a novel method for multi-person performance capture using sparse multi-view cameras.
Revisiting Light Field Rendering with Deep Anti-Aliasing Neural Network
In this paper, we revisit the classic LF rendering framework to address both challenges by incorporating it with advanced deep learning techniques.
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
Regression-based methods have recently shown promising results in reconstructing human meshes from monocular images.
Ranked #5 on
3D Human Pose Estimation
on AGORA
(using extra training data)
3D human pose and shape estimation
3D Human Reconstruction
+2
POSEFusion: Pose-guided Selective Fusion for Single-view Human Volumetric Capture
By contributing a novel reconstruction framework which contains pose-guided keyframe selection and robust implicit surface fusion, our method fully utilizes the advantages of both tracking-based methods and tracking-free inference methods, and finally enables the high-fidelity reconstruction of dynamic surface details even in the invisible regions.
Light Field Reconstruction Using Convolutional Network on EPI and Extended Applications
The main problem in direct reconstruction on the EPI involves an information asymmetry between the spatial and angular dimensions, where the detailed portion in the angular dimensions is damaged by undersampling.
Training Weakly Supervised Video Frame Interpolation With Events
Though trained on low frame-rate videos, our framework outperforms existing models trained with full high frame-rate videos (and events) on both GoPro dataset and a new real event-based dataset.
PoNA: Pose-guided Non-local Attention for Human Pose Transfer
In each block, we propose a pose-guided non-local attention (PoNA) mechanism with a long-range dependency scheme to select more important regions of image features to transfer.
Cross-MPI: Cross-scale Stereo for Image Super-Resolution using Multiplane Images
Various combinations of cameras enrich computational photography, among which reference-based superresolution (RefSR) plays a critical role in multiscale imaging systems.
DeepCloth: Neural Garment Representation for Shape and Style Editing
In this work, we introduce, DeepCloth, a unified framework for garment representation, reconstruction, animation and editing.
Vehicle Reconstruction and Texture Estimation Using Deep Implicit Semantic Template Mapping
We introduce VERTEX, an effective solution to recover 3D shape and intrinsic texture of vehicles from uncalibrated monocular input in real-world street environments.
Deep Implicit Templates for 3D Shape Representation
Deep implicit functions (DIFs), as a kind of 3D shape representation, are becoming more and more popular in the 3D vision community due to their compactness and strong representation power.
NormalGAN: Learning Detailed 3D Human from a Single RGB-D Image
We propose NormalGAN, a fast adversarial learning-based method to reconstruct the complete and detailed 3D human from a single RGB-D image.
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction
To overcome the limitations of regular 3D representations, we propose Parametric Model-Conditioned Implicit Representation (PaMIR), which combines the parametric body model with the free-form deep implicit function.
Ranked #2 on
3D Human Reconstruction
on CAPE
Spatial-Angular Attention Network for Light Field Reconstruction
In this paper, we propose a spatial-angular attention network to perceive correspondences in the light field non-locally, and reconstruction high angular resolution light field in an end-to-end manner.
MulayCap: Multi-layer Human Performance Capture Using A Monocular Video Camera
We introduce MulayCap, a novel human performance capture method using a monocular video camera without the need for pre-scanning.
Learning Event-Based Motion Deblurring
Recovering sharp video sequence from a motion-blurred image is highly ill-posed due to the significant loss of motion information in the blurring process.
Ranked #27 on
Image Deblurring
on GoPro
(using extra training data)
Robust 3D Self-portraits in Seconds
In this paper, we propose an efficient method for robust 3D self-portraits using a single RGBD camera.
4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras
Our method enables a realtime online motion capture system running at 30fps using 5 cameras on a 5-person scene.
Ranked #8 on
3D Multi-Person Pose Estimation
on Shelf
Learning to Reconstruct and Understand Indoor Scenes from Sparse Views
This paper proposes a new method for simultaneous 3D reconstruction and semantic segmentation of indoor scenes.
SimulCap : Single-View Human Performance Capture with Cloth Simulation
This paper proposes a new method for live free-viewpoint human performance capture with dynamic details (e. g., cloth wrinkles) using a single RGBD camera.
DeepHuman: 3D Human Reconstruction from a Single Image
We propose DeepHuman, an image-guided volume-to-volume translation CNN for 3D human reconstruction from a single RGB image.
LapEPI-Net: A Laplacian Pyramid EPI structure for Learning-based Dense Light Field Reconstruction
We then propose a novel network architecture for the LapEPI structure, termed as LapEPI-net.
Capture Dense: Markerless Motion Capture Meets Dense Pose Estimation
Specifically, we first introduce a novel markerless motion capture method that can take advantage of dense parsing capability provided by the dense pose detector.
DDRNet: Depth Map Denoising and Refinement for Consumer Depth Cameras Using Cascaded CNNs
Consumer depth sensors are more and more popular and come to our daily lives marked by its recent integration in the latest Iphone X.
HybridFusion: Real-Time Performance Capture Using a Single Depth Sensor and Sparse IMUs
We propose a light-weight and highly robust real-time human performance capture method based on a single depth camera and sparse inertial measurement units (IMUs).
CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping
The Reference-based Super-resolution (RefSR) super-resolves a low-resolution (LR) image given an external high-resolution (HR) reference image, where the reference image and LR image share similar viewpoint but with significant resolution gap x8.
Structure from Recurrent Motion: From Rigidity to Recurrency
This paper proposes a new method for Non-Rigid Structure-from-Motion (NRSfM) from a long monocular video sequence observing a non-rigid object performing recurrent and possibly repetitive dynamic action.
DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor
We further propose a joint motion tracking method based on the double layer representation to enable robust and fast motion tracking performance.
BodyFusion: Real-Time Capture of Human Motion and Surface Geometry Using a Single Depth Camera
To reduce the ambiguities of the non-rigid deformation parameterization on the surface graph nodes, we take advantage of the internal articulated motion prior for human performance and contribute a skeleton-embedded surface fusion (SSF) method.
SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis
It takes a set of images and their corresponding camera parameters as input and directly infers the 3D model.
Light Field Reconstruction Using Deep Convolutional Network on EPI
In this paper, we take advantage of the clear texture structure of the epipolar plane image (EPI) in the light field data and model the problem of light field reconstruction from a sparse set of views as a CNN-based angular detail restoration on EPI.
Turning an Urban Scene Video into a Cinemagraph
Our approach first warps an input video into the viewpoint of a reference camera.
FlyCap: Markerless Motion Capture Using Multiple Autonomous Flying Cameras
We propose a novel non-rigid surface registration method to track and fuse the depth of the three flying cameras for surface motion tracking of the moving target, and simultaneously calculate the pose of each flying camera.
Robust Non-Rigid Motion Tracking and Surface Reconstruction Using L0 Regularization
We present a new motion tracking method to robustly reconstruct non-rigid geometries and motions from single view depth inputs captured by a consumer depth sensor.
Learning High-level Prior with Convolutional Neural Networks for Semantic Segmentation
Finally, the optimization problem enables us to take advantage of state-of-the-art fully convolutional network structure for the implementation of the above encoders and decoder.
Light Field From Micro-Baseline Image Pair
We first introduce a disparity assisted phase based synthesis (DAPS) strategy that can integrate disparity infor- mation into the phase term of a reference image to warp it to its close neighbor views.
Fourier Analysis on Transient Imaging with a Multifrequency Time-of-Flight Camera
A transient image is the optical impulse response of a scene which visualizes light propagation during an ultra-short time interval.
