Search Results for author:
Found 73 papers, 29 papers with code
Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation
In this paper, we introduce Zero-1-to-G, a novel approach that addresses this problem by enabling direct single-view generation on Gaussian splats using pretrained 2D diffusion models.
ProTracker: Probabilistic Integration for Robust and Accurate Point Tracking
In this paper, we propose ProTracker, a novel framework for robust and accurate long-term dense tracking of arbitrary points in videos.
Extrapolated Urban View Synthesis Benchmark
Our results show that Gaussian Splatting is prone to overfitting to training views.
HDGS: Textured 2D Gaussian Splatting for Enhanced Scene Rendering
We proposed a novel method to align the 2D surfels with texture maps and augment it with per-ray depth sorting and fisher-based pruning for rendering consistency and efficiency.
MotionWavelet: Human Motion Prediction via Wavelet Manifold Learning
Once the Wavelet Manifold is built, WDM trains a diffusion model to generate human motions from Wavelet latent vectors.
DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image
The first and only method for hand-face interaction recovery, Decaf, introduces a global fitting optimization guided by contact and deformation estimation networks trained on studio-collected data with 3D annotations.
GECO: Generative Image-to-3D within a SECOnd
In the first stage, we train a single-step multi-view generative model with score distillation.
RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion
We introduce RealmDreamer, a technique for generation of general forward-facing 3D scenes from text descriptions.
Track Everything Everywhere Fast and Robustly
We propose a novel test-time optimization approach for efficiently and robustly tracking any pixel at any time in a video.
TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videos
We propose TRAM, a two-stage method to reconstruct a human's global trajectory and motion from in-the-wild videos.
Ranked #1 on
3D Human Pose Estimation
on EMDB
NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation
Neural Radiance Field (NeRF) has been widely recognized for its excellence in novel view synthesis and 3D scene reconstruction.
CoMo: Controllable Motion Generation through Language Guided Pose Code Editing
Text-to-motion models excel at efficient human motion generation, but existing approaches lack fine-grained controllability over the generation process.
Adaptive Surface Normal Constraint for Geometric Estimation from Monocular Images
We introduce a novel approach to learn geometries such as depth and surface normal from images while incorporating geometric context.
DressCode: Autoregressively Sewing and Generating Garments from Text Guidance
We first introduce SewingGPT, a GPT-based architecture integrating cross-attention with text-conditioned embedding to generate sewing patterns with text guidance.
Holodeck: Language Guided Generation of 3D Embodied AI Environments
3D simulated environments play a critical role in Embodied AI, but their creation requires expertise and extensive manual effort, restricting their diversity and scope.
DiffusionPhase: Motion Diffusion in Frequency Domain
In this study, we introduce a learning-based method for generating high-quality human motion sequences from text descriptions (e. g., ``A person walks forward").
EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generation
We introduce Efficient Motion Diffusion Model (EMDM) for fast and high-quality human motion generation.
Ranked #7 on
Motion Synthesis
on KIT Motion-Language
Initializing Models with Larger Ones
Weight selection offers a new approach to leverage the power of pretrained models in resource-constrained settings, and we hope it can be a useful tool for training small models in the large-model era.
Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement
In this work, we explore egocentric whole-body motion capture using a single fisheye camera, which simultaneously estimates human body and hand motion.
Ranked #1 on
Egocentric Pose Estimation
on GlobalEgoMocap Test Dataset
(using extra training data)
TLControl: Trajectory and Language Control for Human Motion Synthesis
Controllable human motion synthesis is essential for applications in AR/VR, gaming and embodied AI.
GART: Gaussian Articulated Template Models
We introduce Gaussian Articulated Template Model GART, an explicit, efficient, and expressive representation for non-rigid articulated subject capturing and rendering from monocular videos.
Wonder3D: Single Image to 3D using Cross-Domain Diffusion
In this work, we introduce Wonder3D, a novel method for efficiently generating high-fidelity textured meshes from single-view images. Recent methods based on Score Distillation Sampling (SDS) have shown the potential to recover 3D geometry from 2D diffusion priors, but they typically suffer from time-consuming per-shape optimization and inconsistent geometry.
Ranked #2 on
Single-View 3D Reconstruction
on GSO
State of the Art on Diffusion Models for Visual Computing
The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes.
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image
In this paper, we present a novel diffusion model called that generates multiview-consistent images from a single-view image.
Ranked #3 on
Single-View 3D Reconstruction
on GSO
DreamEditor: Text-Driven 3D Scene Editing with Neural Fields
Neural fields have achieved impressive advancements in view synthesis and scene reconstruction.
BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping
Diffusion models have demonstrated excellent potential for generating diverse images.
NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images
We present a neural rendering-based method called NeRO for reconstructing the geometry and the BRDF of reflective objects from multiview images captured in an unknown environment.
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
Synthesizing visual content that meets users' needs often requires flexible and precise controllability of the pose, shape, expression, and layout of the generated objects.
General Neural Gauge Fields
In this work, we extend this problem to a general paradigm with a taxonomy of discrete \& continuous cases, and develop a learning framework to jointly optimize gauge transformations and neural fields.
Control3Diff: Learning Controllable 3D Diffusion Models from Single-view Images
To address these challenges, We present Control3Diff, a 3D diffusion model that combines the strengths of diffusion models and 3D GANs for versatile, controllable 3D-aware image synthesis for single-view datasets.
Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction
3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images.
F$^{2}$-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories
Based on our analysis, we further propose a novel space-warping method called perspective warping, which allows us to handle arbitrary trajectories in the grid-based NeRF framework.
NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion
Novel view synthesis from a single image requires inferring occluded regions of objects and scenes whilst simultaneously maintaining semantic and physical consistency with the input.
F2-NeRF: Fast Neural Radiance Field Training With Free Camera Trajectories
Existing fast grid-based NeRF training frameworks, like Instant-NGP, Plenoxels, DVGO, or TensoRF, are mainly designed for bounded scenes and rely on space warping to handle unbounded scenes.
Scene-aware Egocentric 3D Human Pose Estimation
To this end, we propose an egocentric depth estimation network to predict the scene depth map from a wide-view egocentric fisheye camera while mitigating the occlusion of the human body with a depth-inpainting network.
Ranked #3 on
Egocentric Pose Estimation
on GlobalEgoMocap Test Dataset
(using extra training data)
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction
Recent methods for neural surface representation and rendering, for example NeuS, have demonstrated the remarkably high-quality reconstruction of static scenes.
NeuralUDF: Learning Unsigned Distance Fields for Multi-view Reconstruction of Surfaces with Arbitrary Topologies
In this paper, we propose to represent surfaces as the Unsigned Distance Function (UDF) and develop a new volume rendering scheme to learn the neural UDF representation.
HDHumans: A Hybrid Approach for High-fidelity Digital Humans
Photo-real digital human avatars are of enormous importance in graphics, as they enable immersive communication over the globe, improve gaming and entertainment experiences, and can be particularly beneficial for AR and VR settings.
Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors
The learned representation can be used to synthesize novel view images of an arbitrary person from a sparse set of cameras, and further animate them with the user's pose control.
Progressively-connected Light Field Network for Efficient View Synthesis
ProLiF encodes a 4D light field, which allows rendering a large batch of rays in one training step for image- or patch-level losses.
NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors
The key idea of NeuRIS is to integrate estimated normal of indoor scenes as a prior in a neural rendering framework for reconstructing large texture-less shapes and, importantly, to do this in an adaptive manner to also enable the reconstruction of irregular shapes with fine details.
Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motions
We also observe that an object's intrinsic physical properties are useful for the object motion prediction, and thus design a set of object dynamic descriptors to encode such intrinsic properties.
GAN2X: Non-Lambertian Inverse Rendering of Image GANs
2D images are observations of the 3D physical world depicted with the geometry, material, and illumination components.
Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data
With a hybrid architecture that separates static and dynamic contents, fluid interactions with static obstacles are reconstructed for the first time without additional geometry input or human labeling.
Direct Dense Pose Estimation
In addition, DDP is computationally more efficient than previous dense pose estimation methods, and it reduces jitters when applied to a video sequence, which is a problem plaguing the previous methods.
Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision
Specifically, we first generate pseudo labels for the EgoPW dataset with a spatio-temporal optimization method by incorporating the external-view supervision.
Ranked #4 on
Egocentric Pose Estimation
on GlobalEgoMocap Test Dataset
(using extra training data)
Multimodal Image Synthesis and Editing: The Generative AI Era
With superb power in modeling the interaction among multimodal information, multimodal image synthesis and editing has become a hot research topic in recent years.
NeRF for Outdoor Scene Relighting
Photorealistic editing of outdoor scenes from photographs requires a profound understanding of the image formation process and an accurate estimation of the scene geometry, reflectance and illumination.
EgoRenderer: Rendering Human Avatars from Egocentric Camera Images
We next combine the target pose image and the textures into a combined feature image, which is transformed into the output color image using a neural image translation network.
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue.
Neural Rays for Occlusion-aware Image-based Rendering
On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction.
Structure-Aware Long Short-Term Memory Network for 3D Cephalometric Landmark Detection
Detecting 3D landmarks on cone-beam computed tomography (CBCT) is crucial to assessing and quantifying the anatomical abnormalities in 3D cephalometric analysis.
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation.
Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control
To address this problem, we utilize a coarse body model as the proxy to unwarp the surrounding 3D space into a canonical pose.
Semi-supervised Anatomical Landmark Detection via Shape-regulated Self-training
The global shape constraint is the inherent property of anatomical landmarks that provides valuable guidance for more consistent pseudo labelling of the unlabeled data, which is ignored in the previously semi-supervised methods.
Real-time Deep Dynamic Characters
We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery.
Estimating Egocentric 3D Human Pose in Global Space
Furthermore, these methods suffer from limited accuracy and temporal instability due to ambiguities caused by the monocular setup and the severe occlusion in a strongly distorted egocentric perspective.
Ranked #5 on
Egocentric Pose Estimation
on SceneEgo
(using extra training data)
Efficient and Differentiable Shadow Computation for Inverse Problems
Differentiable rendering has received increasing interest for image-based inverse problems.
Adaptive Surface Normal Constraint for Depth Estimation
We present a novel method for single image depth estimation using surface normal constraints.
HumanGAN: A Generative Model of Humans Images
We address these limitations and present a generative model for images of dressed humans offering control over pose, local body part appearance and garment style.
Style and Pose Control for Image Synthesis of Humans from a Single Monocular View
Photo-realistic re-rendering of a human from a single image with explicit control over body pose, shape and appearance enables a wide range of applications, such as human appearance transfer, virtual try-on, motion imitation, and novel view synthesis.
Learning Speech-driven 3D Conversational Gestures from Video
We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input.
Ranked #5 on
3D Face Animation
on BEAT2
Pose-Guided Human Animation from a Single Image in the Wild
We present a new pose transfer method for synthesizing a human animation from a single image of a person controlled by a sequence of body poses.
Learnable Motion Coherence for Correspondence Pruning
We propose a novel formulation of fitting coherent motions with a smooth function on a graph of correspondences and show that this formulation allows a closed-form solution by graph Laplacian.
Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
We present a novel method for multi-view depth estimation from a single video, which is a critical task in various applications, such as perception, reconstruction and robot navigation.
SEG-MAT: 3D Shape Segmentation Using Medial Axis Transform
Segmenting arbitrary 3D objects into constituent parts that are structurally meaningful is a fundamental problem encountered in a wide range of computer graphics applications.
Neural Sparse Voxel Fields
We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering.
Vid2Curve: Simultaneous Camera Motion Estimation and Thin Structure Reconstruction from an RGB Video
We propose the first approach that simultaneously estimates camera motion and reconstructs the geometry of complex 3D thin structures in high quality from a color video captured by a handheld camera.
MulayCap: Multi-layer Human Performance Capture Using A Monocular Video Camera
We introduce MulayCap, a novel human performance capture method using a monocular video camera without the need for pre-scanning.
Occlusion-Aware Depth Estimation with Adaptive Normal Constraints
We present a new learning-based method for multi-frame depth estimation from a color video, which is a fundamental problem in scene understanding, robot navigation or handheld 3D reconstruction.
Unsupervised Learning of Intrinsic Structural Representation Points
The 3D structure points produced by our method encode the shape structure intrinsically and exhibit semantic consistency across all the shape instances with similar structures.
Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation
In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space.
Neural Rendering and Reenactment of Human Actor Videos
In contrast to conventional human character rendering, we do not require the availability of a production-quality photo-realistic 3D model of the human, but instead rely on a video sequence in conjunction with a (medium-quality) controllable 3D template model of the person.
