Search Results for author:
Found 48 papers, 17 papers with code
VolumeDeform: Real-time Volumetric Non-rigid Reconstruction
We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints.
BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration
Our approach estimates globally optimized (i. e., bundle adjusted) poses in real-time, supports robust tracking with recovery from gross tracking failures (i. e., relocalization), and re-estimates the 3D model in real-time to ensure global consistency; all within a single framework.
Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging
Many graphics and vision problems can be expressed as non-linear least squares optimizations of objective functions over visual data, such as images and meshes.
FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality
Based on reenactment of a prerecorded stereo video of the person without the HMD, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances.
Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
However, due to difficult occlusions, fast motions, and uniform hand appearance, jointly tracking hand and object pose is more challenging than tracking either of the two separately.
Real-time Halfway Domain Reconstruction of Motion and Geometry
Our extensive experiments and evaluations show that our approach produces high-quality dense reconstructions of 3D geometry and scene flow at real-time frame rates, and compares favorably to the state of the art.
MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction
In this work we propose a novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image.
InverseFaceNet: Deep Monocular Inverse Face Rendering
In contrast, we propose to recover high-quality facial pose, shape, expression, reflectance and illumination using a deep neural network that is trained using a large, synthetically created training corpus.
MonoPerfCap: Human Performance Capture from Monocular Video
Reconstruction from monocular video alone is drastically more challenging, since strong occlusions and the inherent depth ambiguity lead to a highly ill-posed reconstruction problem.
Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz
To alleviate this problem, we present the first approach that jointly learns 1) a regressor for face shape, expression, reflectance and illumination on the basis of 2) a concurrently learned parametric face model.
IMU2Face: Real-time Gesture-driven Facial Reenactment
We present IMU2Face, a gesture-driven facial reenactment system.
HeadOn: Real-time Reenactment of Human Portrait Videos
We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze.
Deep Video Portraits
In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network -- thus taking full control of the target.
IGNOR: Image-guided Neural Object Rendering
Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.
DeepVoxels: Learning Persistent 3D Feature Embeddings
In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis.
FML: Face Model Learning from Videos
In contrast, we propose multi-frame video-based self-supervised training of a deep network that (i) learns a face identity model both in shape and appearance while (ii) jointly learning to reconstruct 3D faces.
Commodity RGB-D Sensors: Data Acquisition
Over the past ten years we have seen a democratization of range sensing technology.
Deferred Neural Rendering: Image Synthesis using Neural Textures
Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline.
DEMEA: Deep Mesh Autoencoders for Non-Rigidly Deforming Objects
Mesh autoencoders are commonly used for dimensionality reduction, sampling and mesh modeling.
Text-based Editing of Talking-head Video
To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material.
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes.
Neural Style-Preserving Visual Dubbing
We present a style-preserving visual dubbing approach from single video inputs, which maintains the signature style of target actors when modifying facial expressions, including mouth motions, to match foreign languages.
DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data
Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus.
Image-guided Neural Object Rendering
Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.
StyleRig: Rigging StyleGAN for 3D Control over Portrait Images
StyleGAN generates photorealistic portrait images of faces with eyes, teeth, hair and context (neck, shoulders, background), but lacks a rig-like control over semantic face parameters that are interpretable in 3D, such as face pose, expressions, and scene illumination.
State of the Art on Neural Rendering
Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e. g., by the integration of differentiable rendering into network training.
Neural Non-Rigid Tracking
We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction by a learned robust optimization.
Face2Face: Real-time Face Capture and Reenactment of RGB Videos
Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.
PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations
At the level of patches, objects across different categories share similarities, which leads to more generalizable models.
PIE: Portrait Image Embedding for Semantic Control
We present the first approach for embedding real portrait images in the latent space of StyleGAN, which allows for intuitive editing of the head pose, facial expression, and scene illumination in the image.
Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction
We introduce Neural Deformation Graphs for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction
We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face.
Learning Compositional Radiance Fields of Dynamic Human Heads
In addition, we show that the learned dynamic radiance field can be used to synthesize novel unseen expressions based on a global animation code.
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video
We show that a single handheld consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views, e. g. a `bullet-time' video effect.
Learning Neural Light Fields With Ray-Space Embedding
Our method supports rendering with a single network evaluation per pixel for small baseline light fields and with only a few evaluations per pixel for light fields with larger baselines.
AutoAvatar: Autoregressive Neural Fields for Dynamic Avatar Modeling
In this work, for the first time, we enable autoregressive modeling of implicit avatars.
Self-supervised Neural Articulated Shape and Appearance Models
In addition, our representation enables a large variety of applications, such as few-shot reconstruction, the generation of novel articulations, and novel view-synthesis.
AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields
However, rendering images with this new paradigm is slow due to the fact that an accurate quadrature of the volume rendering equation requires a large number of samples for each ray.
Neural Pixel Composition for 3D-4D View Synthesis From Multi-Views
We present Neural Pixel Composition (NPC), a novel approach for continuous 3D-4D view synthesis given only a discrete set of multi-view observations as input.
Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering
We further conduct an extensive comparative study of different priors on illumination used in previous work on inverse rendering.
VR-NeRF: High-Fidelity Virtualized Walkable Spaces
We present an end-to-end system for the high-fidelity capture, model reconstruction, and real-time rendering of walkable spaces in virtual reality using neural radiance fields.
Drivable 3D Gaussian Avatars
We present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats.
PyNeRF: Pyramidal Neural Radiance Fields
Compared to Mip-NeRF, we reduce error rates by 20% while training over 60x faster.
HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces
Neural radiance fields provide state-of-the-art view synthesis quality but tend to be slow to render.
SpecNeRF: Gaussian Directional Encoding for Specular Reflections
We show that our Gaussian directional encoding and geometry prior significantly improve the modeling of challenging specular reflections in neural radiance fields, which helps decompose appearance into more physically meaningful components.
URHand: Universal Relightable Hands
To simplify the personalization process while retaining photorealism, we build a powerful universal relightable prior based on neural relighting from multi-view images of hands captured in a light stage with hundreds of identities.
IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images
While numerous 3D reconstruction and novel-view synthesis methods allow for photorealistic rendering of a scene from multi-view images easily captured with consumer cameras, they bake illumination in their representations and fall short of supporting advanced applications like material editing, relighting, and virtual object insertion.
ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models
In this paper, we present a method that leverages pretrained text-to-image models as a prior, and learn to generate multi-view images in a single denoising process from real-world data.
