Search Results for author:
Found 55 papers, 21 papers with code
FRESA:Feedforward Reconstruction of Personalized Skinned Avatars from Few Images
We present a novel method for reconstructing personalized 3D human avatars with realistic animation from only a few images.
Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior
Once the UPM is learned to accurately reproduce the large-scale multi-view human images, we fine-tune the model with an in-the-wild video via inverse rendering to obtain a personalized photorealistic human avatar that can be faithfully animated to novel human motions and rendered from novel views.
URAvatar: Universal Relightable Gaussian Codec Avatars
Unlike existing approaches that estimate parametric reflectance parameters via inverse rendering, our approach directly models learnable radiance transfer that incorporates global light transport in an efficient manner for real-time rendering.
Universal Facial Encoding of Codec Avatars from VR Headsets
The oblique and incomplete views of the face, variability in the donning of headsets, and illumination variation due to the environment are some of the unique challenges in generalization to unseen faces.
Rasterized Edge Gradients: Handling Discontinuities Differentiably
Computing the gradients of a rendering process is paramount for diverse applications in computer vision and graphics.
URHand: Universal Relightable Hands
To simplify the personalization process while retaining photorealism, we build a powerful universal relightable prior based on neural relighting from multi-view images of hands captured in a light stage with hundreds of identities.
RelightableHands: Efficient Neural Relighting of Articulated Hand Models
To achieve generalization, we condition the student model with physics-inspired illumination features such as visibility, diffuse shading, and specular reflections computed on a coarse proxy geometry, maintaining a small computational overhead.
Multiface: A Dataset for Neural Face Rendering
Along with the release of the dataset, we conduct ablation studies on the influence of different model architectures toward the model's interpolation capacity of novel viewpoint and expressions.
Drivable Volumetric Avatars using Texel-Aligned Features
To circumvent this, we propose a novel volumetric avatar representation by extending mixtures of volumetric primitives to articulated objects.
Dressing Avatars: Deep Photorealistic Appearance for Physically Simulated Clothing
The key idea is to introduce a neural clothing appearance model that operates on top of explicit geometry: at training time we use high-fidelity tracking, whereas at animation time we rely on physically simulated geometry.
Garment Avatars: Realistic Cloth Driving using Pattern Registration
Here, we propose an end-to-end pipeline for building drivable representations for clothing.
Driving-Signal Aware Full-Body Avatars
The core intuition behind our method is that better drivability and generalization can be achieved by disentangling the driving signals and remaining generative factors, which are not available during animation.
MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
To improve upon existing models, we propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face.
Ranked #2 on
3D Face Animation
on VOCASET
Pixel Codec Avatars
Telecommunication with photorealistic avatars in virtual or augmented reality is a promising path for achieving authentic face-to-face communication in 3D over remote physical distances.
High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation
This paper addresses previous limitations by learning a deep learning lighting model, that in combination with a high-quality 3D face tracking algorithm, provides a method for subtle and robust facial motion transfer from a regular video to a 3D photo-realistic avatar.
Mixture of Volumetric Primitives for Efficient Neural Rendering
Real-time rendering and animation of humans is a core function in games, movies, and telepresence applications.
Supervision by Registration and Triangulation for Landmark Detection
End-to-end training is made possible by differentiable registration and 3D triangulation modules.
Neural Synthesis of Binaural Audio
We present a neural rendering approach for binaural sound synthesis that can produce realistic and spatially accurate binaural sound in realtime.
Expressive Telepresence via Modular Codec Avatars
It is important to note that traditional person-specific CAs are learned from few training samples, and typically lack robustness as well as limited expressiveness when transferring facial expressions.
Audio- and Gaze-driven Facial Animation of Codec Avatars
Codec Avatars are a recent class of learned, photorealistic face models that accurately represent the geometry and texture of a person in 3D (i. e., for virtual reality), and are almost indistinguishable from video.
Spatiotemporal Bundle Adjustment for Dynamic 3D Human Reconstruction in the Wild
The triangulation constraint, however, is invalid for moving points captured in multiple unsynchronized videos and bundle adjustment is not designed to estimate the temporal alignment between cameras.
Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernels
Learning latent representations of registered meshes is useful for many 3D tasks.
4D Visualization of Dynamic Events from Unconstrained Multi-View Videos
We present a data-driven approach for 4D space-time visualization of dynamic events from videos captured by hand-held multiple cameras.
To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations
In this paper, we introduce a neural architecture named Dyadic Residual-Attention Model (DRAM), which integrates intrapersonal (monadic) and interpersonal (dyadic) dynamics using selective attention to generate sequences of body pose conditioned on audio and body pose of the interlocutor and audio of the human operating the avatar.
Single-Network Whole-Body Pose Estimation
We present the first single-network approach for 2D~whole-body pose estimation, which entails simultaneous localization of body, face, hands, and feet keypoints.
Neural Volumes: Learning Dynamic Renderable Volumes from Images
Modeling and rendering of dynamic scenes is challenging, as natural scenes often contain complex phenomena such as thin structures, evolving topology, translucency, scattering, occlusion, and biological motion.
Shapes and Context: In-the-Wild Image Synthesis & Manipulation
We introduce a data-driven approach for interactively synthesizing in-the-wild images from semantic label maps.
Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction
We present a new research task and a dataset to understand human social interactions via computational methods, to ultimately endow machines with the ability to encode and decode a broad channel of social signals humans use.
LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds
As input, we take a sequence of point clouds to be registered as well as an artist-rigged mesh, i. e. a template mesh equipped with a linear-blend skinning (LBS) deformation space parameterized by a skeleton hierarchy.
OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation
Ranked #4 on
Pose Estimation
on MPII Single Person
Capture Dense: Markerless Motion Capture Meets Dense Pose Estimation
Specifically, we first introduce a novel markerless motion capture method that can take advantage of dense parsing capability provided by the dense pose detector.
Monocular Total Capture: Posing Face, Body, and Hands in the Wild
We present the first method to capture the 3D total motion of a target person from a monocular view input.
Ranked #29 on
Monocular 3D Human Pose Estimation
on Human3.6M
Efficient Online Multi-Person 2D Pose Tracking with Recurrent Spatio-Temporal Affinity Fields
We present an online approach to efficiently and simultaneously detect and track the 2D pose of multiple people in a video sequence.
Ranked #7 on
Pose Tracking
on PoseTrack2017
(using extra training data)
Recycle-GAN: Unsupervised Video Retargeting
We introduce a data-driven approach for unsupervised video retargeting that translates content from one domain to another while preserving the style native to a domain, i. e., if contents of John Oliver's speech were to be transferred to Stephen Colbert, then the generated content/speech should be in Stephen Colbert's style.
Deep Appearance Models for Face Rendering
At inference time, we condition the decoding network on the viewpoint of the camera in order to generate the appropriate texture for rendering.
Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors
In this paper, we present supervision-by-registration, an unsupervised approach to improve the precision of facial landmark detectors on both images and video.
Ranked #1 on
Facial Landmark Detection
on 300-VW (C)
Modeling Facial Geometry Using Compositional VAEs
We propose a method for learning non-linear face geometry representations using deep generative models.
Learning Patch Reconstructability for Accelerating Multi-View Stereo
By prioritizing stereo matching on a subset of patches that are highly reconstructable and also cover the 3D surface, we are able to accelerate MVS with minimal reduction in accuracy and completeness.
Self-supervised Multi-view Person Association and Its Applications
Reliable markerless motion tracking of people participating in a complex group activity from multiple moving cameras is challenging due to frequent occlusions, strong viewpoint and appearance variations, and asynchronous video streams.
Structure from Recurrent Motion: From Rigidity to Recurrency
This paper proposes a new method for Non-Rigid Structure-from-Motion (NRSfM) from a long monocular video sequence observing a non-rigid object performing recurrent and possibly repetitive dynamic action.
Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies
We present a unified deformation model for the markerless capture of multiple scales of human movement, including facial expressions, body motion, and hand gestures.
Deltille Grids for Geometric Camera Calibration
The recent proliferation of high resolution cameras presents an opportunity to achieve unprecedented levels of precision in visual 3D reconstruction.
PixelNN: Example-based Image Synthesis
We present a simple nearest-neighbor (NN) approach that synthesizes high-frequency photorealistic images from an "incomplete" signal such as a low-resolution image, a surface normal map, or edges.
Hand Keypoint Detection in Single Images using Multiview Bootstrapping
The method is used to train a hand keypoint detector for single images.
Panoptic Studio: A Massively Multiview System for Social Interaction Capture
The core challenges in capturing social interactions are: (1) occlusion is functional and frequent; (2) subtle motion needs to be measured over a space large enough to host a social group; (3) human appearance and configuration variation is immense; and (4) attaching markers to the body may prime the nature of interactions.
Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
We present an approach to efficiently detect the 2D pose of multiple people in an image.
Ranked #1 on
Multi-Person Pose Estimation
on COCO
Spatiotemporal Bundle Adjustment for Dynamic 3D Reconstruction
In this paper, we present a spatiotemporal bundle adjustment approach that jointly optimizes four coupled sub-problems: estimating camera intrinsics and extrinsics, triangulating 3D static points, as well as subframe temporal alignment between cameras and estimating 3D trajectories of dynamic points.
How useful is photo-realistic rendering for visual learning?
Data seems cheap to get, and in many ways it is, but the process of creating a high quality labeled dataset from a mass of data is time-consuming and expensive.
Convolutional Pose Machines
Pose Machines provide a sequential prediction framework for learning rich implicit spatial models.
Ranked #2 on
Classification
on RSSCN7
Photogeometric Scene Flow for High-Detail Dynamic 3D Reconstruction
This paper proposes photogeometric scene flow (PGSF) for high-quality dynamic 3D reconstruction.
Panoptic Studio: A Massively Multiview System for Social Motion Capture
We present an approach to capture the 3D structure and motion of a group of people engaged in a social interaction.
MAP Visibility Estimation for Large-Scale Dynamic 3D Reconstruction
Many traditional challenges in reconstructing 3D motion, such as matching across wide baselines and handling occlusion, reduce in significance as the number of unique viewpoints increases.
Tracking Human Pose by Tracking Symmetric Parts
In this work, we present an occlusion aware algorithm for tracking human pose in an image sequence, that addresses the problem of double counting.
Representing and Discovering Adversarial Team Behaviors Using Player Roles
In this paper, we describe a method to represent and discover adversarial group behavior in a continuous domain.
Nonrigid Structure from Motion in Trajectory Space
Existing approaches to nonrigid structure from motion assume that the instantaneous 3D shape of a deforming object is a linear combination of basis shapes, which have to be estimated anew for each video sequence.
