Search Results for author:
Found 215 papers, 51 papers with code
Building Statistical Shape Spaces for 3D Human Modeling
Statistical models of 3D human shape and pose learned from scan databases have developed into valuable tools to solve a variety of vision and graphics problems.
Efficient ConvNet-Based Marker-Less Motion Capture in General Scenes With a Low Number of Cameras
Our approach unites a discriminative image-based joint detection method with a model-based generative motion tracking algorithm through a combined pose optimization energy.
Efficient Multi-view Performance Capture of Fine-Scale Surface Detail
We present a new effective way for performance capture of deforming meshes with fine-scale time-varying surface detail from multi-view video.
Automatic Face Reenactment
We propose an image-based, facial reenactment system that replaces the face of an actor in an existing target video with the face of a user from a source video, while preserving the original target performance.
Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model
In this paper, we propose a new approach that tracks the full skeleton motion of the hand from multiple RGB cameras in real-time.
Semi-supervised Learning with Explicit Relationship Regularization
In many learning tasks, the structure of the target space of a function holds rich information about the relationships between evaluations of functions on different data points.
Local High-order Regularization on Data Manifolds
The iterated graph Laplacian enables high-order regularization, but it has a high computational complexity and so cannot be applied to large problems.
A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation
Generative reconstruction methods compute the 3D configuration (such as pose and/or geometry) of a shape by optimizing the overlap of the projected 3D shape model with images.
Fast and Robust Hand Tracking Using Detection-Guided Optimization
In the optimization step, a novel objective function combines the detected part labels and a Gaussian mixture representation of the depth to estimate a pose that best fits the depth.
Context-guided diffusion for label propagation on graphs
Existing approaches for diffusion on graphs, e. g., for label propagation, are mainly focused on isotropic diffusion, which is induced by the commonly-used graph Laplacian regularizer.
VolumeDeform: Real-time Volumetric Non-rigid Reconstruction
We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints.
BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration
Our approach estimates globally optimized (i. e., bundle adjusted) poses in real-time, supports robust tracking with recovery from gross tracking failures (i. e., relocalization), and re-estimates the 3D model in real-time to ensure global consistency; all within a single framework.
Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging
Many graphics and vision problems can be expressed as non-linear least squares optimizations of objective functions over visual data, such as images and meshes.
General Automatic Human Shape and Motion Capture Using Volumetric Contour Cues
Our method uses a new image formation model with analytic visibility and analytically differentiable alignment energy.
Dense Wide-Baseline Scene Flow From Two Handheld Video Cameras
We finally refine the computed correspondence fields in a variational scene flow formulation.
EgoCap: Egocentric Marker-less Motion Capture with Two Fisheye Cameras
We therefore propose a new method for real-time, marker-less and egocentric motion capture which estimates the full-body skeleton pose from a lightweight stereo pair of fisheye cameras that are attached to a helmet or virtual reality headset.
FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality
Based on reenactment of a prerecorded stereo video of the person without the HMD, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances.
Video Depth-From-Defocus
Many compelling video post-processing effects, in particular aesthetic focus editing and refocusing effects, are feasible if per-frame depth information is available.
Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
However, due to difficult occlusions, fast motions, and uniform hand appearance, jointly tracking hand and object pose is more challenging than tracking either of the two separately.
Model-based Outdoor Performance Capture
We propose a new model-based method to accurately reconstruct human performances captured outdoors in a multi-camera setup.
Real-time Halfway Domain Reconstruction of Motion and Geometry
Our extensive experiments and evaluations show that our approach produces high-quality dense reconstructions of 3D geometry and scene flow at real-time frame rates, and compares favorably to the state of the art.
Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision
We propose a CNN-based approach for 3D human body pose estimation from single RGB images that addresses the issue of limited generalizability of models trained solely on the starkly limited publicly available 3D pose data.
Ranked #17 on
Pose Estimation
on Leeds Sports Poses
EgoCap: Egocentric Marker-less Motion Capture with Two Fisheye Cameras (Extended Abstract)
Marker-based and marker-less optical skeletal motion-capture methods use an outside-in arrangement of cameras placed around a scene, with viewpoints converging on the center.
MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction
In this work we propose a novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image.
InverseFaceNet: Deep Monocular Inverse Face Rendering
In contrast, we propose to recover high-quality facial pose, shape, expression, reflectance and illumination using a deep neural network that is trained using a large, synthetically created training corpus.
Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
We present an approach for real-time, robust and accurate hand pose estimation from moving egocentric RGB-D cameras in cluttered real environments.
VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton.
Ranked #16 on
Pose Estimation
on Leeds Sports Poses
Criteria Sliders: Learning Continuous Database Criteria via Interactive Ranking
Large databases are often organized by hand-labeled metadata, or criteria, which are expensive to collect.
MonoPerfCap: Human Performance Capture from Monocular Video
Reconstruction from monocular video alone is drastically more challenging, since strong occlusions and the inherent depth ambiguity lead to a highly ill-posed reconstruction problem.
HandSeg: An Automatically Labeled Dataset for Hand Segmentation from Depth Images
We propose an automatic method for generating high-quality annotations for depth-based hand segmentation, and introduce a large-scale hand segmentation dataset.
DS*: Tighter Lifting-Free Convex Relaxations for Quadratic Matching Problems
In this work we study convex relaxations of quadratic optimisation problems over permutation matrices.
GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB
We address the highly challenging problem of real-time 3D hand tracking based on a monocular RGB-only sequence.
Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz
To alleviate this problem, we present the first approach that jointly learns 1) a regressor for face shape, expression, reflectance and illumination on the basis of 2) a concurrently learned parametric face model.
Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB
Our approach uses novel occlusion-robust pose-maps (ORPM) which enable full body pose inference even under strong partial occlusions by other people and objects in the scene.
Ranked #3 on
3D Multi-Person Pose Estimation (root-relative)
on MuPoTS-3D
(MPJPE metric)
3D Human Pose Estimation
3D Multi-Person Pose Estimation (absolute)
+2
LIME: Live Intrinsic Material Estimation
We present the first end to end approach for real time material estimation for general object shapes with uniform material that only requires a single color image as input.
Video Based Reconstruction of 3D People Models
This paper describes how to obtain accurate 3D body models and texture of arbitrary people from a single, monocular video in which a person is moving.
Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera
We tackle these challenges based on a novel lightweight setup that converts a standard baseball cap to a device for high-quality pose estimation based on a single cap-mounted fisheye camera.
Ranked #6 on
Egocentric Pose Estimation
on GlobalEgoMocap Test Dataset
(using extra training data)
Synchronisation of Partial Multi-Matchings via Non-negative Factorisations
In order to deal with the inherent non-convexity of the permutation synchronisation problem, we use an initialisation procedure based on a novel rotation scheme applied to the solution of the spectral relaxation.
A Hybrid Model for Identity Obfuscation by Face Replacement
As more and more personal photos are shared and tagged in social media, avoiding privacy risks such as unintended recognition becomes increasingly challenging.
HeadOn: Real-time Reenactment of Human Portrait Videos
We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze.
Deep Video Portraits
In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network -- thus taking full control of the target.
Detailed Human Avatars from Monocular Video
We present a novel method for high detail-preserving human avatar creation from monocular video.
Neural Rendering and Reenactment of Human Actor Videos
In contrast to conventional human character rendering, we do not require the availability of a production-quality photo-realistic 3D model of the human, but instead rely on a video sequence in conjunction with a (medium-quality) controllable 3D template model of the person.
LiveCap: Real-time Human Performance Capture from Monocular Video
Our method is the first real-time monocular approach for full-body performance capture.
Higher-order Projected Power Iterations for Scalable Multi-Matching
The matching of multiple objects (e. g. shapes or images) is a fundamental problem in vision and graphics.
IGNOR: Image-guided Neural Object Rendering
Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.
On Implicit Filter Level Sparsity in Convolutional Neural Networks
We investigate filter level sparsity that emerges in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained with adaptive gradient descent techniques and L2 regularization or weight decay.
FML: Face Model Learning from Videos
In contrast, we propose multi-frame video-based self-supervised training of a deep network that (i) learns a face identity model both in shape and appearance while (ii) jointly learning to reconstruct 3D faces.
Learning a Disentangled Embedding for Monocular 3D Shape Retrieval and Pose Estimation
We propose a novel approach to jointly perform 3D shape retrieval and pose estimation from monocular images. In order to make the method robust to real-world image variations, e. g. complex textures and backgrounds, we learn an embedding space from 3D data that only includes the relevant information, namely the shape and pose.
Learning to Reconstruct People in Clothing from a Single RGB Camera
We present a learning-based model to infer the personalized 3D shape of people from a few frames (1-8) of a monocular video in which the person is moving, in less than 10 seconds with a reconstruction accuracy of 5mm.
In the Wild Human Pose Estimation Using Explicit 2D Features and Intermediate 3D Representations
Convolutional Neural Network based approaches for monocular 3D human pose estimation usually require a large amount of training images with 3D pose annotations.
Tex2Shape: Detailed Full Human Body Geometry From a Single Image
From a partial texture, we estimate detailed normal and vector displacement maps, which can be applied to a low-resolution smooth body model to add detail and clothing.
IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D Reconstruction
The majority of the existing methods for non-rigid 3D surface regression from monocular 2D images require an object template or point tracks over multiple frames as an input, and are still far from real-time processing rates.
Implicit Filter Sparsification In Convolutional Neural Networks
We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained with adaptive gradient descent techniques and L2 regularization or weight decay.
Emergence of Implicit Filter Sparsity in Convolutional Neural Networks
We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained using adaptive gradient descent techniques with L2 regularization or weight decay.
DEMEA: Deep Mesh Autoencoders for Non-Rigidly Deforming Objects
Mesh autoencoders are commonly used for dimensionality reduction, sampling and mesh modeling.
EgoFace: Egocentric Face Performance Capture and Videorealistic Reenactment
Our lightweight setup allows operations in uncontrolled environments, and lends itself to telepresence applications such as video-conferencing from dynamic environments.
Text-based Editing of Talking-head Video
To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material.
XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera
The first stage is a convolutional neural network (CNN) that estimates 2D and 3D pose features along with identity assignments for all visible joints of all individuals. We contribute a new architecture for this CNN, called SelecSLS Net, that uses novel selective long and short range skip connections to improve the information flow allowing for a drastically faster network without compromising accuracy.
Ranked #7 on
3D Multi-Person Pose Estimation
on MuPoTS-3D
3D Multi-Person Human Pose Estimation
3D Multi-Person Pose Estimation
+1
DispVoxNets: Non-Rigid Point Set Alignment with Supervised Learning Proxies
We introduce a supervised-learning framework for non-rigid point set alignment of a new kind - Displacements on Voxels Networks (DispVoxNets) - which abstracts away from the point set representation and regresses 3D displacement fields on regularly sampled proxy 3D voxel grids.
Real-Time Global Illumination Decomposition of Videos
We propose the first approach for the decomposition of a monocular color video into direct and indirect illumination components in real time.
Multi-Garment Net: Learning to Dress 3D People from Images
We present Multi-Garment Network (MGN), a method to predict body shape and clothing, layered on top of the SMPL model from a few frames (1-8) of a video.
3D Human Pose Estimation
3D Shape Reconstruction From A Single 2D Image
EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera
The high frame rate is a critical requirement for capturing fast human motions.
3D Morphable Face Models -- Past, Present and Future
In this paper, we provide a detailed survey of 3D Morphable Face Models over the 20 years since they were first proposed.
Intrinsic Dynamic Shape Prior for Fast, Sequential and Dense Non-Rigid Structure from Motion with Detection of Temporally-Disjoint Rigidity
The reasons for the slow dissemination are the severe ill-posedness, high sensitivity to motion and deformation cues and the difficulty to obtain reliable point tracks in the vast majority of practical scenarios.
Neural Style-Preserving Visual Dubbing
We present a style-preserving visual dubbing approach from single video inputs, which maintains the signature style of target actors when modifying facial expressions, including mouth motions, to match foreign languages.
Convex Optimisation for Inverse Kinematics
We consider the problem of inverse kinematics (IK), where one wants to find the parameters of a given kinematic skeleton that best explain a set of observed 3D joint locations.
DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data
Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus.
Neural Voice Puppetry: Audio-driven Facial Reenactment
Neural Voice Puppetry has a variety of use-cases, including audio-driven video avatars, video dubbing, and text-driven video synthesis of a talking head.
A Quantum Computational Approach to Correspondence Problems on Point Sets
Modern adiabatic quantum computers (AQC) are already used to solve difficult combinatorial optimisation problems in various domains of science.
Image-guided Neural Object Rendering
Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.
Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation
In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space.
MINA: Convex Mixed-Integer Programming for Non-Rigid Shape Alignment
We present a convex mixed-integer programming formulation for non-rigid shape matching.
DeepCap: Monocular Human Performance Capture Using Weak Supervision
Human performance capture is a highly important computer vision problem with many applications in movie production and virtual/augmented reality.
Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data
We present a novel method for monocular hand shape and pose estimation at unprecedented runtime performance of 100fps and at state-of-the-art accuracy.
StyleRig: Rigging StyleGAN for 3D Control over Portrait Images
StyleGAN generates photorealistic portrait images of faces with eyes, teeth, hair and context (neck, shoulders, background), but lacks a rig-like control over semantic face parameters that are interpretable in 3D, such as face pose, expressions, and scene illumination.
Occlusion-Aware Depth Estimation with Adaptive Normal Constraints
We present a new learning-based method for multi-frame depth estimation from a color video, which is a fundamental problem in scene understanding, robot navigation or handheld 3D reconstruction.
HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Map
The input to our method is a 3D voxelized depth map, and we rely on two hand shape representations.
State of the Art on Neural Rendering
Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e. g., by the integration of differentiable rendering into network training.
Vid2Curve: Simultaneous Camera Motion Estimation and Thin Structure Reconstruction from an RGB Video
We propose the first approach that simultaneously estimates camera motion and reconstructs the geometry of complex 3D thin structures in high quality from a color video captured by a handheld camera.
VideoForensicsHQ: Detecting High-quality Manipulated Face Videos
We introduce a new benchmark dataset for face video forgery detection, of unprecedented quality.
Generative Model-Based Loss to the Rescue: A Method to Overcome Annotation Errors for Depth-Based Hand Pose Estimation
We propose to use a model-based generative loss for training hand pose estimators on depth images based on a volumetric hand model.
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction
In this work, we present methodology that combines detail-rich implicit functions and parametric representations in order to reconstruct 3D models of people that remain controllable and accurate even in the presence of clothing.
Neural Sparse Voxel Fields
We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering.
Face2Face: Real-time Face Capture and Reenactment of RGB Videos
Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.
PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations
At the level of patches, objects across different categories share similarities, which leads to more generalizable models.
PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time
We, therefore, present PhysCap, the first algorithm for physically plausible, real-time and marker-less human 3D motion capture with a single colour camera at 25 fps.
Monocular Reconstruction of Neural Face Reflectance Fields
The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing.
PIE: Portrait Image Embedding for Semantic Control
We present the first approach for embedding real portrait images in the latent space of StyleGAN, which allows for intuitive editing of the head pose, facial expression, and scene illumination in the image.
Fast Gravitational Approach for Rigid Point Set Registration with Ordinary Differential Equations
This article introduces a new physics-based method for rigid point set alignment called Fast Gravitational Approach (FGA).
Learning Complete 3D Morphable Face Models from Images and Videos
Our network design and loss functions ensure a disentangled parameterization of not only identity and albedo, but also, for the first time, an expression basis.
LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration
Formulating this closed loop is not straightforward because it is not trivial to force the output of the NN to be on the surface of the human model - outside this surface the human model is not even defined.
Deep Physics-aware Inference of Cloth Deformation for Monocular Human Performance Capture
Recent monocular human performance capture approaches have shown compelling dense tracking results of the full body from a single RGB camera.
Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
We present a novel method for multi-view depth estimation from a single video, which is a critical task in various applications, such as perception, reconstruction and robot navigation.
i3DMM: Deep Implicit 3D Morphable Model of Human Heads
Our approach has the following favorable properties: (i) It is the first full head morphable model that includes hair.
Pose-Guided Human Animation from a Single Image in the Wild
We present a new pose transfer method for synthesizing a human animation from a single image of a person controlled by a sequence of body poses.
Monocular Real-time Full Body Capture with Inter-part Correlations
We present the first method for real-time full body capture that estimates shape and motion of body and hands together with a dynamic 3D face model from a single color image.
Ranked #11 on
3D Hand Pose Estimation
on FreiHAND
EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream
Due to the different data modality of event cameras compared to classical cameras, existing methods cannot be directly applied to and re-trained for event streams.
High-Fidelity Neural Human Motion Transfer from Monocular Video
We address these limitations for the first time in the literature and present a new framework which performs high-fidelity and temporally-consistent human motion transfer with natural pose-dependent non-rigid deformations, for several types of loose garments.
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video
We show that a single handheld consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views, e. g. a `bullet-time' video effect.
Neural Re-Rendering of Humans from a Single Image
Human re-rendering from a single image is a starkly under-constrained problem, and state-of-the-art algorithms often exhibit undesired artefacts, such as over-smoothing, unrealistic distortions of the body parts and garments, or implausible changes of the texture.
Quantum Permutation Synchronization
We present QuantumSync, the first quantum algorithm for solving a synchronization problem in the context of computer vision.
Learning Speech-driven 3D Conversational Gestures from Video
We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input.
Ranked #4 on
Gesture Generation
on BEAT2
Style and Pose Control for Image Synthesis of Humans from a Single Monocular View
Photo-realistic re-rendering of a human from a single image with explicit control over body pose, shape and appearance enables a wide range of applications, such as human appearance transfer, virtual try-on, motion imitation, and novel view synthesis.
HumanGAN: A Generative Model of Humans Images
We address these limitations and present a generative model for images of dressed humans offering control over pose, local body part appearance and garment style.
PhotoApp: Photorealistic Appearance Editing of Head Portraits
We present an approach for high-quality intuitive editing of the camera viewpoint and scene illumination in a portrait image.
Synthesis of Compositional Animations from Textual Descriptions
Our model can generate plausible pose sequences for short sentences describing single actions as well as long compositional sentences describing multiple sequential and superimposed actions.
Adaptive Surface Normal Constraint for Depth Estimation
We present a novel method for single image depth estimation using surface normal constraints.
Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing
In this paper, we build our work on the aforementioned approaches and propose a new method that greatly improves reconstruction quality and robustness in general scenes.
Ranked #12 on
3D Face Reconstruction
on NoW Benchmark
Efficient and Differentiable Shadow Computation for Inverse Problems
Differentiable rendering has received increasing interest for image-based inverse problems.
Estimating Egocentric 3D Human Pose in Global Space
Furthermore, these methods suffer from limited accuracy and temporal instability due to ambiguities caused by the monocular setup and the severe occlusion in a strongly distorted egocentric perspective.
Ranked #4 on
Egocentric Pose Estimation
on SceneEgo
(using extra training data)
Differentiable Event Stream Simulator for Non-Rigid 3D Tracking
This paper introduces the first differentiable simulator of event streams, i. e., streams of asynchronous brightness change signals recorded by event cameras.
Neural Monocular 3D Human Motion Capture with Physical Awareness
We present a new trainable system for physically plausible markerless 3D human motion capture, which achieves state-of-the-art results in a broad range of challenging scenarios.
Real-time Deep Dynamic Characters
We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery.
Q-Match: Iterative Shape Matching via Quantum Annealing
Finding shape correspondences can be formulated as an NP-hard quadratic assignment problem (QAP) that becomes infeasible for shapes with high sampling density.
Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control
To address this problem, we utilize a coarse body model as the proxy to unwarp the surrounding 3D space into a canonical pose.
Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera
We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands.
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation.
Fast Simultaneous Gravitational Alignment of Multiple Point Sets
The problem of simultaneous rigid alignment of multiple unordered point sets which is unbiased towards any of the inputs has recently attracted increasing interest, and several reliable methods have been newly proposed.
RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video
Moreover, we demonstrate that our approach offers previously unseen two-hand tracking performance from RGB, and quantitatively and qualitatively outperforms existing RGB-based methods that were not explicitly designed for two-hand interactions.
HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks
To address the limitations of the existing methods, we develop HandVoxNet++, i. e., a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner.
NRST: Non-rigid Surface Tracking from Monocular Video
Our texture term exploits the orientation information in the micro-structures of the objects, e. g., the yarn patterns of fabrics.
Egocentric Videoconferencing
Even holding a mobile phone camera in the front of the face while sitting for a long duration is not convenient.
Self-supervised Outdoor Scene Relighting
Outdoor scene relighting is a challenging problem that requires good understanding of the scene geometry, illumination and albedo.
Adiabatic Quantum Graph Matching with Permutation Matrix Constraints
In this work, we address such problems with emerging quantum computing technology and propose several reformulations of QAPs as unconstrained problems suitable for efficient execution on quantum hardware.
StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN
We demonstrate that it suffices to train our temporal architecture on only 10 minutes of footage of 1 subject for about 6 hours.
Neural Rays for Occlusion-aware Image-based Rendering
On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction.
Gravity-Aware Monocular 3D Human-Object Reconstruction
We evaluate GraviCap on a new dataset with ground-truth annotations for persons and different objects undergoing free flights.
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue.
A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis
Motivated by the observation that a 3D object should look realistic from multiple viewpoints, these methods introduce a multi-view constraint as regularization to learn valid 3D radiance fields from 2D images.
Advances in Neural Rendering
The reconstruction of such a scene representation from observations using differentiable rendering losses is known as inverse graphics or inverse rendering.
A Deeper Look into DeepCap
Human performance capture is a highly important computer vision problem with many applications in movie production and virtual/augmented reality.
EgoRenderer: Rendering Human Avatars from Egocentric Camera Images
We next combine the target pose image and the textures into a combined feature image, which is transformed into the output color image using a neural image translation network.
NeRF for Outdoor Scene Relighting
Photorealistic editing of outdoor scenes from photographs requires a profound understanding of the image formation process and an accurate estimation of the scene geometry, reflectance and illumination.
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering
The wide span of viewing positions within these scenes yields multi-scale renderings with very different levels of detail, which poses great challenges to neural radiance field and biases it towards compromised results.
Multimodal Image Synthesis and Editing: The Generative AI Era
With superb power in modeling the interaction among multimodal information, multimodal image synthesis and editing has become a hot research topic in recent years.
f-SfT: Shape-From-Template With a Physics-Based Deformation Model
In contrast to previous works, this paper proposes a new SfT approach explaining 2D observations through physical simulations accounting for forces and material properties.
Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision
Specifically, we first generate pseudo labels for the EgoPW dataset with a spatio-temporal optimization method by incorporating the external-view supervision.
Ranked #4 on
Egocentric Pose Estimation
on GlobalEgoMocap Test Dataset
(using extra training data)
Playable Environments: Video Manipulation in Space and Time
We present Playable Environments - a new representation for interactive video generation and manipulation in space and time.
φ-SfT: Shape-from-Template with a Physics-Based Deformation Model
In contrast to previous works, this paper proposes a new SfT approach explaining 2D observations through physical simulations accounting for forces and material properties.
Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images
Our model can disentangle the geometry and appearance variations in the scene, i. e., we can independently sample from the geometry and appearance spaces of the generative model.
FIRe: Fast Inverse Rendering using Directional and Signed Distance Functions
To this end, we learn a signed distance function (SDF) along with our DDF model to represent a class of shapes.
Direct Dense Pose Estimation
In addition, DDP is computationally more efficient than previous dense pose estimation methods, and it reduces jitters when applied to a video sequence, which is a problem plaguing the previous methods.
BEHAVE: Dataset and Method for Tracking Human Object Interactions
We present BEHAVE dataset, the first full body human- object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them.
HULC: 3D Human Motion Capture with Pose Manifold Sampling and Dense Contact Guidance
Marker-less monocular 3D human motion capture (MoCap) with scene interactions is a challenging research topic relevant for extended reality, robotics and virtual avatar generation.
Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data
With a hybrid architecture that separates static and dynamic contents, fluid interactions with static obstacles are reconstructed for the first time without additional geometry input or human labeling.
Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model
Capturing general deforming scenes from monocular RGB video is crucial for many computer graphics and vision applications.
GAN2X: Non-Lambertian Inverse Rendering of Image GANs
2D images are observations of the 3D physical world depicted with the geometry, material, and illumination components.
EventNeRF: Neural Radiance Fields from a Single Colour Event Camera
Asynchronously operating event cameras find many applications due to their high dynamic range, vanishingly low motion blur, low latency and low data bandwidth.
Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motions
We also observe that an object's intrinsic physical properties are useful for the object motion prediction, and thus design a set of object dynamic descriptors to encode such intrinsic properties.
NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors
The key idea of NeuRIS is to integrate estimated normal of indoor scenes as a prior in a neural rendering framework for reconstructing large texture-less shapes and, importantly, to do this in an adaptive manner to also enable the reconstruction of irregular shapes with fine details.
Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination
Given a set of images of a scene, the re-rendering of this scene from novel views and lighting conditions is an important and challenging problem in Computer Vision and Graphics.
UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture
We present UnrealEgo, i. e., a new large-scale naturalistic dataset for egocentric 3D human pose estimation.
Ranked #2 on
Egocentric Pose Estimation
on UnrealEgo
MoCapDeform: Monocular 3D Human Motion Capture in Deformable Scenes
3D human motion capture from monocular RGB images respecting interactions of a subject with complex and possibly deformable environments is a very challenging, ill-posed and under-explored problem.
Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors
The learned representation can be used to synthesize novel view images of an arbitrary person from a sparse set of cameras, and further animate them with the user's pose control.
Voxurf: Voxel-based Efficient and Accurate Neural Surface Reconstruction
Previous methods based on neural volume rendering mostly train a fully implicit model with MLPs, which typically require hours of training for a single scene.
HandFlow: Quantifying View-Dependent 3D Ambiguity in Two-Hand Reconstruction with Normalizing Flow
Through this, we demonstrate the quality of our probabilistic reconstruction and show that explicit ambiguity modeling is better-suited for this challenging problem.
HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances
Furthermore, we show that HiFECap outperforms the state-of-the-art human performance capture approaches qualitatively and quantitatively while for the first time capturing all aspects of the human.
HDHumans: A Hybrid Approach for High-fidelity Digital Humans
Photo-real digital human avatars are of enormous importance in graphics, as they enable immersive communication over the globe, improve gaming and entertainment experiences, and can be particularly beneficial for AR and VR settings.
State of the Art in Dense Monocular Non-Rigid 3D Reconstruction
3D reconstruction of deformable (or non-rigid) scenes from a set of monocular 2D image observations is a long-standing and actively researched area of computer vision and graphics.
gCoRF: Generative Compositional Radiance Fields
We start with a global generative model (GAN) and learn to decompose it into different semantic parts using supervision from 2D segmentation masks.
Batch-based Model Registration for Fast 3D Sherd Reconstruction
In this work, we aim to develop a portable, high-throughput, and accurate reconstruction system for efficient digitization of fragments excavated in archaeological sites.
An Implicit Parametric Morphable Dental Model
Current dental models use an explicit mesh scene representation and model only the teeth, ignoring the gum.
GlowGAN: Unsupervised Learning of HDR Images from LDR Images in the Wild
Most in-the-wild images are stored in Low Dynamic Range (LDR) form, serving as a partial observation of the High Dynamic Range (HDR) visual world.
NeuralUDF: Learning Unsigned Distance Fields for Multi-view Reconstruction of Surfaces with Arbitrary Topologies
In this paper, we propose to represent surfaces as the Unsigned Distance Function (UDF) and develop a new volume rendering scheme to learn the neural UDF representation.
Fast Non-Rigid Radiance Fields from Monocularized Data
The reconstruction and novel view synthesis of dynamic scenes recently gained increased attention.
MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis
Conventional methods for human motion synthesis are either deterministic or struggle with the trade-off between motion diversity and motion quality.
Ranked #4 on
Motion Synthesis
on AIST++
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction
Recent methods for neural surface representation and rendering, for example NeuS, have demonstrated the remarkably high-quality reconstruction of static scenes.
IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions
Is it possible to synthesize such motion plausibly with a diverse set of objects and instructions?
Scene-aware Egocentric 3D Human Pose Estimation
To this end, we propose an egocentric depth estimation network to predict the scene depth map from a wide-view egocentric fisheye camera while mitigating the occlusion of the human body with a depth-inpainting network.
Ranked #3 on
Egocentric Pose Estimation
on GlobalEgoMocap Test Dataset
(using extra training data)
Imitator: Personalized Speech-driven 3D Facial Animation
To address this, we present Imitator, a speech-driven facial expression synthesis method, which learns identity-specific details from a short input video and produces novel facial expressions matching the identity-specific speaking style and facial idiosyncrasies of the target actor.
F2-NeRF: Fast Neural Radiance Field Training With Free Camera Trajectories
Existing fast grid-based NeRF training frameworks, like Instant-NGP, Plenoxels, DVGO, or TensoRF, are mainly designed for bounded scenes and rely on space warping to handle unbounded scenes.
Scene-Aware 3D Multi-Human Motion Capture from a Single Camera
In this work, we consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera.
NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion
Novel view synthesis from a single image requires inferring occluded regions of objects and scenes whilst simultaneously maintaining semantic and physical consistency with the input.
Regularized Vector Quantization for Tokenized Image Synthesis
The first is a prior distribution regularization which measures the discrepancy between a prior token distribution and the predicted token distribution to avoid codebook collapse and low codebook utilization.
Grid-guided Neural Radiance Fields for Large Urban Scenes
An alternative solution is to use a feature grid representation, which is computationally efficient and can naturally scale to a large scene with increased grid resolutions.
HQ3DAvatar: High Quality Controllable 3D Head Avatar
This paper presents a novel approach to building highly photorealistic digital head avatars.
CCuantuMM: Cycle-Consistent Quantum-Hybrid Matching of Multiple Shapes
Jointly matching multiple, non-rigidly deformed 3D shapes is a challenging, $\mathcal{NP}$-hard problem.
F$^{2}$-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories
Based on our analysis, we further propose a novel space-warping method called perspective warping, which allows us to handle arbitrary trajectories in the grid-based NeRF framework.
GVP: Generative Volumetric Primitives
Advances in 3D-aware generative models have pushed the boundary of image synthesis with explicit camera control.
OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images
Enhancing the robustness of vision algorithms in real-world scenarios is challenging.
EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors
We integrate the two techniques together in EgoLocate, a system that simultaneously performs human motion capture (mocap), localization, and mapping in real time from sparse body-mounted sensors, including 6 inertial measurement units (IMUs) and a monocular phone camera.
General Neural Gauge Fields
In this work, we extend this problem to a general paradigm with a taxonomy of discrete \& continuous cases, and develop a learning framework to jointly optimize gauge transformations and neural fields.
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
Synthesizing visual content that meets users' needs often requires flexible and precise controllability of the pose, shape, expression, and layout of the generated objects.
Weakly Supervised 3D Open-vocabulary Segmentation
Open-vocabulary segmentation of 3D scenes is a fundamental function of human perception and thus a crucial objective in computer vision research.
AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars
Our method edits the full head in a canonical space, and then propagates these edits to remaining time steps via a pretrained deformation network.
VINECS: Video-based Neural Character Skinning
However, the former methods typically predict solely static skinning weights, which perform poorly for highly articulated poses, and the latter ones either require dense 3D character scans in different poses or cannot generate an explicit mesh with vertex correspondence over time.
WaveNeRF: Wavelet-based Generalizable Neural Radiance Fields
Neural Radiance Field (NeRF) has shown impressive performance in novel view synthesis via implicit scene representation.
SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes
We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner.
ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors
Given an unseen object and a reference pose-object pair, we optimise for the object-aware pose that is closest in the feature space to the reference pose.
NeuralClothSim: Neural Deformation Fields Meet the Kirchhoff-Love Thin Shell Theory
Cloth simulation is an extensively studied problem, with a plethora of solutions available in computer graphics literature.
Decaf: Monocular Deformation Capture for Face and Hand Interactions
At the core of our neural approach are a variational auto-encoder supplying the hand-face depth prior and modules that guide the 3D tracking by estimating the contacts and the deformations.
Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering
We further conduct an extensive comparative study of different priors on illumination used in previous work on inverse rendering.
State of the Art on Diffusion Models for Visual Computing
The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes.
Wonder3D: Single Image to 3D using Cross-Domain Diffusion
In this work, we introduce Wonder3D, a novel method for efficiently generating high-fidelity textured meshes from single-view images. Recent methods based on Score Distillation Sampling (SDS) have shown the potential to recover 3D geometry from 2D diffusion priors, but they typically suffer from time-consuming per-shape optimization and inconsistent geometry.
3D-QAE: Fully Quantum Auto-Encoding of 3D Point Clouds
It is trained on collections of 3D point clouds to produce their compressed representations.
DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields
The generated data is applicable across various computer vision tasks, including video segmentation and 3D point cloud segmentation.
Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement
In this work, we explore egocentric whole-body motion capture using a single fisheye camera, which simultaneously estimates human body and hand motion.
Ranked #1 on
Egocentric Pose Estimation
on GlobalEgoMocap Test Dataset
(using extra training data)
ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions
Current approaches for 3D human motion synthesis generate high-quality animations of digital humans performing a wide variety of actions and gestures.
Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models
The experiments demonstrate the superior performance of Surf-D in shape generation across multiple modalities as conditions.
