Search Results for author:
Found 121 papers, 65 papers with code
MeshDiffusion: Score-based Generative 3D Mesh Modeling
We consider the task of generating realistic 3D shapes, which is useful for a variety of applications such as automatic scene generation and physical simulation.
Detecting Human-Object Contact in Images
To build HOT, we use two data sources: (1) We use the PROX dataset of 3D human meshes moving in 3D scenes, and automatically annotate 2D image areas for contact via 3D mesh proximity and projection.
PointAvatar: Deformable Point-based Head Avatars from Videos
The ability to create realistic, animatable and relightable head avatars from casual video sequences would open up wide ranging applications in communication and entertainment.
HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics
We propose a method that leverages graph neural networks, multi-level message passing, and unsupervised training to enable real-time prediction of realistic clothing dynamics.
ECON: Explicit Clothed humans Optimized via Normal integration
To increase robustness for these cases, existing work uses an explicit parametric body model to constrain surface reconstruction, but this limits the recovery of free-form surfaces such as loose clothing that deviates from the body.
MIME: Human-Aware 3D Scene Generation
We propose MIME (Mining Interaction and Movement to infer 3D Environments), which is a generative model of indoor scenes that produces furniture layouts that are consistent with the human movement.
Generating Holistic 3D Human Motion from Speech
This work addresses the problem of generating 3D holistic body motions from human speech.
Fast-SNARF: A Fast Deformer for Articulated Neural Fields
A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space.
SUPR: A Sparse Unified Part-Based Human Representation
Using novel 4D scans of feet, we train a model with an extended kinematic tree that captures the range of motion of the toes.
Capturing and Animation of Body and Clothing from Monocular Video
Building on this insight, we propose SCARF (Segmented Clothed Avatar Radiance Field), a hybrid model combining a mesh-based body with a neural radiance field.
SmartMocap: Joint Estimation of Human and Camera Motion using Uncalibrated RGB Cameras
Second, we learn a probability distribution of short human motion sequences ($\sim$1sec) relative to the ground plane and leverage it to disambiguate between the camera and human motion.
InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction
With this method we capture the InterCap dataset, which contains 10 subjects (5 males and 5 females) interacting with 10 objects of various sizes and affordances, including contact with the hands or feet.
Neural Point-based Shape Modeling of Humans in Challenging Clothing
Specifically, we extend point-based methods with a coarse stage, that replaces canonicalization with a learned pose-independent "coarse shape" that can capture the rough surface geometry of clothing like skirts.
TEACH: Temporal Action Composition for 3D Humans
In particular, our goal is to enable the synthesis of a series of actions, which we refer to as temporal action composition.
LED: Latent Variable-based Estimation of Density
Modern generative models are roughly divided into two main categories: (1) models that can produce high-quality random samples, but cannot estimate the exact density of new data points and (2) those that provide exact density estimation, at the expense of sample quality and compactness of the latent space.
Capturing and Inferring Dense Full-Body Human-Scene Contact
We capture a new dataset called RICH for "Real scenes, Interaction, Contact and Humans."
Accurate 3D Body Shape Regression using Metric and Semantic Attributes
Since paired data with images and 3D body shape are rare, we exploit two sources of information: (1) we collect internet images of diverse "fashion" models together with a small set of anthropometric measurements; (2) we collect linguistic shape attributes for a wide range of 3D body meshes and the model images.
Ranked #5 on
3D Human Shape Estimation
on SSP-3D
Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation
Our experimental results show significant improvement compared to state-of-the-art methods on albedo estimation, both in terms of accuracy and fairness.
Articulated Objects in Free-form Hand Interaction
Consequently, we introduce ARCTIC - the first dataset of free-form interactions of hands and articulated objects.
TEMOS: Generating diverse human motions from textual descriptions
In contrast to most previous work which focuses on generating a single, deterministic, motion from a textual description, we design a variational approach that can produce multiple diverse human motions.
EMOCA: Emotion Driven Monocular Face Capture and Animation
While EMOCA achieves 3D reconstruction errors that are on par with the current best methods, it significantly outperforms them in terms of the quality of the reconstructed expression and the perceived emotional content.
Ranked #9 on
3D Face Reconstruction
on REALY (side-view)
OSSO: Obtaining Skeletal Shape from Outside
We address the problem of inferring the anatomic skeleton of a person, in an arbitrary pose, from the 3D surface of the body; i. e. we predict the inside (bones) from the outside (skin).
BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Information
But, even with a better shape model, the problem of regressing dog shape from an image is still challenging because we lack paired images with 3D ground truth.
LocATe: End-to-end Localization of Actions in 3D with Transformers
An important component of this problem is 3D Temporal Action Localization (3D-TAL), which involves recognizing what actions a person is performing, and when.
Human-Aware Object Placement for Visual Environment Reconstruction
In fact, we demonstrate that these human-scene interactions (HSIs) can be leveraged to improve the 3D reconstruction of a scene from a monocular RGB video.
AirPose: Multi-View Fusion Network for Aerial 3D Human Pose and Shape Estimation
In this letter, we present a novel markerless 3D human motion capture (MoCap) system for unstructured, outdoor environments that uses a team of autonomous unmanned aerial vehicles (UAVs) with on-board RGB cameras and computation.
gDNA: Towards Generative Detailed Neural Avatars
Furthermore, we show that our method can be used on the task of fitting human models to raw scans, outperforming the previous state-of-the-art.
Embodied Hands: Modeling and Capturing Hands and Bodies Together
We attach MANO to a standard parameterized 3D body shape model (SMPL), resulting in a fully articulated body and hand model (SMPL+H).
GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping
This is challenging, as it requires the avatar to walk towards the object with foot-ground contact, orient the head towards it, reach out, and grasp it with a realistic hand pose and hand-object contact.
ICON: Implicit Clothed humans Obtained from Normals
First, ICON infers detailed clothed-human normals (front/back) conditioned on the SMPL(-X) normals.
Ranked #1 on
3D Human Reconstruction
on CAPE
Putting People in their Place: Monocular Regression of 3D People in Depth
To do so, we exploit a 3D body model space that lets BEV infer shapes from infants to adults.
Ranked #1 on
3D Depth Estimation
on Relative Human
(using extra training data)
I M Avatar: Implicit Morphable Head Avatars from Videos
Traditional 3D morphable face models (3DMMs) provide fine-grained control over expression but cannot easily capture geometric and appearance details.
InvGAN: Invertible GANs
Our \textbf{InvGAN}, short for Invertible GAN, successfully embeds real images to the latent space of a high quality generative model.
Learning Realistic Human Reposing using Cyclic Self-Supervision with 3D Shape, Pose, and Appearance Consistency
Synthesizing images of a person in novel poses from a single image is a highly ambiguous task.
SOMA: Solving Optical Marker-Based MoCap Automatically
Commercial auto-labeling tools require a specific calibration procedure at capture time, which is not possible for archival data.
Learning to Regress Bodies from Images using Differentiable Semantic Rendering
For Minimally-Clothed regions, we define the DSR-MC loss, which encourages a tight match between a rendered SMPL body and the minimally-clothed regions of the image.
Ranked #14 on
3D Human Pose Estimation
on MPI-INF-3DHP
(PA-MPJPE metric)
SPEC: Seeing People in the Wild with an Estimated Camera
We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose.
Ranked #1 on
3D Human Pose Estimation
on AGORA
The Power of Points for Modeling Humans in Clothing
The geometry feature can be optimized to fit a previously unseen scan of a person in clothing, enabling the scan to be reposed realistically.
Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation
In natural conversation and interaction, our hands often overlap or are in contact with each other.
Ranked #1 on
3D Interacting Hand Pose Estimation
on InterHand2.6M
hSMAL: Detailed Horse Shape and Pose Reconstruction for Motion Pattern Recognition
In this paper we present our preliminary work on model-based behavioral analysis of horse motion.
BABEL: Bodies, Action and Behavior with English Labels
To address this, we present BABEL, a large dataset with language labels describing the actions being performed in mocap sequences.
Ranked #1 on
Action Classification
on BABEL
Collaborative Regression of Expressive Bodies using Moderation
Second, human shape is highly correlated with gender, but existing work ignores this.
Ranked #1 on
3D Multi-Person Mesh Recovery
on AGORA
AGORA: Avatars in Geography Optimized for Regression Analysis
Additionally, we fine-tune methods on AGORA and show improved performance on both AGORA and 3DPW, confirming the realism of the dataset.
PARE: Part Attention Regressor for 3D Human Body Estimation
Despite significant progress, we show that state of the art 3D human pose and shape estimation methods remain sensitive to partial occlusion and can produce dramatically wrong predictions although much of the body is observable.
Ranked #2 on
3D Multi-Person Pose Estimation
on AGORA
3D human pose and shape estimation
3D Multi-Person Pose Estimation
SCALE: Modeling Clothed Humans with a Surface Codec of Articulated Local Elements
We demonstrate the efficacy of our surface representation by learning models of complex clothing from point clouds.
LEAP: Learning Articulated Occupancy of People
Substantial progress has been made on modeling rigid 3D objects using deep implicit representations.
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
By sampling from this latent space and querying a certain duration through a series of positional encodings, we synthesize variable-length motion sequences conditioned on a categorical action.
SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes
However, this is problematic since the backward warp field is pose dependent and thus requires large amounts of data to learn.
On Self-Contact and Human Pose
Third, we develop a novel HPS optimization method, SMPLify-XMC, that includes contact constraints and uses the known 3DCP body pose during fitting to create near ground-truth poses for MTP images.
Ranked #31 on
3D Human Pose Estimation
on 3DPW
(MPJPE metric)
SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks
We present SCANimate, an end-to-end trainable framework that takes raw 3D scans of a clothed human and turns them into an animatable avatar.
Populating 3D Scenes by Learning Human-Scene Interaction
Second, we show that POSA's learned representation of body-scene interaction supports monocular human pose estimation that is consistent with a 3D scene, improving on the state of the art.
Learning an Animatable Detailed 3D Face Model from In-The-Wild Images
Some methods produce faces that cannot be realistically animated because they do not model how wrinkles vary with expression.
We are More than Our Joints: Predicting how 3D Bodies Move
We note that motion prediction methods accumulate errors over time, resulting in joints or markers that diverge from true human bodies.
Monocular, One-stage, Regression of Multiple 3D People
Through a body-center-guided sampling process, the body mesh parameters of all people in the image are easily extracted from the Mesh Parameter map.
Ranked #1 on
3D Multi-Person Mesh Recovery
on Relative Human
(using extra training data)
GRAB: A Dataset of Whole-Body Human Grasping of Objects
Training computers to understand, model, and synthesize human grasping requires a rich dataset containing complex 3D object shapes, detailed contact information, hand pose and shape, and the 3D body motion over time.
Monocular Expressive Body Regression through Body-Driven Attention
To understand how people look, interact, or perform tasks, we need to quickly and accurately capture their 3D body, face, and hands together from an RGB image.
STAR: Sparse Trained Articulated Human Body Regressor
The SMPL body model is widely used for the estimation, synthesis, and analysis of 3D human pose and shape.
SMPLpix: Neural Avatars from 3D Human Models
Recent advances in deep generative models have led to an unprecedented level of realism for synthetically generated images of humans.
PLACE: Proximity Learning of Articulation and Contact in 3D Environments
To synthesize realistic human-scene interactions, it is essential to effectively represent the physical contact and proximity between the body and the world.
Perpetual Motion: Generating Unbounded Human Motion
To address this problem, we propose a model to generate non-deterministic, \textit{ever-changing}, perpetual human motion, in which the global trajectory and the body pose are cross-conditioned.
AirCapRL: Autonomous Aerial Human Motion Capture using Deep Reinforcement Learning
We focus on vision-based MoCap, where the objective is to estimate the trajectory of body pose and shape of a single moving person using multiple micro aerial vehicles.
Chained Representation Cycling: Learning to Estimate 3D Human Pose and Shape by Cycling Between Representations
The goal of many computer vision systems is to transform image pixels into 3D representations.
VIBE: Video Inference for Human Body Pose and Shape Estimation
Human motion is fundamental to understanding behavior.
Ranked #14 on
3D Human Pose Estimation
on MPI-INF-3DHP
Generating 3D People in Scenes without People
However, this is a challenging task for a computer as solving it requires that (1) the generated human bodies to be semantically plausible within the 3D environment (e. g. people sitting on the sofa or cooking near the stove), and (2) the generated human-scene interaction to be physically feasible such that the human body and scene do not interpenetrate while, at the same time, body-scene contact supports physical interactions.
Learning Multi-Human Optical Flow
Therefore, we develop a dataset of multi-human optical flow and train optical flow networks on this dataset.
Attacking Optical Flow
In this paper, we extend adversarial patch attacks to optical flow networks and show that such attacks can compromise their performance.
Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop
Our approach is self-improving by nature, since better network estimates can lead the optimization to better solutions, while more accurate optimization fits provide better supervision for the network.
Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images "In the Wild"
In contrast to research on human pose, shape and texture estimation, training data for endangered species is limited, the animals are in complex natural scenes with occlusion, they are naturally camouflaged, travel in herds, and look similar to each other.
Resolving 3D Human Pose Ambiguities with 3D Scene Constraints
To motivate this, we show that current 3D human pose estimation methods produce results that are not consistent with the 3D scene.
Learning to Dress 3D People in Generative Clothing
To our knowledge, this is the first generative model that directly dresses 3D human body meshes and generalizes to different poses.
Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision
The estimation of 3D face shape from a single image must be robust to variations in lighting, head pose, expression, facial hair, makeup, and occlusions.
Capture, Learning, and Synthesis of 3D Speaking Styles
To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers.
Expressive Body Capture: 3D Hands, Face, and Body from a Single Image
We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild.
Ranked #1 on
3D Human Reconstruction
on Expressive hands and faces dataset (EHF)
(TR V2V (mm), left hand metric)
Learning joint reconstruction of hands and manipulated objects
Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation.
Ranked #10 on
3D Hand Pose Estimation
on FreiHAND
(PA-F@5mm metric)
AMASS: Archive of Motion Capture as Surface Shapes
We achieve this using a new method, MoSh++, that converts mocap data into realistic 3D human meshes represented by a rigged body model; here we use SMPL [doi:10. 1145/2816795. 2818013], which is widely used and provides a standard skeletal representation as well as a fully rigged surface mesh.
Learning and Tracking the 3D Body Shape of Freely Moving Infants from RGB-D sequences
To demonstrate the applicability of SMIL, we fit the model to RGB-D sequences of freely moving infants and show, with a case study, that our method captures enough motion detail for General Movements Assessment (GMA), a method used in clinical practice for early detection of neurodevelopmental disorders in infants.
Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time
To learn from sufficient data, we synthesize IMU data from motion capture datasets.
Temporal Interpolation as an Unsupervised Pretraining Task for Optical Flow Estimation
The difficulty of annotating training data is a major obstacle to using CNNs for low-level tasks in video.
Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera
In this work, we propose a method that combines a single hand-held camera and a set of Inertial Measurement Units (IMUs) attached at the body limbs to estimate accurate 3D poses in the wild.
Generating 3D faces using Convolutional Mesh Autoencoders
To address this, we introduce a versatile model that learns a non-linear representation of a face using spectral convolutions on a mesh surface.
Learning Human Optical Flow
Given this, we devise an optical flow algorithm specifically for human motion and show that it is superior to generic flow methods.
Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape From Images
Animals are widespread in nature and the analysis of their shape and motion is important in many fields and industries.
Resisting Adversarial Attacks using Gaussian Mixture Variational Autoencoders
Our model has the form of a variational autoencoder, with a Gaussian mixture prior on the latent vector.
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation
We address the unsupervised learning of several interconnected problems in low-level vision: single view depth prediction, camera motion estimation, optical flow, and segmentation of a video into the static scene and moving regions.
Ranked #47 on
Monocular Depth Estimation
on KITTI Eigen split
On the Integration of Optical Flow and Action Recognition
Here we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better.
End-to-end Recovery of Human Shape and Pose
The main objective is to minimize the reprojection loss of keypoints, which allow our model to be trained using images in-the-wild that only have ground truth 2D annotations.
Ranked #1 on
Weakly-supervised 3D Human Pose Estimation
on Human3.6M
(3D Annotations metric)
Learning a model of facial shape and expression from 4D scans
FLAME is low-dimensional but more expressive than the FaceWarehouse model and the Basel Face Model.
Ranked #6 on
3D Face Reconstruction
on NoW Benchmark
Towards Accurate Markerless Human Shape and Pose Estimation over Time
Existing marker-less motion capture methods often assume known backgrounds, static cameras, and sequence specific motion priors, which narrows its application scenarios.
Dynamic FAUST: Registering Human Bodies in Motion
We propose a new mesh registration method that uses both 3D geometry and texture information to register all scans in a sequence to a common reference topology.
Slow Flow: Exploiting High-Speed Cameras for Accurate and Diverse Optical Flow Reference Data
Existing optical flow datasets are limited in size and variability due to the difficulty of capturing dense ground truth.
Semantic Multi-View Stereo: Jointly Estimating Objects and Voxels
Due to its probabilistic nature, the approach is able to cope with the approximate geometry of the 3D models as well as input shapes that are not present in the scene.
On human motion prediction using recurrent neural networks
Human motion modelling is a classical problem at the intersection of graphics and computer vision, with applications spanning human-computer interaction, motion synthesis, and motion prediction for virtual and augmented reality.
Optical Flow in Mostly Rigid Scenes
Existing algorithms typically focus on either recovering motion and structure under the assumption of a purely static world or optical flow for general unconstrained scenes.
Ranked #8 on
Optical Flow Estimation
on Sintel-clean
Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs
We address the problem of making human motion capture in the wild more practical by using a small set of inertial sensors attached to the body.
Unite the People: Closing the Loop Between 3D and 2D Human Representations
With a comprehensive set of experiments, we show how this data can be used to train discriminative models that produce results with an unprecedented level of detail: our models predict 31 segments and 91 landmark locations on the body.
Ranked #1 on
Monocular 3D Human Pose Estimation
on Human3.6M
(Use Video Sequence metric)
3D human pose and shape estimation
Monocular 3D Human Pose Estimation
Learning from Synthetic Humans
In this work we present SURREAL (Synthetic hUmans foR REAL tasks): a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data.
3D Menagerie: Modeling the 3D shape and pose of animals
The best human body models are learned from thousands of 3D scans of people in specific poses, which is infeasible with live animals.
Optical Flow Estimation using a Spatial Pyramid Network
We learn to compute optical flow by combining a classical spatial-pyramid formulation with deep learning.
Ranked #8 on
Dense Pixel Correspondence Estimation
on HPatches
Dense Pixel Correspondence Estimation
Optical Flow Estimation
Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
We then fit (top-down) a recently published statistical body shape model, called SMPL, to the 2D joints.
Ranked #27 on
3D Human Pose Estimation
on HumanEva-I
Patches, Planes and Probabilities: A Non-Local Prior for Volumetric 3D Reconstruction
In this paper, we propose a non-local structured prior for volumetric multi-view 3D reconstruction.
Video Segmentation via Object Flow
Video object segmentation is challenging due to fast moving objects, deforming shapes, and cluttered backgrounds.
Ranked #70 on
Semi-Supervised Video Object Segmentation
on DAVIS 2016
Occlusion Boundary Detection via Deep Exploration of Context
Occlusion boundaries contain rich perceptual information about the underlying scene structure.
Optical Flow with Semantic Segmentation and Localized Layers
Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow.
Intrinsic Depth: Improving Depth Transfer With Intrinsic Images
In contrast to raw RGB values, albedo and shading provide a richer, more physical, foundation for depth transfer.
Detailed Full-Body Reconstructions of Moving People From Monocular RGB-D Sequences
The method then uses geometry and image texture over time to obtain accurate shape, pose, and appearance information despite unconstrained motion, partial views, varying resolution, occlusion, and soft tissue deformation.
Efficient Sparse-to-Dense Optical Flow Estimation Using a Learned Basis and Layers
Given a set of sparse matches, we regress to dense optical flow using a learned set of full-frame basis flow fields.
The Stitched Puppet: A Graphical Model of 3D Human Shape and Pose
We propose a new 3D model of the human body that is both realistic and part-based.
Pose-Conditioned Joint Angle Limits for 3D Human Pose Reconstruction
Second, we define a general parametrization of body pose and a new, multi-stage, method to estimate 3D pose from 2D joint locations using an over-complete dictionary of poses.
Ranked #98 on
3D Human Pose Estimation
on Human3.6M
(PA-MPJPE metric)
Model Transport: Towards Scalable Transfer Learning on Manifolds
We demonstrate the approach by transferring PCA and logistic-regression models of real-world data involving 3D shapes and image descriptors.
FAUST: Dataset and Evaluation for 3D Mesh Registration
We address this with a novel mesh registration technique that combines 3D shape and appearance information to produce high-quality alignments.
Grassmann Averages for Scalable Robust PCA
We exploit that averages can be made robust to formulate the Robust Grassmann Average (RGA) as a form of robust PCA.
Model-based anthropometry: Predicting measurements from 3D human scans in multiple poses
Then, we extract features from the registered model (rather than from the scan); these include, limb lengths, circumferences, and statistical features of global shape.
A Fully-Connected Layered Model of Foreground and Background Flow
Layered models allow scene segmentation and motion estimation to be formulated together and to inform one another.
From Deformations to Parts: Motion-based Segmentation of 3D Objects
We develop a method for discovering the parts of an articulated object from aligned meshes capturing various three-dimensional (3D) poses.
A Geometric take on Metric Learning
We then show that this structure gives us a principled way to perform dimensionality reduction and regression according to the learned metrics.
Layered image motion with explicit occlusions, temporal consistency, and depth ordering
We present a new probabilistic model of optical flow in layers that addresses many of the shortcomings of previous approaches.
Synergies in learning words and their referents
This paper presents Bayesian non-parametric models that simultaneously learn to segment words from phoneme strings and learn the referents of some of those words, and shows that there is a synergistic interaction in the acquisition of these two kinds of linguistic information.
A Database and Evaluation Methodology for Optical Flow
The quantitative evaluation of optical flow algorithms by Barron et al. (1994) led to significant advances in performance.
An Additive Latent Feature Model for Transparent Object Recognition
The appearance of a transparent patch is determined in part by the refraction of a background pattern through a transparent medium: the energy from the background usually dominates the patch appearance.
Training Factor Graphs with Reinforcement Learning for Efficient MAP Inference
Large, relational factor graphs with structure defined by first-order logic or other languages give rise to notoriously difficult inference problems.
Explaining human multiple object tracking as resource-constrained approximate inference in a dynamic probabilistic model
Multiple object tracking is a task commonly used to investigate the architecture of human visual attention.
