Search Results for author:
Found 156 papers, 82 papers with code
AWOL: Analysis WithOut synthesis using Language
This involves learning a mapping between the latent space of a vision-language model and the parameter space of the 3D model, which we do using a small set of shape and text pairs.
Explorative Inbetweening of Time and Space
We introduce bounded generation as a generalized task to control video generation to synthesize arbitrary camera and subject motion based only on a given start and end frame.
Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation
To generate composite animations from a multi-track timeline, we propose a new test-time denoising method.
EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
We propose EMAGE, a framework to generate full-body human gestures from audio and masked gestures, encompassing facial, local body, hands, and global movements.
Ranked #1 on
3D Face Animation
on BEAT2
HMP: Hand Motion Priors for Pose and Shape Estimation from Video
Therefore, we develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions.
Environment-Specific People
The method is analyzed quantitatively and qualitatively, and we show that ESP outperforms state-of-the-art on the task of contextual full-body generation.
HAAR: Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles
We present HAAR, a new strand-based generative model for 3D human hairstyles.
WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion
We address these limitations with WHAM (World-grounded Humans with Accurate Motion), which accurately and efficiently reconstructs 3D human motion in a global coordinate system from video.
Ranked #1 on
3D Human Pose Estimation
on EMDB
Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion
Once trained, AMUSE synthesizes 3D human gestures directly from speech with control over the expressed emotions and style by combining the content from the driving speech with the emotion and style of another speech sequence.
PoseGPT: Chatting about 3D Human Pose
We introduce PoseGPT, a framework employing Large Language Models (LLMs) to understand and reason about 3D human poses from images or textual descriptions.
HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video
Since humans interact with diverse objects every day, the holistic 3D capture of these interactions is important to understand and model human behaviour.
Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT).
FLARE: Fast Learning of Animatable and Relightable Mesh Avatars
Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems.
Ghost on the Shell: An Expressive Representation of General 3D Shapes
Recent work has focused on the former, and methods for reconstructing open surfaces do not support fast reconstruction with material and lighting or unconditional generative modelling.
PACE: Human and Camera Motion Estimation from in-the-wild Videos
This design combines the strengths of SLAM and motion priors, which leads to significant improvements in human and camera motion estimation.
Pairwise Similarity Learning is SimPLE
In this paper, we focus on a general yet important learning problem, pairwise similarity learning (PSL).
DECO: Dense Estimation of 3D Human-Scene Contact In The Wild
In contrast, we focus on inferring dense, 3D contact between the full body surface and objects in arbitrary images.
Ranked #2 on
Contact Detection
on BEHAVE
Text-Guided Generation and Editing of Compositional 3D Avatars
We argue that existing methods are limited because they employ a monolithic modeling approach, using a single representation for the head, face, hair, and accessories.
Learning Disentangled Avatars with Hybrid 3D Representations
Towards this end, both explicit and implicit 3D representations are heavily studied for a holistic modeling and capture of the whole human (e. g., body, clothing, face and hair), but neither representation is an optimal choice in terms of representation efficacy since different parts of the human avatar have different modeling desiderata.
POCO: 3D Pose and Shape Estimation with Confidence
To address this, we develop POCO, a novel framework for training HPS regressors to estimate not only a 3D human body, but also their confidence, in a single feed-forward pass.
GRIP: Generating Interaction Poses Using Latent Consistency and Spatial Cues
In contrast, we introduce GRIP, a learning-based method that takes, as input, the 3D motion of the body and the object, and synthesizes realistic motion for both hands before, during, and after object interaction.
SCULPT: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes
We present SCULPT, a novel 3D generative model for clothed and textured 3D meshes of humans.
TADA! Text to Animatable Digital Avatars
We introduce TADA, a simple-yet-effective approach that takes textual descriptions and produces expressive 3D avatars with high-quality geometry and lifelike textures, that can be animated and rendered with traditional graphics pipelines.
Adversarial Likelihood Estimation With One-Way Flows
Generative Adversarial Networks (GANs) can produce high-quality samples, but do not provide an estimate of the probability density around the samples.
BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion
BEDLAM is useful for a variety of tasks and all images, ground truth bodies, 3D clothing, support code, and more are available for research purposes.
Emotional Speech-Driven Animation with Content-Emotion Disentanglement
While the best recent methods generate 3D animations that are synchronized with the input audio, they largely ignore the impact of emotions on facial expressions.
Instant Multi-View Head Capture through Learnable Registration
We use raw MVS scans as supervision during training, but, once trained, TEMPEH directly predicts 3D heads in dense correspondence without requiring scans.
TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments
Although the estimation of 3D human pose and shape (HPS) is rapidly progressing, current methods still cannot reliably estimate moving humans in global coordinates, which is critical for many applications.
Ranked #49 on
3D Human Pose Estimation
on 3DPW
AG3D: Learning to Generate 3D Avatars from 2D Image Collections
The key to progress is hence to learn generative models of 3D avatars from abundant unstructured 2D image collections.
TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis
We show that maintaining the motion generation loss, along with the contrastive training, is crucial to obtain good performance.
Generalizing Neural Human Fitting to Unseen Poses With Articulated SE(3) Equivariance
Learning-based methods address this but do not generalize well when the input pose is far from those seen during training.
SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation
Motivated by the observation that the correspondence between actions and body parts is encoded in powerful language models, we extract this knowledge by prompting GPT-3 with text such as "what are the body parts involved in the action <action name>?
Reconstructing Signing Avatars From Video Using Linguistic Priors
A perceptual study shows that SGNify's 3D reconstructions are significantly more comprehensible and natural than those of previous methods and are on par with the source videos.
3D Human Pose Estimation via Intuitive Physics
Inspired by biomechanics, we infer the pressure heatmap on the body, the Center of Pressure (CoP) from the heatmap, and the SMPL body's Center of Mass (CoM).
Ranked #1 on
3D Human Pose Estimation
on RICH
MeshDiffusion: Score-based Generative 3D Mesh Modeling
We consider the task of generating realistic 3D shapes, which is useful for a variety of applications such as automatic scene generation and physical simulation.
Detecting Human-Object Contact in Images
To build HOT, we use two data sources: (1) We use the PROX dataset of 3D human meshes moving in 3D scenes, and automatically annotate 2D image areas for contact via 3D mesh proximity and projection.
BITE: Beyond Priors for Improved Three-D Dog Pose Estimation
To that end, we exploit contact with the ground as a form of side information.
PointAvatar: Deformable Point-based Head Avatars from Videos
The ability to create realistic, animatable and relightable head avatars from casual video sequences would open up wide ranging applications in communication and entertainment.
HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics
We propose a method that leverages graph neural networks, multi-level message passing, and unsupervised training to enable real-time prediction of realistic clothing dynamics.
ECON: Explicit Clothed humans Optimized via Normal integration
To increase robustness for these cases, existing work uses an explicit parametric body model to constrain surface reconstruction, but this limits the recovery of free-form surfaces such as loose clothing that deviates from the body.
Generating Holistic 3D Human Motion from Speech
This work addresses the problem of generating 3D holistic body motions from human speech.
Ranked #2 on
Gesture Generation
on BEAT2
MIME: Human-Aware 3D Scene Generation
We propose MIME (Mining Interaction and Movement to infer 3D Environments), which is a generative model of indoor scenes that produces furniture layouts that are consistent with the human movement.
Ranked #2 on
Indoor Scene Synthesis
on PRO-teXt
2D Semantic Segmentation task 1 (8 classes)
3D Semantic Scene Completion
+2
Fast-SNARF: A Fast Deformer for Articulated Neural Fields
A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space.
SUPR: A Sparse Unified Part-Based Human Representation
Using novel 4D scans of feet, we train a model with an extended kinematic tree that captures the range of motion of the toes.
Capturing and Animation of Body and Clothing from Monocular Video
Building on this insight, we propose SCARF (Segmented Clothed Avatar Radiance Field), a hybrid model combining a mesh-based body with a neural radiance field.
SmartMocap: Joint Estimation of Human and Camera Motion using Uncalibrated RGB Cameras
Second, we learn a probability distribution of short human motion sequences ($\sim$1sec) relative to the ground plane and leverage it to disambiguate between the camera and human motion.
InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction
With this method we capture the InterCap dataset, which contains 10 subjects (5 males and 5 females) interacting with 10 objects of various sizes and affordances, including contact with the hands or feet.
Neural Point-based Shape Modeling of Humans in Challenging Clothing
Specifically, we extend point-based methods with a coarse stage, that replaces canonicalization with a learned pose-independent "coarse shape" that can capture the rough surface geometry of clothing like skirts.
TEACH: Temporal Action Composition for 3D Humans
In particular, our goal is to enable the synthesis of a series of actions, which we refer to as temporal action composition.
LED: Latent Variable-based Estimation of Density
Modern generative models are roughly divided into two main categories: (1) models that can produce high-quality random samples, but cannot estimate the exact density of new data points and (2) those that provide exact density estimation, at the expense of sample quality and compactness of the latent space.
Capturing and Inferring Dense Full-Body Human-Scene Contact
We capture a new dataset called RICH for "Real scenes, Interaction, Contact and Humans."
Ranked #3 on
Contact Detection
on BEHAVE
Accurate 3D Body Shape Regression using Metric and Semantic Attributes
Since paired data with images and 3D body shape are rare, we exploit two sources of information: (1) we collect internet images of diverse "fashion" models together with a small set of anthropometric measurements; (2) we collect linguistic shape attributes for a wide range of 3D body meshes and the model images.
Ranked #6 on
3D Human Shape Estimation
on SSP-3D
Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation
Our experimental results show significant improvement compared to state-of-the-art methods on albedo estimation, both in terms of accuracy and fairness.
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation
In part this is because there exist no datasets with ground-truth 3D annotations for the study of physically consistent and synchronised motion of hands and articulated objects.
TEMOS: Generating diverse human motions from textual descriptions
In contrast to most previous work which focuses on generating a single, deterministic, motion from a textual description, we design a variational approach that can produce multiple diverse human motions.
EMOCA: Emotion Driven Monocular Face Capture and Animation
While EMOCA achieves 3D reconstruction errors that are on par with the current best methods, it significantly outperforms them in terms of the quality of the reconstructed expression and the perceived emotional content.
Ranked #13 on
3D Face Reconstruction
on REALY (side-view)
OSSO: Obtaining Skeletal Shape from Outside
We address the problem of inferring the anatomic skeleton of a person, in an arbitrary pose, from the 3D surface of the body; i. e. we predict the inside (bones) from the outside (skin).
BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Information
But, even with a better shape model, the problem of regressing dog shape from an image is still challenging because we lack paired images with 3D ground truth.
LocATe: End-to-end Localization of Actions in 3D with Transformers
An important component of this problem is 3D Temporal Action Localization (3D-TAL), which involves recognizing what actions a person is performing, and when.
Human-Aware Object Placement for Visual Environment Reconstruction
In fact, we demonstrate that these human-scene interactions (HSIs) can be leveraged to improve the 3D reconstruction of a scene from a monocular RGB video.
AirPose: Multi-View Fusion Network for Aerial 3D Human Pose and Shape Estimation
In this letter, we present a novel markerless 3D human motion capture (MoCap) system for unstructured, outdoor environments that uses a team of autonomous unmanned aerial vehicles (UAVs) with on-board RGB cameras and computation.
gDNA: Towards Generative Detailed Neural Avatars
Furthermore, we show that our method can be used on the task of fitting human models to raw scans, outperforming the previous state-of-the-art.
Embodied Hands: Modeling and Capturing Hands and Bodies Together
We attach MANO to a standard parameterized 3D body shape model (SMPL), resulting in a fully articulated body and hand model (SMPL+H).
GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping
This is challenging, as it requires the avatar to walk towards the object with foot-ground contact, orient the head towards it, reach out, and grasp it with a realistic hand pose and hand-object contact.
ICON: Implicit Clothed humans Obtained from Normals
First, ICON infers detailed clothed-human normals (front/back) conditioned on the SMPL(-X) normals.
Ranked #1 on
3D Human Reconstruction
on CAPE
Putting People in their Place: Monocular Regression of 3D People in Depth
To do so, we exploit a 3D body model space that lets BEV infer shapes from infants to adults.
Ranked #1 on
3D Depth Estimation
on Relative Human
(using extra training data)
I M Avatar: Implicit Morphable Head Avatars from Videos
Traditional 3D morphable face models (3DMMs) provide fine-grained control over expression but cannot easily capture geometric and appearance details.
InvGAN: Invertible GANs
Our \textbf{InvGAN}, short for Invertible GAN, successfully embeds real images to the latent space of a high quality generative model.
Learning Realistic Human Reposing using Cyclic Self-Supervision with 3D Shape, Pose, and Appearance Consistency
Synthesizing images of a person in novel poses from a single image is a highly ambiguous task.
SOMA: Solving Optical Marker-Based MoCap Automatically
Commercial auto-labeling tools require a specific calibration procedure at capture time, which is not possible for archival data.
Learning to Regress Bodies from Images using Differentiable Semantic Rendering
For Minimally-Clothed regions, we define the DSR-MC loss, which encourages a tight match between a rendered SMPL body and the minimally-clothed regions of the image.
Ranked #52 on
3D Human Pose Estimation
on 3DPW
(using extra training data)
SPEC: Seeing People in the Wild with an Estimated Camera
We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose.
Ranked #1 on
3D Multi-Person Pose Estimation
on AGORA
The Power of Points for Modeling Humans in Clothing
The geometry feature can be optimized to fit a previously unseen scan of a person in clothing, enabling the scan to be reposed realistically.
Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation
In natural conversation and interaction, our hands often overlap or are in contact with each other.
Ranked #7 on
3D Interacting Hand Pose Estimation
on InterHand2.6M
hSMAL: Detailed Horse Shape and Pose Reconstruction for Motion Pattern Recognition
In this paper we present our preliminary work on model-based behavioral analysis of horse motion.
BABEL: Bodies, Action and Behavior with English Labels
To address this, we present BABEL, a large dataset with language labels describing the actions being performed in mocap sequences.
Ranked #1 on
Action Classification
on BABEL
Collaborative Regression of Expressive Bodies using Moderation
Second, human shape is highly correlated with gender, but existing work ignores this.
AGORA: Avatars in Geography Optimized for Regression Analysis
Additionally, we fine-tune methods on AGORA and show improved performance on both AGORA and 3DPW, confirming the realism of the dataset.
PARE: Part Attention Regressor for 3D Human Body Estimation
Despite significant progress, we show that state of the art 3D human pose and shape estimation methods remain sensitive to partial occlusion and can produce dramatically wrong predictions although much of the body is observable.
Ranked #2 on
3D Multi-Person Pose Estimation
on AGORA
3D human pose and shape estimation
3D Multi-Person Pose Estimation
SCALE: Modeling Clothed Humans with a Surface Codec of Articulated Local Elements
We demonstrate the efficacy of our surface representation by learning models of complex clothing from point clouds.
LEAP: Learning Articulated Occupancy of People
Substantial progress has been made on modeling rigid 3D objects using deep implicit representations.
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
By sampling from this latent space and querying a certain duration through a series of positional encodings, we synthesize variable-length motion sequences conditioned on a categorical action.
SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes
However, this is problematic since the backward warp field is pose dependent and thus requires large amounts of data to learn.
On Self-Contact and Human Pose
Third, we develop a novel HPS optimization method, SMPLify-XMC, that includes contact constraints and uses the known 3DCP body pose during fitting to create near ground-truth poses for MTP images.
Ranked #73 on
3D Human Pose Estimation
on 3DPW
(MPJPE metric)
SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks
We present SCANimate, an end-to-end trainable framework that takes raw 3D scans of a clothed human and turns them into an animatable avatar.
Populating 3D Scenes by Learning Human-Scene Interaction
Second, we show that POSA's learned representation of body-scene interaction supports monocular human pose estimation that is consistent with a 3D scene, improving on the state of the art.
Ranked #4 on
Contact Detection
on BEHAVE
Learning an Animatable Detailed 3D Face Model from In-The-Wild Images
Some methods produce faces that cannot be realistically animated because they do not model how wrinkles vary with expression.
We are More than Our Joints: Predicting how 3D Bodies Move
We note that motion prediction methods accumulate errors over time, resulting in joints or markers that diverge from true human bodies.
Monocular, One-stage, Regression of Multiple 3D People
Through a body-center-guided sampling process, the body mesh parameters of all people in the image are easily extracted from the Mesh Parameter map.
Ranked #1 on
3D Multi-Person Mesh Recovery
on Relative Human
(using extra training data)
GRAB: A Dataset of Whole-Body Human Grasping of Objects
Training computers to understand, model, and synthesize human grasping requires a rich dataset containing complex 3D object shapes, detailed contact information, hand pose and shape, and the 3D body motion over time.
Monocular Expressive Body Regression through Body-Driven Attention
To understand how people look, interact, or perform tasks, we need to quickly and accurately capture their 3D body, face, and hands together from an RGB image.
STAR: Sparse Trained Articulated Human Body Regressor
The SMPL body model is widely used for the estimation, synthesis, and analysis of 3D human pose and shape.
SMPLpix: Neural Avatars from 3D Human Models
Recent advances in deep generative models have led to an unprecedented level of realism for synthetically generated images of humans.
PLACE: Proximity Learning of Articulation and Contact in 3D Environments
To synthesize realistic human-scene interactions, it is essential to effectively represent the physical contact and proximity between the body and the world.
Perpetual Motion: Generating Unbounded Human Motion
To address this problem, we propose a model to generate non-deterministic, \textit{ever-changing}, perpetual human motion, in which the global trajectory and the body pose are cross-conditioned.
AirCapRL: Autonomous Aerial Human Motion Capture using Deep Reinforcement Learning
We focus on vision-based MoCap, where the objective is to estimate the trajectory of body pose and shape of a single moving person using multiple micro aerial vehicles.
Chained Representation Cycling: Learning to Estimate 3D Human Pose and Shape by Cycling Between Representations
The goal of many computer vision systems is to transform image pixels into 3D representations.
VIBE: Video Inference for Human Body Pose and Shape Estimation
Human motion is fundamental to understanding behavior.
Ranked #34 on
Monocular 3D Human Pose Estimation
on Human3.6M
Generating 3D People in Scenes without People
However, this is a challenging task for a computer as solving it requires that (1) the generated human bodies to be semantically plausible within the 3D environment (e. g. people sitting on the sofa or cooking near the stove), and (2) the generated human-scene interaction to be physically feasible such that the human body and scene do not interpenetrate while, at the same time, body-scene contact supports physical interactions.
Learning Multi-Human Optical Flow
Therefore, we develop a dataset of multi-human optical flow and train optical flow networks on this dataset.
Attacking Optical Flow
In this paper, we extend adversarial patch attacks to optical flow networks and show that such attacks can compromise their performance.
Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop
Our approach is self-improving by nature, since better network estimates can lead the optimization to better solutions, while more accurate optimization fits provide better supervision for the network.
Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images "In the Wild"
In contrast to research on human pose, shape and texture estimation, training data for endangered species is limited, the animals are in complex natural scenes with occlusion, they are naturally camouflaged, travel in herds, and look similar to each other.
Resolving 3D Human Pose Ambiguities with 3D Scene Constraints
To motivate this, we show that current 3D human pose estimation methods produce results that are not consistent with the 3D scene.
Learning to Dress 3D People in Generative Clothing
To our knowledge, this is the first generative model that directly dresses 3D human body meshes and generalizes to different poses.
Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision
The estimation of 3D face shape from a single image must be robust to variations in lighting, head pose, expression, facial hair, makeup, and occlusions.
Capture, Learning, and Synthesis of 3D Speaking Styles
To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers.
Expressive Body Capture: 3D Hands, Face, and Body from a Single Image
We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild.
Ranked #1 on
3D Human Reconstruction
on Expressive hands and faces dataset (EHF)
(TR V2V (mm), left hand metric)
Learning joint reconstruction of hands and manipulated objects
Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation.
Ranked #7 on
hand-object pose
on DexYCB
AMASS: Archive of Motion Capture as Surface Shapes
We achieve this using a new method, MoSh++, that converts mocap data into realistic 3D human meshes represented by a rigged body model; here we use SMPL [doi:10. 1145/2816795. 2818013], which is widely used and provides a standard skeletal representation as well as a fully rigged surface mesh.
Learning and Tracking the 3D Body Shape of Freely Moving Infants from RGB-D sequences
To demonstrate the applicability of SMIL, we fit the model to RGB-D sequences of freely moving infants and show, with a case study, that our method captures enough motion detail for General Movements Assessment (GMA), a method used in clinical practice for early detection of neurodevelopmental disorders in infants.
Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time
To learn from sufficient data, we synthesize IMU data from motion capture datasets.
Temporal Interpolation as an Unsupervised Pretraining Task for Optical Flow Estimation
The difficulty of annotating training data is a major obstacle to using CNNs for low-level tasks in video.
Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera
In this work, we propose a method that combines a single hand-held camera and a set of Inertial Measurement Units (IMUs) attached at the body limbs to estimate accurate 3D poses in the wild.
Generating 3D faces using Convolutional Mesh Autoencoders
To address this, we introduce a versatile model that learns a non-linear representation of a face using spectral convolutions on a mesh surface.
Ranked #4 on
Face Alignment
on FaceScape
Learning Human Optical Flow
Given this, we devise an optical flow algorithm specifically for human motion and show that it is superior to generic flow methods.
Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape From Images
Animals are widespread in nature and the analysis of their shape and motion is important in many fields and industries.
Resisting Adversarial Attacks using Gaussian Mixture Variational Autoencoders
Our model has the form of a variational autoencoder, with a Gaussian mixture prior on the latent vector.
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation
We address the unsupervised learning of several interconnected problems in low-level vision: single view depth prediction, camera motion estimation, optical flow, and segmentation of a video into the static scene and moving regions.
Ranked #65 on
Monocular Depth Estimation
on KITTI Eigen split
On the Integration of Optical Flow and Action Recognition
Here we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better.
End-to-end Recovery of Human Shape and Pose
The main objective is to minimize the reprojection loss of keypoints, which allow our model to be trained using images in-the-wild that only have ground truth 2D annotations.
Ranked #1 on
Weakly-supervised 3D Human Pose Estimation
on Human3.6M
(3D Annotations metric)
Learning a model of facial shape and expression from 4D scans
FLAME is low-dimensional but more expressive than the FaceWarehouse model and the Basel Face Model.
Ranked #3 on
Face Alignment
on FaceScape
Towards Accurate Markerless Human Shape and Pose Estimation over Time
Existing marker-less motion capture methods often assume known backgrounds, static cameras, and sequence specific motion priors, which narrows its application scenarios.
Slow Flow: Exploiting High-Speed Cameras for Accurate and Diverse Optical Flow Reference Data
Existing optical flow datasets are limited in size and variability due to the difficulty of capturing dense ground truth.
Dynamic FAUST: Registering Human Bodies in Motion
We propose a new mesh registration method that uses both 3D geometry and texture information to register all scans in a sequence to a common reference topology.
Semantic Multi-View Stereo: Jointly Estimating Objects and Voxels
Due to its probabilistic nature, the approach is able to cope with the approximate geometry of the 3D models as well as input shapes that are not present in the scene.
On human motion prediction using recurrent neural networks
Human motion modelling is a classical problem at the intersection of graphics and computer vision, with applications spanning human-computer interaction, motion synthesis, and motion prediction for virtual and augmented reality.
Optical Flow in Mostly Rigid Scenes
Existing algorithms typically focus on either recovering motion and structure under the assumption of a purely static world or optical flow for general unconstrained scenes.
Ranked #13 on
Optical Flow Estimation
on Sintel-clean
Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs
We address the problem of making human motion capture in the wild more practical by using a small set of inertial sensors attached to the body.
Unite the People: Closing the Loop Between 3D and 2D Human Representations
With a comprehensive set of experiments, we show how this data can be used to train discriminative models that produce results with an unprecedented level of detail: our models predict 31 segments and 91 landmark locations on the body.
Ranked #1 on
Monocular 3D Human Pose Estimation
on Human3.6M
(Use Video Sequence metric)
3D human pose and shape estimation
Monocular 3D Human Pose Estimation
Learning from Synthetic Humans
In this work we present SURREAL (Synthetic hUmans foR REAL tasks): a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data.
3D Menagerie: Modeling the 3D shape and pose of animals
The best human body models are learned from thousands of 3D scans of people in specific poses, which is infeasible with live animals.
Optical Flow Estimation using a Spatial Pyramid Network
We learn to compute optical flow by combining a classical spatial-pyramid formulation with deep learning.
Ranked #8 on
Dense Pixel Correspondence Estimation
on HPatches
Dense Pixel Correspondence Estimation
Optical Flow Estimation
Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
We then fit (top-down) a recently published statistical body shape model, called SMPL, to the 2D joints.
Ranked #29 on
3D Human Pose Estimation
on HumanEva-I
Occlusion Boundary Detection via Deep Exploration of Context
Occlusion boundaries contain rich perceptual information about the underlying scene structure.
Patches, Planes and Probabilities: A Non-Local Prior for Volumetric 3D Reconstruction
In this paper, we propose a non-local structured prior for volumetric multi-view 3D reconstruction.
Video Segmentation via Object Flow
Video object segmentation is challenging due to fast moving objects, deforming shapes, and cluttered backgrounds.
Ranked #74 on
Semi-Supervised Video Object Segmentation
on DAVIS 2016
(using extra training data)
Optical Flow with Semantic Segmentation and Localized Layers
Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow.
Intrinsic Depth: Improving Depth Transfer With Intrinsic Images
In contrast to raw RGB values, albedo and shading provide a richer, more physical, foundation for depth transfer.
Detailed Full-Body Reconstructions of Moving People From Monocular RGB-D Sequences
The method then uses geometry and image texture over time to obtain accurate shape, pose, and appearance information despite unconstrained motion, partial views, varying resolution, occlusion, and soft tissue deformation.
Efficient Sparse-to-Dense Optical Flow Estimation Using a Learned Basis and Layers
Given a set of sparse matches, we regress to dense optical flow using a learned set of full-frame basis flow fields.
The Stitched Puppet: A Graphical Model of 3D Human Shape and Pose
We propose a new 3D model of the human body that is both realistic and part-based.
Pose-Conditioned Joint Angle Limits for 3D Human Pose Reconstruction
Second, we define a general parametrization of body pose and a new, multi-stage, method to estimate 3D pose from 2D joint locations using an over-complete dictionary of poses.
Ranked #121 on
3D Human Pose Estimation
on Human3.6M
(PA-MPJPE metric)
Grassmann Averages for Scalable Robust PCA
We exploit that averages can be made robust to formulate the Robust Grassmann Average (RGA) as a form of robust PCA.
Model Transport: Towards Scalable Transfer Learning on Manifolds
We demonstrate the approach by transferring PCA and logistic-regression models of real-world data involving 3D shapes and image descriptors.
FAUST: Dataset and Evaluation for 3D Mesh Registration
We address this with a novel mesh registration technique that combines 3D shape and appearance information to produce high-quality alignments.
Model-based anthropometry: Predicting measurements from 3D human scans in multiple poses
Then, we extract features from the registered model (rather than from the scan); these include, limb lengths, circumferences, and statistical features of global shape.
A Fully-Connected Layered Model of Foreground and Background Flow
Layered models allow scene segmentation and motion estimation to be formulated together and to inform one another.
A Geometric take on Metric Learning
We then show that this structure gives us a principled way to perform dimensionality reduction and regression according to the learned metrics.
From Deformations to Parts: Motion-based Segmentation of 3D Objects
We develop a method for discovering the parts of an articulated object from aligned meshes capturing various three-dimensional (3D) poses.
Synergies in learning words and their referents
This paper presents Bayesian non-parametric models that simultaneously learn to segment words from phoneme strings and learn the referents of some of those words, and shows that there is a synergistic interaction in the acquisition of these two kinds of linguistic information.
Layered image motion with explicit occlusions, temporal consistency, and depth ordering
We present a new probabilistic model of optical flow in layers that addresses many of the shortcomings of previous approaches.
A Database and Evaluation Methodology for Optical Flow
The quantitative evaluation of optical flow algorithms by Barron et al. (1994) led to significant advances in performance.
An Additive Latent Feature Model for Transparent Object Recognition
The appearance of a transparent patch is determined in part by the refraction of a background pattern through a transparent medium: the energy from the background usually dominates the patch appearance.
Training Factor Graphs with Reinforcement Learning for Efficient MAP Inference
Large, relational factor graphs with structure defined by first-order logic or other languages give rise to notoriously difficult inference problems.
Explaining human multiple object tracking as resource-constrained approximate inference in a dynamic probabilistic model
Multiple object tracking is a task commonly used to investigate the architecture of human visual attention.
