Search Results for author:
Found 74 papers, 34 papers with code
Human from Blur: Human Pose Tracking from Blurry Images
The key idea is to tackle the inverse problem of image deblurring by modeling the forward problem with a 3D human model, a texture map, and a sequence of poses to describe human motion.
Learning Human-to-Robot Handovers from Point Clouds
We propose the first framework to learn control policies for vision-based human-to-robot handovers, a critical task for human-robot interaction.
Hi4D: 4D Instance Segmentation of Close Human Interaction
We propose Hi4D, a method and dataset for the automatic analysis of physically close human-human interaction under prolonged contact.
Efficient Learning of High Level Plans from Play
Real-world robotic manipulation tasks remain an elusive challenge, since they involve both fine-grained environment interaction, as well as the ability to plan for long-horizon goals.
X-Avatar: Expressive Human Avatars
Our method models bodies, hands, facial expressions and appearance in a holistic fashion and can be learned from either full 3D scans or RGB-D data.
Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition
Specifically, we define a temporally consistent human representation in canonical space and formulate a global optimization over the background model, the canonical human shape and texture, and per-frame human pose parameters.
Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction
This action context together with the next video frame is processed by the multimodal fusion module to forecast the next object interaction.
InstantAvatar: Learning Avatars from Monocular Video in 60 Seconds
To achieve this efficiency we propose a carefully designed and engineered system, that leverages emerging acceleration structures for neural fields, in combination with an efficient empty space-skipping strategy for dynamic scenes.
HARP: Personalized Hand Reconstruction from a Monocular RGB Video
We present HARP (HAnd Reconstruction and Personalization), a personalized hand avatar creation approach that takes a short monocular RGB video of a human hand as input and reconstructs a faithful hand avatar exhibiting a high-fidelity appearance and geometry.
PointAvatar: Deformable Point-based Head Avatars from Videos
The ability to create realistic, animatable and relightable head avatars from casual video sequences would open up wide ranging applications in communication and entertainment.
HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics
We propose a method that leverages graph neural networks, multi-level message passing, and unsupervised training to enable real-time prediction of realistic clothing dynamics.
GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields
We propose GazeNeRF, a 3D-aware method for the task of gaze redirection.
DINER: Depth-aware Image-based NEural Radiance fields
We present Depth-aware Image-based NEural Radiance fields (DINER).
Fast-SNARF: A Fast Deformer for Articulated Neural Fields
A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space.
Robust Real-Time Tracking of Axis-Symmetric Magnets via Neural Networks
We propose to overcome these limitations by using neural networks to infer the marker's position and orientation.
Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors
We further qualitatively evaluate the effectiveness of our method on real images and demonstrate its generalizability towards interaction types and object categories.
TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning
We introduce TempCLR, a new time-coherent contrastive learning approach for the structured regression task of 3D hand reconstruction.
EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes
The challenge of synthesizing eyes is multifold as it requires 1) appropriate representations for the various components of the eye and the periocular region for coherent viewpoint synthesis, capable of representing diffuse, refractive and highly reflective surfaces, 2) disentangling skin and eye appearance from environmental illumination such that it may be rendered under novel lighting conditions, and 3) capturing eyeball motion and the deformation of the surrounding skin to enable re-gazing.
SFP: State-free Priors for Exploration in Off-Policy Reinforcement Learning
Several methods, such as behavioral priors, are able to leverage offline data in order to efficiently accelerate reinforcement learning on complex tasks.
Articulated Objects in Free-form Hand Interaction
Consequently, we introduce ARCTIC - the first dataset of free-form interactions of hands and articulated objects.
LiP-Flow: Learning Inference-time Priors for Codec Avatars via Normalizing Flows in Latent Space
To mitigate this asymmetry, we introduce a prior model that is conditioned on the runtime inputs and tie this prior space to the 3D face model via a normalizing flow in the latent space.
PINA: Learning a Personalized Implicit Neural Avatar from a Single RGB-D Video Sequence
We present a novel method to learn Personalized Implicit Neural Avatars (PINA) from a short RGB-D sequence.
gDNA: Towards Generative Detailed Neural Avatars
Furthermore, we show that our method can be used on the task of fitting human models to raw scans, outperforming the previous state-of-the-art.
SAGA: Stochastic Whole-Body Grasping with Contact
Given an initial pose and the generated whole-body grasping pose as the start and end of the motion respectively, we design a novel contact-aware generative motion infilling module to generate a diverse set of grasp-oriented motions.
I M Avatar: Implicit Morphable Head Avatars from Videos
Traditional 3D morphable face models (3DMMs) provide fine-grained control over expression but cannot easily capture geometric and appearance details.
D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions
We introduce the dynamic grasp synthesis task: given an object with a known 6D pose and a grasp reference, our goal is to generate motions that move the object to a target 6D pose.
Human Performance Capture from Monocular Video in the Wild
In this work, we propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses, without any additional input.
Render In-between: Motion Guided Video Synthesis for Action Interpolation
We propose to address these issues in a motion-guided frame-upsampling framework that is capable of producing realistic human motion and appearance.
Shape-aware Multi-Person Pose Estimation from Multi-View Images
In this paper we contribute a simple yet effective approach for estimating 3D poses of multiple people from multi-view images.
Ranked #16 on
3D Multi-Person Pose Estimation
on Shelf
3D Multi-Person Pose Estimation
Multi-Person Pose Estimation
SPEC: Seeing People in the Wild with an Estimated Camera
We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose.
Ranked #1 on
3D Human Pose Estimation
on AGORA
A Skeleton-Driven Neural Occupancy Representation for Articulated Hands
We present Hand ArticuLated Occupancy (HALO), a novel representation of articulated hands that bridges the advantages of 3D keypoints and neural implicit surfaces and can be used in end-to-end trainable architectures.
Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation
In natural conversation and interaction, our hands often overlap or are in contact with each other.
Ranked #1 on
3D Interacting Hand Pose Estimation
on InterHand2.6M
PeCLR: Self-Supervised 3D Hand Pose Estimation from monocular RGB via Equivariant Contrastive Learning
Encouraged by the success of contrastive learning on image classification tasks, we propose a new self-supervised method for the structured regression task of 3D hand pose estimation.
Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation
Hand pose estimation is difficult due to different environmental conditions, object- and self-occlusion as well as diversity in hand shape and appearance.
PARE: Part Attention Regressor for 3D Human Body Estimation
Despite significant progress, we show that state of the art 3D human pose and shape estimation methods remain sensitive to partial occlusion and can produce dramatically wrong predictions although much of the body is observable.
Ranked #2 on
3D Multi-Person Pose Estimation
on AGORA
3D human pose and shape estimation
3D Multi-Person Pose Estimation
VariTex: Variational Neural Face Textures
In this paper, we propose VariTex - to the best of our knowledge the first method that learns a variational latent feature space of neural face textures, which allows sampling of novel identities.
SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes
However, this is problematic since the backward warp field is pose dependent and thus requires large amounts of data to learn.
Improved Learning of Robot Manipulation Tasks via Tactile Intrinsic Motivation
In this paper we address the challenge of exploration in deep reinforcement learning for robotic manipulation tasks.
The Six Hug Commandments: Design and Evaluation of a Human-Sized Hugging Robot with Visual and Haptic Perception
We followed all six tenets to create a new robotic platform, HuggieBot 2. 0, that has a soft, warm, inflated body (HuggieChest) and uses visual and haptic sensing to deliver closed-loop hugging.
Robotics
Spatial Attention Improves Iterative 6D Object Pose Estimation
Our main insight is that after the initial pose estimate, it is important to pay attention to distinct spatial features of the object in order to improve the estimation accuracy during alignment.
EM-POSE: 3D Human Pose Estimation From Sparse Electromagnetic Trackers
To this end, we present a method to estimate SMPL parameters from 6-12 EM sensors.
Self-Learning Transformations for Improving Gaze and Head Redirection
Furthermore, we show that in the presence of limited amounts of real-world training data, our method allows for improvements in the downstream task of semi-supervised cross-dataset gaze estimation.
Convolutional Autoencoders for Human Motion Infilling
At the heart of our approach lies the idea to cast motion infilling as an inpainting problem and to train a convolutional de-noising autoencoder on image-like representations of motion sequences.
Human Body Model Fitting by Learned Gradient Descent
We propose a novel algorithm for the fitting of 3D human shape to images.
Category Level Object Pose Estimation via Neural Analysis-by-Synthesis
Many object pose estimation algorithms rely on the analysis-by-synthesis framework which requires explicit representations of individual object instances.
ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation
We show that our dataset can significantly improve the robustness of gaze estimation methods across different head poses and gaze angles.
Ranked #1 on
Gaze Estimation
on MPSGaze
Towards End-to-end Video-based Eye-Tracking
Estimating eye-gaze from images alone is a challenging task, in large parts due to un-observable person-specific factors.
CoSE: Compositional Stroke Embeddings
We demonstrate qualitatively and quantitatively that our proposed approach is able to model the appearance of individual strokes, as well as the compositional structure of larger diagram drawings.
A Spatio-temporal Transformer for 3D Human Motion Prediction
We propose a novel Transformer-based architecture for the task of generative modelling of 3D human motion.
Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints
Estimating 3D hand pose from 2D images is a difficult, inverse problem due to the inherent scale and depth ambiguities.
The DIDI dataset: Digital Ink Diagram data
We are releasing a dataset of diagram drawings with dynamic drawing information.
Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
We present HiDe, a novel hierarchical reinforcement learning architecture that successfully solves long horizon control tasks and generalizes to unseen test scenarios.
Hierarchical Reinforcement Learning as a Model of Human Task Interleaving
The results support hierarchical RL as a plausible model of task interleaving.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Content-Consistent Generation of Realistic Eyes with Style
Accurately labeled real-world training data can be scarce, and hence recent works adapt, modify or generate images to boost target datasets.
Structured Prediction Helps 3D Human Motion Modelling
This is implemented via a hierarchy of small-sized neural networks connected analogously to the kinematic chains in the human body as well as a joint-wise decomposition in the loss function.
Learning Functionally Decomposed Hierarchies for Continuous Navigation Tasks
Hierarchical Reinforcement Learning (HRL) has held the promise to enhance the capabilities of RL agents via operation on different levels of temporal abstraction.
Sample Efficient Learning of Path Following and Obstacle Avoidance Behavior for Quadrotors
This enables an easy to implement learning algorithm that is robust to errors of the model used in the model predictive controller.
Demonstration-Guided Deep Reinforcement Learning of Control Policies for Dexterous Human-Robot Interaction
In this paper, we propose a method for training control policies for human-robot interactions such as handshakes or hand claps via Deep Reinforcement Learning.
Few-Shot Adaptive Gaze Estimation
Inter-personal anatomical differences limit the accuracy of person-independent gaze estimation networks.
Ranked #1 on
Gaze Estimation
on MPII Gaze
(using extra training data)
Photo-Realistic Monocular Gaze Redirection Using Generative Adversarial Networks
In this work, we present a novel method to alleviate this problem by leveraging generative adversarial training to synthesize an eye image conditioned on a target gaze direction.
STCN: Stochastic Temporal Convolutional Networks
Convolutional architectures have recently been shown to be competitive on many sequence modelling tasks when compared to the de-facto standard of recurrent neural networks (RNNs), while providing computational and modeling advantages due to inherent parallelism.
Unpaired Pose Guided Human Image Generation
This paper studies the task of full generative modelling of realistic images of humans, guided only by coarse sketch of the pose, while providing control over the specific instance or type of outfit worn by the user.
Monocular Neural Image Based Rendering with Continuous View Control
The approach is self-supervised and only requires 2D images and associated view transforms for training.
End-to-end Learning for Graph Decomposition
The new optimization problem can be viewed as a Conditional Random Field (CRF) in which the random variables are associated with the binary edge labels of the initial graph and the hard constraints are introduced in the CRF as high-order potentials.
Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time
To learn from sufficient data, we synthesize IMU data from motion capture datasets.
Deep Pictorial Gaze Estimation
In this paper, we introduce a novel deep neural network architecture specifically designed for the task of gaze estimation from single eye input.
Learn-to-Score: Efficient 3D Scene Exploration by Predicting View Utility
We propose to learn a better utility function that predicts the usefulness of future viewpoints.
Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings
Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras.
Cross-modal Deep Variational Hand Pose Estimation
Furthermore, we show that our proposed method can be used without changes on depth images and performs comparably to specialized methods.
DeepWriting: Making Digital Ink Editable via Deep Generative Modeling
Digital ink promises to combine the flexibility and aesthetics of handwriting and the ability to process, search and edit digital text.
Guiding InfoGAN with Semi-Supervision
In this paper we propose a new semi-supervised GAN architecture (ss-InfoGAN) for image synthesis that leverages information from few labels (as little as 0. 22%, max.
Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction
We introduce a new method that efficiently computes a set of viewpoints and trajectories for high-quality 3D reconstructions in outdoor environments.
Learning Human Motion Models for Long-term Predictions
Furthermore, we propose new evaluation protocols to assess the quality of synthetic motion sequences even for which no ground truth data exists.
Thin-Slicing Network: A Deep Structured Model for Pose Estimation in Videos
Temporal information can provide additional cues about the location of body joints and help to alleviate these issues.
Ranked #4 on
Pose Estimation
on UPenn Action
