Search Results for author:
Found 112 papers, 52 papers with code
WorldPose: A World Cup Dataset for Global 3D Human Pose Estimation
We present WorldPose, a novel dataset for advancing research in multi-person global pose estimation in the wild, featuring footage from the 2022 FIFA World Cup.
Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models
Classifier-free guidance (CFG) is crucial for improving both generation quality and alignment between the input condition and final output in diffusion models.
Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures
We then train a conditional Neural Radiance Field prior on this synthetic dataset and, at inference time, fine-tune the model on a very sparse set of real images of a single subject.
Human Hair Reconstruction with Strand-Aligned 3D Gaussians
We introduce a new hair modeling method that uses a dual representation of classical hair strands and 3D Gaussians to produce accurate and realistic strand-based reconstructions from multi-view data.
ReLoo: Reconstructing Humans Dressed in Loose Garments from Monocular Video in the Wild
Our method, ReLoo, overcomes this limitation and reconstructs high-quality 3D models of humans dressed in loose garments from monocular in-the-wild videos.
Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video
We introduce Gaussian Garments, a novel approach for reconstructing realistic simulation-ready garment assets from multi-view videos.
AvatarPose: Avatar-guided 3D Pose Estimation of Close Human Interaction from Sparse Multi-view Videos
Despite progress in human motion capture, existing multi-view methods often face challenges in estimating the 3D pose and shape of multiple closely interacting people.
No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models
Additionally, by leveraging the time-step information encoded in all diffusion networks, we propose an extension of CFG, called time-step guidance (TSG), which can be applied to any diffusion model, including unconditional ones.
EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting
Human activities are inherently complex, often involving numerous object interactions.
RILe: Reinforced Imitation Learning
The trainer optimizes for long-term cumulative rewards from the discriminator, enabling it to provide nuanced feedback that accounts for the complexity of the task and the student's current capabilities.
MultiPly: Reconstruction of Multiple People from Monocular Video in the Wild
We present MultiPly, a novel framework to reconstruct multiple people in 3D from monocular in-the-wild videos.
LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent Diffusion Models
Advances in latent diffusion models (LDMs) have revolutionized high-resolution image generation, but the design space of the autoencoder that is central to these systems remains underexplored.
ContourCraft: Learning to Resolve Intersections in Neural Multi-Garment Simulations
In this work, we present \moniker{}, a learning-based solution for handling intersections in neural cloth simulations.
4D-DRESS: A 4D Dataset of Real-world Human Clothing with Semantic Annotations
Addressing this gap, we introduce 4D-DRESS, the first real-world 4D dataset advancing human clothing research with its high-quality 4D textured scans and garment meshes.
WANDR: Intention-guided Human Motion Generation
To address this, we introduce WANDR, a data-driven model that takes an avatar's initial pose and a goal's 3D position and generates natural human motions that place the end effector (wrist) on the goal location.
GraspXL: Generating Grasping Motions for Diverse Objects at Scale
Moreover, we show that our framework can be deployed to different dexterous hands and work with reconstructed or generated objects.
Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects
A holistic 3Dunderstanding of such interactions from egocentric views is important for tasks in robotics, AR/VR, action recognition and motion generation.
RHOBIN Challenge: Reconstruction of Human Object Interaction
Modeling the interaction between humans and objects has been an emerging research direction in recent years.
Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles
We present HAAR a new strand-based generative model for 3D human hairstyles.
HAAR: Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles
We present HAAR, a new strand-based generative model for 3D human hairstyles.
G-MEMP: Gaze-Enhanced Multimodal Ego-Motion Prediction in Driving
In contrast, we focus on inferring the ego trajectory of a driver's vehicle using their gaze data.
HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video
Since humans interact with diverse objects every day, the holistic 3D capture of these interactions is important to understand and model human behaviour.
PALM: Predicting Actions through Language Models
Traditional methods heavily rely on representation learning that is trained on a large amount of video data.
A Unified Approach for Text- and Image-guided 4D Scene Generation
Large-scale diffusion generative models are greatly simplifying image, video and 3D asset creation from user-provided text prompts and images.
SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion
For the former, we employ a powerful generative diffusion model to hallucinate unseen back-view appearance based on the input images.
Ranked #1 on
3D Human Reconstruction
on CustomHumans
SynH2R: Synthesizing Hand-Object Motions for Learning Human-to-Robot Handovers
In this paper, we introduce a framework that can generate plausible human grasping motions suitable for training the robot.
CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling
While conditional diffusion models are known to have good coverage of the data distribution, they still face limitations in output diversity, particularly when sampled with a high classifier-free guidance scale for optimal image quality or when trained on small datasets.
Ranked #14 on
Image Generation
on ImageNet 256x256
FLARE: Fast Learning of Animatable and Relightable Mesh Avatars
Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems.
PACE: Human and Camera Motion Estimation from in-the-wild Videos
This design combines the strengths of SLAM and motion priors, which leads to significant improvements in human and camera motion estimation.
Preface: A Data-driven Volumetric Prior for Few-shot Ultra High-resolution Face Synthesis
NeRFs have enabled highly realistic synthesis of human faces including complex appearance and reflectance effects of hair and skin.
Physically Plausible Full-Body Hand-Object Interaction Synthesis
Through a hierarchical framework, we first learn skill priors for both body and hand movements in a decoupled setting.
ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation
ArtiGrasp leverages reinforcement learning and physics simulations to train a policy that controls the global and local hand pose.
EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild
EMDB is a novel dataset that contains high-quality 3D SMPL pose and shape parameters with global body and camera trajectories for in-the-wild videos.
Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023
We present Palm, a solution to the Long-Term Action Anticipation (LTA) task utilizing vision-language and large language models.
EFE: End-to-end Frame-to-Gaze Estimation
In this paper, we propose a frame-to-gaze network that directly predicts both 3D gaze origin and 3D gaze direction from the raw frame out of the camera without any face or eye cropping.
AG3D: Learning to Generate 3D Avatars from 2D Image Collections
The key to progress is hence to learn generative models of 3D avatars from abundant unstructured 2D image collections.
Learning Locally Editable Virtual Humans
To this end, we construct a trainable feature codebook to store local geometry and texture features on the vertices of a deformable body model, thus exploiting its consistent topology under articulation.
Human from Blur: Human Pose Tracking from Blurry Images
The key idea is to tackle the inverse problem of image deblurring by modeling the forward problem with a 3D human model, a texture map, and a sequence of poses to describe human motion.
Learning Human-to-Robot Handovers from Point Clouds
We propose the first framework to learn control policies for vision-based human-to-robot handovers, a critical task for human-robot interaction.
Hi4D: 4D Instance Segmentation of Close Human Interaction
We propose Hi4D, a method and dataset for the automatic analysis of physically close human-human interaction under prolonged contact.
Efficient Learning of High Level Plans from Play
Real-world robotic manipulation tasks remain an elusive challenge, since they involve both fine-grained environment interaction, as well as the ability to plan for long-horizon goals.
X-Avatar: Expressive Human Avatars
Our method models bodies, hands, facial expressions and appearance in a holistic fashion and can be learned from either full 3D scans or RGB-D data.
Ranked #2 on
3D Human Reconstruction
on 4D-DRESS
Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition
Specifically, we define a temporally consistent human representation in canonical space and formulate a global optimization over the background model, the canonical human shape and texture, and per-frame human pose parameters.
Ranked #19 on
3D Human Reconstruction
on 4D-DRESS
Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation
This task requires an understanding of the spatio-temporal context formed by past actions on objects, coined action context.
InstantAvatar: Learning Avatars from Monocular Video in 60 Seconds
To achieve this efficiency we propose a carefully designed and engineered system, that leverages emerging acceleration structures for neural fields, in combination with an efficient empty space-skipping strategy for dynamic scenes.
HARP: Personalized Hand Reconstruction from a Monocular RGB Video
We present HARP (HAnd Reconstruction and Personalization), a personalized hand avatar creation approach that takes a short monocular RGB video of a human hand as input and reconstructs a faithful hand avatar exhibiting a high-fidelity appearance and geometry.
PointAvatar: Deformable Point-based Head Avatars from Videos
The ability to create realistic, animatable and relightable head avatars from casual video sequences would open up wide ranging applications in communication and entertainment.
HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics
We propose a method that leverages graph neural networks, multi-level message passing, and unsupervised training to enable real-time prediction of realistic clothing dynamics.
Ranked #3 on
Physical Simulations
on 4D-DRESS
GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields
We propose GazeNeRF, a 3D-aware method for the task of gaze redirection.
DINER: Depth-aware Image-based NEural Radiance fields
We present Depth-aware Image-based NEural Radiance fields (DINER).
Fast-SNARF: A Fast Deformer for Articulated Neural Fields
A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space.
Utilizing Synthetic Data in Supervised Learning for Robust 5-DoF Magnetic Marker Localization
However, traditionally, the tracking of magnetic markers is computationally expensive due to the requirement for iterative optimization procedures.
Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors
We further qualitatively evaluate the effectiveness of our method on real images and demonstrate its generalizability towards interaction types and object categories.
TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning
We introduce TempCLR, a new time-coherent contrastive learning approach for the structured regression task of 3D hand reconstruction.
EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes
The challenge of synthesizing eyes is multifold as it requires 1) appropriate representations for the various components of the eye and the periocular region for coherent viewpoint synthesis, capable of representing diffuse, refractive and highly reflective surfaces, 2) disentangling skin and eye appearance from environmental illumination such that it may be rendered under novel lighting conditions, and 3) capturing eyeball motion and the deformation of the surrounding skin to enable re-gazing.
SFP: State-free Priors for Exploration in Off-Policy Reinforcement Learning
Several methods, such as behavioral priors, are able to leverage offline data in order to efficiently accelerate reinforcement learning on complex tasks.
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation
In part this is because there exist no datasets with ground-truth 3D annotations for the study of physically consistent and synchronised motion of hands and articulated objects.
LiP-Flow: Learning Inference-time Priors for Codec Avatars via Normalizing Flows in Latent Space
To mitigate this asymmetry, we introduce a prior model that is conditioned on the runtime inputs and tie this prior space to the 3D face model via a normalizing flow in the latent space.
PINA: Learning a Personalized Implicit Neural Avatar from a Single RGB-D Video Sequence
We present a novel method to learn Personalized Implicit Neural Avatars (PINA) from a short RGB-D sequence.
gDNA: Towards Generative Detailed Neural Avatars
Furthermore, we show that our method can be used on the task of fitting human models to raw scans, outperforming the previous state-of-the-art.
SAGA: Stochastic Whole-Body Grasping with Contact
Given an initial pose and the generated whole-body grasping pose as the start and end of the motion respectively, we design a novel contact-aware generative motion infilling module to generate a diverse set of grasp-oriented motions.
I M Avatar: Implicit Morphable Head Avatars from Videos
Traditional 3D morphable face models (3DMMs) provide fine-grained control over expression but cannot easily capture geometric and appearance details.
D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions
We introduce the dynamic grasp synthesis task: given an object with a known 6D pose and a grasp reference, our goal is to generate motions that move the object to a target 6D pose.
Human Performance Capture from Monocular Video in the Wild
In this work, we propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses, without any additional input.
Render In-between: Motion Guided Video Synthesis for Action Interpolation
We propose to address these issues in a motion-guided frame-upsampling framework that is capable of producing realistic human motion and appearance.
Shape-aware Multi-Person Pose Estimation from Multi-View Images
In this paper we contribute a simple yet effective approach for estimating 3D poses of multiple people from multi-view images.
Ranked #16 on
3D Multi-Person Pose Estimation
on Shelf
3D Multi-Person Pose Estimation
Multi-Person Pose Estimation
SPEC: Seeing People in the Wild with an Estimated Camera
We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose.
Ranked #1 on
3D Multi-Person Pose Estimation
on AGORA
A Skeleton-Driven Neural Occupancy Representation for Articulated Hands
We present Hand ArticuLated Occupancy (HALO), a novel representation of articulated hands that bridges the advantages of 3D keypoints and neural implicit surfaces and can be used in end-to-end trainable architectures.
Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation
In natural conversation and interaction, our hands often overlap or are in contact with each other.
Ranked #7 on
3D Interacting Hand Pose Estimation
on InterHand2.6M
Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation
Hand pose estimation is difficult due to different environmental conditions, object- and self-occlusion as well as diversity in hand shape and appearance.
PeCLR: Self-Supervised 3D Hand Pose Estimation from monocular RGB via Equivariant Contrastive Learning
Encouraged by the success of contrastive learning on image classification tasks, we propose a new self-supervised method for the structured regression task of 3D hand pose estimation.
Ranked #16 on
3D Hand Pose Estimation
on FreiHAND
PARE: Part Attention Regressor for 3D Human Body Estimation
Despite significant progress, we show that state of the art 3D human pose and shape estimation methods remain sensitive to partial occlusion and can produce dramatically wrong predictions although much of the body is observable.
Ranked #2 on
3D Multi-Person Pose Estimation
on AGORA
3D human pose and shape estimation
3D Multi-Person Pose Estimation
VariTex: Variational Neural Face Textures
In this paper, we propose VariTex - to the best of our knowledge the first method that learns a variational latent feature space of neural face textures, which allows sampling of novel identities.
SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes
However, this is problematic since the backward warp field is pose dependent and thus requires large amounts of data to learn.
Ranked #3 on
3D Human Reconstruction
on 4D-DRESS
Improved Learning of Robot Manipulation Tasks via Tactile Intrinsic Motivation
In this paper we address the challenge of exploration in deep reinforcement learning for robotic manipulation tasks.
The Six Hug Commandments: Design and Evaluation of a Human-Sized Hugging Robot with Visual and Haptic Perception
We followed all six tenets to create a new robotic platform, HuggieBot 2. 0, that has a soft, warm, inflated body (HuggieChest) and uses visual and haptic sensing to deliver closed-loop hugging.
Robotics
Spatial Attention Improves Iterative 6D Object Pose Estimation
Our main insight is that after the initial pose estimate, it is important to pay attention to distinct spatial features of the object in order to improve the estimation accuracy during alignment.
EM-POSE: 3D Human Pose Estimation From Sparse Electromagnetic Trackers
To this end, we present a method to estimate SMPL parameters from 6-12 EM sensors.
Self-Learning Transformations for Improving Gaze and Head Redirection
Furthermore, we show that in the presence of limited amounts of real-world training data, our method allows for improvements in the downstream task of semi-supervised cross-dataset gaze estimation.
Convolutional Autoencoders for Human Motion Infilling
At the heart of our approach lies the idea to cast motion infilling as an inpainting problem and to train a convolutional de-noising autoencoder on image-like representations of motion sequences.
Human Body Model Fitting by Learned Gradient Descent
We propose a novel algorithm for the fitting of 3D human shape to images.
Ranked #11 on
3D Human Pose Estimation
on EMDB
Category Level Object Pose Estimation via Neural Analysis-by-Synthesis
Many object pose estimation algorithms rely on the analysis-by-synthesis framework which requires explicit representations of individual object instances.
ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation
We show that our dataset can significantly improve the robustness of gaze estimation methods across different head poses and gaze angles.
Ranked #1 on
Gaze Estimation
on ETH-XGaze
(using extra training data)
Towards End-to-end Video-based Eye-Tracking
Estimating eye-gaze from images alone is a challenging task, in large parts due to un-observable person-specific factors.
CoSE: Compositional Stroke Embeddings
We demonstrate qualitatively and quantitatively that our proposed approach is able to model the appearance of individual strokes, as well as the compositional structure of larger diagram drawings.
A Spatio-temporal Transformer for 3D Human Motion Prediction
We propose a novel Transformer-based architecture for the task of generative modelling of 3D human motion.
Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints
Estimating 3D hand pose from 2D images is a difficult, inverse problem due to the inherent scale and depth ambiguities.
Ranked #10 on
3D Hand Pose Estimation
on DexYCB
The DIDI dataset: Digital Ink Diagram data
We are releasing a dataset of diagram drawings with dynamic drawing information.
Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
We present HiDe, a novel hierarchical reinforcement learning architecture that successfully solves long horizon control tasks and generalizes to unseen test scenarios.
Hierarchical Reinforcement Learning as a Model of Human Task Interleaving
The results support hierarchical RL as a plausible model of task interleaving.
Hierarchical Reinforcement Learning
reinforcement-learning
+2
Content-Consistent Generation of Realistic Eyes with Style
Accurately labeled real-world training data can be scarce, and hence recent works adapt, modify or generate images to boost target datasets.
Structured Prediction Helps 3D Human Motion Modelling
This is implemented via a hierarchy of small-sized neural networks connected analogously to the kinematic chains in the human body as well as a joint-wise decomposition in the loss function.
Learning Functionally Decomposed Hierarchies for Continuous Navigation Tasks
Hierarchical Reinforcement Learning (HRL) has held the promise to enhance the capabilities of RL agents via operation on different levels of temporal abstraction.
Sample Efficient Learning of Path Following and Obstacle Avoidance Behavior for Quadrotors
This enables an easy to implement learning algorithm that is robust to errors of the model used in the model predictive controller.
Demonstration-Guided Deep Reinforcement Learning of Control Policies for Dexterous Human-Robot Interaction
In this paper, we propose a method for training control policies for human-robot interactions such as handshakes or hand claps via Deep Reinforcement Learning.
Few-Shot Adaptive Gaze Estimation
Inter-personal anatomical differences limit the accuracy of person-independent gaze estimation networks.
Ranked #1 on
Gaze Estimation
on MPII Gaze
(using extra training data)
Photo-Realistic Monocular Gaze Redirection Using Generative Adversarial Networks
In this work, we present a novel method to alleviate this problem by leveraging generative adversarial training to synthesize an eye image conditioned on a target gaze direction.
STCN: Stochastic Temporal Convolutional Networks
Convolutional architectures have recently been shown to be competitive on many sequence modelling tasks when compared to the de-facto standard of recurrent neural networks (RNNs), while providing computational and modeling advantages due to inherent parallelism.
Unpaired Pose Guided Human Image Generation
This paper studies the task of full generative modelling of realistic images of humans, guided only by coarse sketch of the pose, while providing control over the specific instance or type of outfit worn by the user.
Monocular Neural Image Based Rendering with Continuous View Control
The approach is self-supervised and only requires 2D images and associated view transforms for training.
End-to-end Learning for Graph Decomposition
The new optimization problem can be viewed as a Conditional Random Field (CRF) in which the random variables are associated with the binary edge labels of the initial graph and the hard constraints are introduced in the CRF as high-order potentials.
Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time
To learn from sufficient data, we synthesize IMU data from motion capture datasets.
Deep Pictorial Gaze Estimation
In this paper, we introduce a novel deep neural network architecture specifically designed for the task of gaze estimation from single eye input.
Learn-to-Score: Efficient 3D Scene Exploration by Predicting View Utility
We propose to learn a better utility function that predicts the usefulness of future viewpoints.
Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings
Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras.
Cross-modal Deep Variational Hand Pose Estimation
Furthermore, we show that our proposed method can be used without changes on depth images and performs comparably to specialized methods.
DeepWriting: Making Digital Ink Editable via Deep Generative Modeling
Digital ink promises to combine the flexibility and aesthetics of handwriting and the ability to process, search and edit digital text.
Guiding InfoGAN with Semi-Supervision
In this paper we propose a new semi-supervised GAN architecture (ss-InfoGAN) for image synthesis that leverages information from few labels (as little as 0. 22%, max.
Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction
We introduce a new method that efficiently computes a set of viewpoints and trajectories for high-quality 3D reconstructions in outdoor environments.
Learning Human Motion Models for Long-term Predictions
Furthermore, we propose new evaluation protocols to assess the quality of synthetic motion sequences even for which no ground truth data exists.
Thin-Slicing Network: A Deep Structured Model for Pose Estimation in Videos
Temporal information can provide additional cues about the location of body joints and help to alleviate these issues.
Ranked #4 on
Pose Estimation
on UPenn Action
KinectFusion: Real-Time Dense Surface Mapping and Tracking
We present a system for accurate real-time mapping of complex and arbitrary indoor scenes in variable lighting conditions, using only a moving low-cost depth camera and commodity graphics hardware.
