Search Results for author:
Found 35 papers, 18 papers with code
GraspDiffusion: Synthesizing Realistic Whole-body Hand-Object Interaction
In this paper, we propose GraspDiffusion, a novel generative method that creates realistic scenes of human-object interaction.
Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses
In this paper, we present a method to reconstruct the world and multiple dynamic humans in 3D from a monocular video input.
PEGASUS: Personalized Generative 3D Avatars with Composable Attributes
We present PEGASUS, a method for constructing a personalized generative 3D face avatar from monocular video sources.
GALA: Generating Animatable Layered Assets from a Single Scan
Through a series of decomposition steps, we obtain multiple layers of 3D assets in a shared canonical space normalized in terms of poses and human shapes, hence supporting effortless composition to novel identities and reanimation with novel poses.
Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models
To construct the distribution, we present a novel pipeline that synthesizes diverse and realistic 3D HOI samples given any 3D object mesh.
ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions
To address this challenge, we introduce the ParaHome system, designed to capture and parameterize dynamic 3D movements of humans and objects within a common home environment.
Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera
We present a lightweight and affordable motion capture method based on two smartwatches and a head-mounted camera.
CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images
We present multiple strategies to leverage the synthesized images, including (1) the first method to leverage a generative image model for 3D human-object spatial relation learning; (2) a framework to reason about the 3D spatial relations from inconsistent 2D cues in a self-supervised manner via 3D occupancy reasoning with pose canonicalization; (3) semantic clustering to disambiguate different types of interactions with the same object types; and (4) a novel metric to assess the quality of 3D spatial learning of interaction.
NCHO: Unsupervised Learning for Neural 3D Composition of Humans and Objects
Our compositional model is interaction-aware, meaning the spatial relationship between humans and objects, and the mutual shape change by physical contact is fully incorporated.
Chupa: Carving 3D Clothed Humans from Skinned Shape Priors using 2D Diffusion Probabilistic Models
We propose a 3D generation pipeline that uses diffusion models to generate realistic human digital avatars.
Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments
The key motivation of LAMA is to build a unified framework to encompass a series of everyday motions including locomotion, scene interaction, and object manipulation.
CHORUS : Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images
We present multiple strategies to leverage the synthesized images, including (1) the first method to leverage a generative image model for 3D human-object spatial relation learning; (2) a framework to reason about the 3D spatial relations from inconsistent 2D cues in a self-supervised manner via 3D occupancy reasoning with pose canonicalization; (3) semantic clustering to disambiguate different types of interactions with the same object types; and (4) a novel metric to assess the quality of 3D spatial learning of interaction.
Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion
We present a framework for modeling interactional communication in dyadic conversations: given multimodal inputs of a speaker, we autoregressively output multiple possibilities of corresponding listener motion.
BANMo: Building Animatable 3D Neural Models from Many Casual Videos
Our key insight is to merge three schools of thought; (1) classic deformable shape models that make use of articulated bones and blend skinning, (2) volumetric neural radiance fields (NeRFs) that are amenable to gradient-based optimization, and (3) canonical embeddings that generate correspondences between pixels and an articulated model.
Modeling human intention inference in continuous 3D domains by inverse planning and body kinematics
We describe Generative Body Kinematics model, which predicts human intention inference in this domain using Bayesian inverse planning and inverse body kinematics.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos
We evaluate this approach on our dataset, demonstrating that human-object relations can significantly reduce the ambiguity of articulated object reconstructions from challenging real-world videos.
FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration
Most existing monocular 3D pose estimation approaches only focus on a single body part, neglecting the fact that the essential nuance of human motion is conveyed through a concert of subtle movements of face, hands, and body.
3D Multi-bodies: Fitting Sets of Plausible 3D Human Models to Ambiguous Image Data
We consider the problem of obtaining dense 3D reconstructions of humans from single and partially occluded views.
Ranked #3 on
Multi-Hypotheses 3D Human Pose Estimation
on AH36M
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration
To construct FrankMocap, we build the state-of-the-art monocular 3D "hand" motion capture method by taking the hand part of the whole body parametric model (SMPL-X).
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild
We present a method that infers spatial arrangements and shapes of humans and objects in a globally consistent 3D scene, all from a single image in-the-wild captured in an uncontrolled environment.
Ranked #3 on
3D Object Reconstruction
on BEHAVE
Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics
We demonstrate the efficacy of our method on hand gesture synthesis from body motion input, and as a strong body prior for single-view image-based 3D hand pose estimation.
Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation
Remarkably, the resulting annotations are sufficient to train from scratch 3D pose regressor networks that outperform the current state-of-the-art on in-the-wild benchmarks such as 3DPW.
Ranked #24 on
3D Human Pose Estimation
on MPI-INF-3DHP
(PA-MPJPE metric)
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization
Although current approaches have demonstrated the potential in real world settings, they still fail to produce reconstructions with the level of detail often present in the input images.
Ranked #1 on
3D Object Reconstruction From A Single Image
on BUFF
Single-Network Whole-Body Pose Estimation
We present the first single-network approach for 2D~whole-body pose estimation, which entails simultaneous localization of body, face, hands, and feet keypoints.
Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction
We present a new research task and a dataset to understand human social interactions via computational methods, to ultimately endow machines with the ability to encode and decode a broad channel of social signals humans use.
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions
The body pose of a person wearing a camera is of great interest for applications in augmented reality, healthcare, and robotics, yet much of the person's body is out of view for a typical wearable camera.
Capture Dense: Markerless Motion Capture Meets Dense Pose Estimation
Specifically, we first introduce a novel markerless motion capture method that can take advantage of dense parsing capability provided by the dense pose detector.
Monocular Total Capture: Posing Face, Body, and Hands in the Wild
We present the first method to capture the 3D total motion of a target person from a monocular view input.
Ranked #28 on
Monocular 3D Human Pose Estimation
on Human3.6M
Structure from Recurrent Motion: From Rigidity to Recurrency
This paper proposes a new method for Non-Rigid Structure-from-Motion (NRSfM) from a long monocular video sequence observing a non-rigid object performing recurrent and possibly repetitive dynamic action.
Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies
We present a unified deformation model for the markerless capture of multiple scales of human movement, including facial expressions, body motion, and hand gestures.
Hand Keypoint Detection in Single Images using Multiview Bootstrapping
The method is used to train a hand keypoint detector for single images.
Panoptic Studio: A Massively Multiview System for Social Interaction Capture
The core challenges in capturing social interactions are: (1) occlusion is functional and frequent; (2) subtle motion needs to be measured over a space large enough to host a social group; (3) human appearance and configuration variation is immense; and (4) attaching markers to the body may prime the nature of interactions.
Panoptic Studio: A Massively Multiview System for Social Motion Capture
We present an approach to capture the 3D structure and motion of a group of people engaged in a social interaction.
MAP Visibility Estimation for Large-Scale Dynamic 3D Reconstruction
Many traditional challenges in reconstructing 3D motion, such as matching across wide baselines and handling occlusion, reduce in significance as the number of unique viewpoints increases.
