no code implementations • 12 Sep 2024 • Boxiang Rong, Artur Grigorev, Wenbo Wang, Michael J. Black, Bernhard Thomaszewski, Christina Tsalicoglou, Otmar Hilliges
We introduce Gaussian Garments, a novel approach for reconstructing realistic simulation-ready garment assets from multi-view videos.
no code implementations • 5 Sep 2024 • Shashank Tripathi, Omid Taheri, Christoph Lassner, Michael J. Black, Daniel Holden, Carsten Stoll
Generating realistic human motion is essential for many computer vision and graphics applications.
no code implementations • 15 Aug 2024 • Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z. Xiao, Katherine M. Collins, Joshua B. Tenenbaum, Adrian Weller, Michael J. Black, Bernhard Schölkopf
While LLMs exhibit impressive skills in general program synthesis and analysis, symbolic graphics programs offer a new layer of evaluation: they allow us to test an LLM's ability to answer different-grained semantic-level questions of the images or 3D geometries without a vision encoder.
no code implementations • 1 Aug 2024 • Nikos Athanasiou, Alpár Ceske, Markos Diomataris, Michael J. Black, Gül Varol
Access to this data allows us to train a conditional diffusion model, TMED, that takes both the source motion and the edit text as input.
no code implementations • 12 Jun 2024 • Mert Albaba, Sammy Christen, Thomas Langarek, Christoph Gebhardt, Otmar Hilliges, Michael J. Black
The trainer optimizes for long-term cumulative rewards from the discriminator, enabling it to provide nuanced feedback that accounts for the complexity of the task and the student's current capabilities.
1 code implementation • 23 May 2024 • Yuliang Xiu, Yufei Ye, Zhen Liu, Dimitrios Tzionas, Michael J. Black
We address this novel "Album2Human" task by developing PuzzleAvatar, a novel model that generates a faithful 3D avatar (in a canonical pose) from a personal OOTD album, while bypassing the challenging estimation of body and camera pose.
no code implementations • 15 May 2024 • Artur Grigorev, Giorgio Becherini, Michael J. Black, Otmar Hilliges, Bernhard Thomaszewski
In this work, we present \moniker{}, a learning-based solution for handling intersections in neural cloth simulations.
no code implementations • 7 May 2024 • Jing Lin, Yao Feng, Weiyang Liu, Michael J. Black
The novel features of ChatHuman include leveraging academic publications to guide the application of 3D human-related tools, employing a retrieval-augmented generation model to generate in-context-learning examples for handling new tools, and discriminating and integrating tool results to enhance 3D human understanding.
1 code implementation • CVPR 2024 • Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, Yao Feng, Michael J. Black
We address the problem of regressing 3D human pose and shape from a single image, with a focus on 3D accuracy.
Ranked #32 on 3D Human Pose Estimation on 3DPW
no code implementations • 23 Apr 2024 • Peter Kulits, Haiwen Feng, Weiyang Liu, Victoria Abrevaya, Michael J. Black
Inverse graphics -- the task of inverting an image into physical variables that, when rendered, enable reproduction of the observed scene -- is a fundamental challenge in computer vision and graphics.
no code implementations • CVPR 2024 • Markos Diomataris, Nikos Athanasiou, Omid Taheri, Xi Wang, Otmar Hilliges, Michael J. Black
To address this, we introduce WANDR, a data-driven model that takes an avatar's initial pose and a goal's 3D position and generates natural human motions that place the end effector (wrist) on the goal location.
no code implementations • 16 Apr 2024 • Hongwei Yi, Justus Thies, Michael J. Black, Xue Bin Peng, Davis Rempe
Our approach begins with pre-training a scene-agnostic text-to-motion diffusion model, emphasizing goal-reaching constraints on large-scale motion-capture datasets.
no code implementations • 3 Apr 2024 • Silvia Zuffi, Michael J. Black
This involves learning a mapping between the latent space of a vision-language model and the parameter space of the 3D model, which we do using a small set of shape and text pairs.
no code implementations • 21 Mar 2024 • Haiwen Feng, Zheng Ding, Zhihao Xia, Simon Niklaus, Victoria Abrevaya, Michael J. Black, Xuaner Zhang
We introduce bounded generation as a generalized task to control video generation to synthesize arbitrary camera and subject motion based only on a given start and end frame.
no code implementations • 16 Jan 2024 • Mathis Petrovich, Or Litany, Umar Iqbal, Michael J. Black, Gül Varol, Xue Bin Peng, Davis Rempe
To generate composite animations from a multi-track timeline, we propose a new test-time denoising method.
no code implementations • CVPR 2024 • Marilyn Keller, Vaibhav Arora, Abdelmouttaleb Dakri, Shivam Chandhok, Jürgen Machann, Andreas Fritsche, Michael J. Black, Sergi Pujades
We first create a dataset of human tissues by segmenting full-body MRI scans and registering the SMPL body mesh to the body surface.
no code implementations • CVPR 2024 • Silvia Zuffi, Ylva Mellbin, Ci Li, Markus Hoeschle, Hedvig Kjellström, Senya Polikovsky, Elin Hernlund, Michael J. Black
We introduce VAREN a novel 3D articulated parametric shape model learned from 3D scans of many real horses.
no code implementations • CVPR 2024 • Vanessa Sklyarova, Egor Zakharov, Otmar Hilliges, Michael J. Black, Justus Thies
We present HAAR a new strand-based generative model for 3D human hairstyles.
1 code implementation • CVPR 2024 • Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, Michael J. Black
We propose EMAGE, a framework to generate full-body human gestures from audio and masked gestures, encompassing facial, local body, hands, and global movements.
Ranked #1 on 3D Face Animation on BEAT2
no code implementations • 27 Dec 2023 • Enes Duran, Muhammed Kocabas, Vasileios Choutas, Zicong Fan, Michael J. Black
Therefore, we develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions.
Ranked #3 on 3D Hand Pose Estimation on HO-3D v3
no code implementations • 22 Dec 2023 • Mirela Ostrek, Carol O'Sullivan, Michael J. Black, Justus Thies
We present ESP, a novel method for context-aware full-body generation, that enables photo-realistic synthesis and inpainting of people wearing clothing that is semantically appropriate for the scene depicted in an input photograph.
1 code implementation • 18 Dec 2023 • Vanessa Sklyarova, Egor Zakharov, Otmar Hilliges, Michael J. Black, Justus Thies
We present HAAR, a new strand-based generative model for 3D human hairstyles.
1 code implementation • CVPR 2024 • Soyong Shin, Juyong Kim, Eni Halilaj, Michael J. Black
We address these limitations with WHAM (World-grounded Humans with Accurate Motion), which accurately and efficiently reconstructs 3D human motion in a global coordinate system from video.
Ranked #1 on 3D Human Pose Estimation on 3DPW
1 code implementation • CVPR 2024 • Kiran Chhatre, Radek Daněček, Nikos Athanasiou, Giorgio Becherini, Christopher Peters, Michael J. Black, Timo Bolkart
Once trained, AMUSE synthesizes 3D human gestures directly from speech with control over the expressed emotions and style by combining the content from the driving speech with the emotion and style of another speech sequence.
no code implementations • CVPR 2024 • Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, Michael J. Black
Additionally, ChatPose empowers LLMs to apply their extensive world knowledge in reasoning about human poses, leading to two advanced tasks: speculative pose generation and reasoning about pose estimation.
1 code implementation • CVPR 2024 • Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Muhammed Kocabas, Xu Chen, Michael J. Black, Otmar Hilliges
Since humans interact with diverse objects every day, the holistic 3D capture of these interactions is important to understand and model human behaviour.
1 code implementation • 10 Nov 2023 • Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, Bernhard Schölkopf
We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT).
1 code implementation • 26 Oct 2023 • Shrisha Bharadwaj, Yufeng Zheng, Otmar Hilliges, Michael J. Black, Victoria Fernandez-Abrevaya
Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems.
no code implementations • 23 Oct 2023 • Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu, Liam Paull, Michael J. Black, Bernhard Schölkopf
Recent work has focused on the former, and methods for reconstructing open surfaces do not support fast reconstruction with material and lighting or unconditional generative modelling.
no code implementations • 20 Oct 2023 • Muhammed Kocabas, Ye Yuan, Pavlo Molchanov, Yunrong Guo, Michael J. Black, Otmar Hilliges, Jan Kautz, Umar Iqbal
This design combines the strengths of SLAM and motion priors, which leads to significant improvements in human and camera motion estimation.
2 code implementations • ICCV 2023 • Yandong Wen, Weiyang Liu, Yao Feng, Bhiksha Raj, Rita Singh, Adrian Weller, Michael J. Black, Bernhard Schölkopf
In this paper, we focus on a general yet important learning problem, pairwise similarity learning (PSL).
1 code implementation • ICCV 2023 • Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas, Michael J. Black
In contrast, we focus on inferring dense, 3D contact between the full body surface and objects in arbitrary images.
Ranked #2 on Contact Detection on BEHAVE
no code implementations • 13 Sep 2023 • Hao Zhang, Yao Feng, Peter Kulits, Yandong Wen, Justus Thies, Michael J. Black
We argue that existing methods are limited because they employ a monolithic modeling approach, using a single representation for the head, face, hair, and accessories.
no code implementations • 12 Sep 2023 • Yao Feng, Weiyang Liu, Timo Bolkart, Jinlong Yang, Marc Pollefeys, Michael J. Black
Towards this end, both explicit and implicit 3D representations are heavily studied for a holistic modeling and capture of the whole human (e. g., body, clothing, face and hair), but neither representation is an optimal choice in terms of representation efficacy since different parts of the human avatar have different modeling desiderata.
1 code implementation • 24 Aug 2023 • Sai Kumar Dwivedi, Cordelia Schmid, Hongwei Yi, Michael J. Black, Dimitrios Tzionas
To address this, we develop POCO, a novel framework for training HPS regressors to estimate not only a 3D human body, but also their confidence, in a single feed-forward pass.
no code implementations • 22 Aug 2023 • Omid Taheri, Yi Zhou, Dimitrios Tzionas, Yang Zhou, Duygu Ceylan, Soren Pirk, Michael J. Black
In contrast, we introduce GRIP, a learning-based method that takes, as input, the 3D motion of the body and the object, and synthesizes realistic motion for both hands before, during, and after object interaction.
no code implementations • CVPR 2024 • Soubhik Sanyal, Partha Ghosh, Jinlong Yang, Michael J. Black, Justus Thies, Timo Bolkart
We use intermediate activations of the learned geometry model to condition our texture generator.
no code implementations • 21 Aug 2023 • Tingting Liao, Hongwei Yi, Yuliang Xiu, Jiaxaing Tang, Yangyi Huang, Justus Thies, Michael J. Black
We introduce TADA, a simple-yet-effective approach that takes textual descriptions and produces expressive 3D avatars with high-quality geometry and lifelike textures, that can be animated and rendered with traditional graphics pipelines.
no code implementations • 19 Jul 2023 • Omri Ben-Dov, Pravir Singh Gupta, Victoria Abrevaya, Michael J. Black, Partha Ghosh
Generative Adversarial Networks (GANs) can produce high-quality samples, but do not provide an estimate of the probability density around the samples.
2 code implementations • CVPR 2023 • Michael J. Black, Priyanka Patel, Joachim Tesch, Jinlong Yang
BEDLAM is useful for a variety of tasks and all images, ground truth bodies, 3D clothing, support code, and more are available for research purposes.
no code implementations • 15 Jun 2023 • Radek Daněček, Kiran Chhatre, Shashank Tripathi, Yandong Wen, Michael J. Black, Timo Bolkart
While the best recent methods generate 3D animations that are synchronized with the input audio, they largely ignore the impact of emotions on facial expressions.
1 code implementation • CVPR 2023 • Timo Bolkart, Tianye Li, Michael J. Black
We use raw MVS scans as supervision during training, but, once trained, TEMPEH directly predicts 3D heads in dense correspondence without requiring scans.
3 code implementations • CVPR 2023 • Yu Sun, Qian Bao, Wu Liu, Tao Mei, Michael J. Black
Although the estimation of 3D human pose and shape (HPS) is rapidly progressing, current methods still cannot reliably estimate moving humans in global coordinates, which is critical for many applications.
Ranked #55 on 3D Human Pose Estimation on 3DPW
no code implementations • ICCV 2023 • Zijian Dong, Xu Chen, Jinlong Yang, Michael J. Black, Otmar Hilliges, Andreas Geiger
The key to progress is hence to learn generative models of 3D avatars from abundant unstructured 2D image collections.
1 code implementation • ICCV 2023 • Mathis Petrovich, Michael J. Black, Gül Varol
We show that maintaining the motion generation loss, along with the contrastive training, is crucial to obtain good performance.
no code implementations • ICCV 2023 • Haiwen Feng, Peter Kulits, Shichen Liu, Michael J. Black, Victoria Abrevaya
Learning-based methods address this but do not generalize well when the input pose is far from those seen during training.
no code implementations • ICCV 2023 • Nikos Athanasiou, Mathis Petrovich, Michael J. Black, Gül Varol
Motivated by the observation that the correspondence between actions and body parts is encoded in powerful language models, we extract this knowledge by prompting GPT-3 with text such as "what are the body parts involved in the action <action name>?
no code implementations • CVPR 2023 • Maria-Paola Forte, Peter Kulits, Chun-Hao Huang, Vasileios Choutas, Dimitrios Tzionas, Katherine J. Kuchenbecker, Michael J. Black
A perceptual study shows that SGNify's 3D reconstructions are significantly more comprehensible and natural than those of previous methods and are on par with the source videos.
no code implementations • CVPR 2023 • Shashank Tripathi, Lea Müller, Chun-Hao P. Huang, Omid Taheri, Michael J. Black, Dimitrios Tzionas
Inspired by biomechanics, we infer the pressure heatmap on the body, the Center of Pressure (CoP) from the heatmap, and the SMPL body's Center of Mass (CoM).
Ranked #3 on 3D Human Pose Estimation on RICH
1 code implementation • 14 Mar 2023 • Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahrai, Liam Paull, Weiyang Liu
We consider the task of generating realistic 3D shapes, which is useful for a variety of applications such as automatic scene generation and physical simulation.
1 code implementation • CVPR 2023 • Yixin Chen, Sai Kumar Dwivedi, Michael J. Black, Dimitrios Tzionas
To build HOT, we use two data sources: (1) We use the PROX dataset of 3D human meshes moving in 3D scenes, and automatically annotate 2D image areas for contact via 3D mesh proximity and projection.
no code implementations • CVPR 2023 • Nadine Rüegg, Shashank Tripathi, Konrad Schindler, Michael J. Black, Silvia Zuffi
To that end, we exploit contact with the ground as a form of side information.
1 code implementation • CVPR 2023 • Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, Otmar Hilliges
The ability to create realistic, animatable and relightable head avatars from casual video sequences would open up wide ranging applications in communication and entertainment.
2 code implementations • CVPR 2023 • Artur Grigorev, Bernhard Thomaszewski, Michael J. Black, Otmar Hilliges
We propose a method that leverages graph neural networks, multi-level message passing, and unsupervised training to enable real-time prediction of realistic clothing dynamics.
Ranked #3 on Physical Simulations on 4D-DRESS
1 code implementation • CVPR 2023 • Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, Michael J. Black
To increase robustness for these cases, existing work uses an explicit parametric body model to constrain surface reconstruction, but this limits the recovery of free-form surfaces such as loose clothing that deviates from the body.
Ranked #7 on 3D Human Reconstruction on CustomHumans
no code implementations • CVPR 2023 • Hongwei Yi, Chun-Hao P. Huang, Shashank Tripathi, Lea Hering, Justus Thies, Michael J. Black
We propose MIME (Mining Interaction and Movement to infer 3D Environments), which is a generative model of indoor scenes that produces furniture layouts that are consistent with the human movement.
Ranked #2 on Indoor Scene Synthesis on PRO-teXt
2D Semantic Segmentation task 1 (8 classes) 3D Semantic Scene Completion +2
2 code implementations • CVPR 2023 • Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, DaCheng Tao, Michael J. Black
This work addresses the problem of generating 3D holistic body motions from human speech.
Ranked #3 on Gesture Generation on BEAT2
1 code implementation • 28 Nov 2022 • Xu Chen, Tianjian Jiang, Jie Song, Max Rietmann, Andreas Geiger, Michael J. Black, Otmar Hilliges
A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space.
1 code implementation • 25 Oct 2022 • Ahmed A. A. Osman, Timo Bolkart, Dimitrios Tzionas, Michael J. Black
Using novel 4D scans of feet, we train a model with an extended kinematic tree that captures the range of motion of the toes.
1 code implementation • 4 Oct 2022 • Yao Feng, Jinlong Yang, Marc Pollefeys, Michael J. Black, Timo Bolkart
Building on this insight, we propose SCARF (Segmented Clothed Avatar Radiance Field), a hybrid model combining a mesh-based body with a neural radiance field.
1 code implementation • 28 Sep 2022 • Nitin Saini, Chun-Hao P. Huang, Michael J. Black, Aamir Ahmad
Second, we learn a probability distribution of short human motion sequences ($\sim$1sec) relative to the ground plane and leverage it to disambiguate between the camera and human motion.
no code implementations • 26 Sep 2022 • Yinghao Huang, Omid Tehari, Michael J. Black, Dimitrios Tzionas
With this method we capture the InterCap dataset, which contains 10 subjects (5 males and 5 females) interacting with 10 objects of various sizes and affordances, including contact with the hands or feet.
no code implementations • 14 Sep 2022 • Qianli Ma, Jinlong Yang, Michael J. Black, Siyu Tang
Specifically, we extend point-based methods with a coarse stage, that replaces canonicalization with a learned pose-independent "coarse shape" that can capture the rough surface geometry of clothing like skirts.
1 code implementation • 9 Sep 2022 • Nikos Athanasiou, Mathis Petrovich, Michael J. Black, Gül Varol
In particular, our goal is to enable the synthesis of a series of actions, which we refer to as temporal action composition.
no code implementations • 23 Jun 2022 • Omri Ben-Dov, Pravir Singh Gupta, Victoria Fernandez Abrevaya, Michael J. Black, Partha Ghosh
Modern generative models are roughly divided into two main categories: (1) models that can produce high-quality random samples, but cannot estimate the exact density of new data points and (2) those that provide exact density estimation, at the expense of sample quality and compactness of the latent space.
2 code implementations • CVPR 2022 • Chun-Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, Michael J. Black
We capture a new dataset called RICH for "Real scenes, Interaction, Contact and Humans."
Ranked #3 on Contact Detection on BEHAVE
1 code implementation • CVPR 2022 • Vasileios Choutas, Lea Muller, Chun-Hao P. Huang, Siyu Tang, Dimitrios Tzionas, Michael J. Black
Since paired data with images and 3D body shape are rare, we exploit two sources of information: (1) we collect internet images of diverse "fashion" models together with a small set of anthropometric measurements; (2) we collect linguistic shape attributes for a wide range of 3D body meshes and the model images.
Ranked #6 on 3D Human Shape Estimation on SSP-3D
no code implementations • 8 May 2022 • Haiwen Feng, Timo Bolkart, Joachim Tesch, Michael J. Black, Victoria Abrevaya
Our experimental results show significant improvement compared to state-of-the-art methods on albedo estimation, both in terms of accuracy and fairness.
1 code implementation • CVPR 2023 • Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, Otmar Hilliges
In part this is because there exist no datasets with ground-truth 3D annotations for the study of physically consistent and synchronised motion of hands and articulated objects.
1 code implementation • 25 Apr 2022 • Mathis Petrovich, Michael J. Black, Gül Varol
In contrast to most previous work which focuses on generating a single, deterministic, motion from a textual description, we design a variational approach that can produce multiple diverse human motions.
Ranked #5 on Motion Synthesis on Inter-X
1 code implementation • CVPR 2022 • Radek Danecek, Michael J. Black, Timo Bolkart
While EMOCA achieves 3D reconstruction errors that are on par with the current best methods, it significantly outperforms them in terms of the quality of the reconstructed expression and the perceived emotional content.
Ranked #14 on 3D Face Reconstruction on REALY (side-view)
1 code implementation • CVPR 2022 • Marilyn Keller, Silvia Zuffi, Michael J. Black, Sergi Pujades
We address the problem of inferring the anatomic skeleton of a person, in an arbitrary pose, from the 3D surface of the body; i. e. we predict the inside (bones) from the outside (skin).
no code implementations • CVPR 2022 • Nadine Rueegg, Silvia Zuffi, Konrad Schindler, Michael J. Black
But, even with a better shape model, the problem of regressing dog shape from an image is still challenging because we lack paired images with 3D ground truth.
no code implementations • 21 Mar 2022 • Jiankai Sun, Bolei Zhou, Michael J. Black, Arjun Chandrasekaran
An important component of this problem is 3D Temporal Action Localization (3D-TAL), which involves recognizing what actions a person is performing, and when.
1 code implementation • CVPR 2022 • Hongwei Yi, Chun-Hao P. Huang, Dimitrios Tzionas, Muhammed Kocabas, Mohamed Hassan, Siyu Tang, Justus Thies, Michael J. Black
In fact, we demonstrate that these human-scene interactions (HSIs) can be leveraged to improve the 3D reconstruction of a scene from a monocular RGB video.
1 code implementation • 20 Jan 2022 • Nitin Saini, Elia Bonetto, Eric Price, Aamir Ahmad, Michael J. Black
In this letter, we present a novel markerless 3D human motion capture (MoCap) system for unstructured, outdoor environments that uses a team of autonomous unmanned aerial vehicles (UAVs) with on-board RGB cameras and computation.
no code implementations • CVPR 2022 • Xu Chen, Tianjian Jiang, Jie Song, Jinlong Yang, Michael J. Black, Andreas Geiger, Otmar Hilliges
Furthermore, we show that our method can be used on the task of fitting human models to raw scans, outperforming the previous state-of-the-art.
no code implementations • 7 Jan 2022 • Javier Romero, Dimitrios Tzionas, Michael J. Black
We attach MANO to a standard parameterized 3D body shape model (SMPL), resulting in a fully articulated body and hand model (SMPL+H).
1 code implementation • CVPR 2022 • Omid Taheri, Vasileios Choutas, Michael J. Black, Dimitrios Tzionas
This is challenging, as it requires the avatar to walk towards the object with foot-ground contact, orient the head towards it, reach out, and grasp it with a realistic hand pose and hand-object contact.
2 code implementations • CVPR 2022 • Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, Michael J. Black
First, ICON infers detailed clothed-human normals (front/back) conditioned on the SMPL(-X) normals.
Ranked #1 on 3D Human Reconstruction on CAPE
4 code implementations • CVPR 2022 • Yu Sun, Wu Liu, Qian Bao, Yili Fu, Tao Mei, Michael J. Black
To do so, we exploit a 3D body model space that lets BEV infer shapes from infants to adults.
Ranked #1 on 3D Depth Estimation on Relative Human (using extra training data)
1 code implementation • CVPR 2022 • Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, Otmar Hilliges
Traditional 3D morphable face models (3DMMs) provide fine-grained control over expression but cannot easily capture geometric and appearance details.
no code implementations • 8 Dec 2021 • Partha Ghosh, Dominik Zietlow, Michael J. Black, Larry S. Davis, Xiaochen Hu
Our \textbf{InvGAN}, short for Invertible GAN, successfully embeds real images to the latent space of a high quality generative model.
no code implementations • ICCV 2021 • Soubhik Sanyal, Alex Vorobiov, Timo Bolkart, Matthew Loper, Betty Mohler, Larry Davis, Javier Romero, Michael J. Black
Synthesizing images of a person in novel poses from a single image is a highly ambiguous task.
2 code implementations • ICCV 2021 • Nima Ghorbani, Michael J. Black
Commercial auto-labeling tools require a specific calibration procedure at capture time, which is not possible for archival data.
1 code implementation • ICCV 2021 • Sai Kumar Dwivedi, Nikos Athanasiou, Muhammed Kocabas, Michael J. Black
For Minimally-Clothed regions, we define the DSR-MC loss, which encourages a tight match between a rendered SMPL body and the minimally-clothed regions of the image.
Ranked #58 on 3D Human Pose Estimation on 3DPW (using extra training data)
1 code implementation • ICCV 2021 • Muhammed Kocabas, Chun-Hao P. Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, Michael J. Black
We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose.
Ranked #1 on 3D Multi-Person Pose Estimation on AGORA
no code implementations • ICCV 2021 • Qianli Ma, Jinlong Yang, Siyu Tang, Michael J. Black
The geometry feature can be optimized to fit a previously unseen scan of a person in clothing, enabling the scan to be reposed realistically.
1 code implementation • 1 Jul 2021 • Zicong Fan, Adrian Spurr, Muhammed Kocabas, Siyu Tang, Michael J. Black, Otmar Hilliges
In natural conversation and interaction, our hands often overlap or are in contact with each other.
Ranked #7 on 3D Interacting Hand Pose Estimation on InterHand2.6M
no code implementations • 18 Jun 2021 • Ci Li, Nima Ghorbani, Sofia Broomé, Maheen Rashid, Michael J. Black, Elin Hernlund, Hedvig Kjellström, Silvia Zuffi
In this paper we present our preliminary work on model-based behavioral analysis of horse motion.
1 code implementation • CVPR 2021 • Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, Michael J. Black
To address this, we present BABEL, a large dataset with language labels describing the actions being performed in mocap sequences.
Ranked #1 on Action Classification on BABEL
1 code implementation • 11 May 2021 • Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, Michael J. Black
Second, human shape is highly correlated with gender, but existing work ignores this.
1 code implementation • CVPR 2021 • Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi, Michael J. Black
Additionally, we fine-tune methods on AGORA and show improved performance on both AGORA and 3DPW, confirming the realism of the dataset.
1 code implementation • ICCV 2021 • Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, Michael J. Black
Despite significant progress, we show that state of the art 3D human pose and shape estimation methods remain sensitive to partial occlusion and can produce dramatically wrong predictions although much of the body is observable.
Ranked #2 on 3D Multi-Person Pose Estimation on AGORA
3D human pose and shape estimation 3D Multi-Person Pose Estimation
1 code implementation • CVPR 2021 • Qianli Ma, Shunsuke Saito, Jinlong Yang, Siyu Tang, Michael J. Black
We demonstrate the efficacy of our surface representation by learning models of complex clothing from point clouds.
1 code implementation • CVPR 2021 • Marko Mihajlovic, Yan Zhang, Michael J. Black, Siyu Tang
Substantial progress has been made on modeling rigid 3D objects using deep implicit representations.
2 code implementations • ICCV 2021 • Mathis Petrovich, Michael J. Black, Gül Varol
By sampling from this latent space and querying a certain duration through a series of positional encodings, we synthesize variable-length motion sequences conditioned on a categorical action.
1 code implementation • ICCV 2021 • Xu Chen, Yufeng Zheng, Michael J. Black, Otmar Hilliges, Andreas Geiger
However, this is problematic since the backward warp field is pose dependent and thus requires large amounts of data to learn.
Ranked #3 on 3D Human Reconstruction on 4D-DRESS
1 code implementation • CVPR 2021 • Lea Müller, Ahmed A. A. Osman, Siyu Tang, Chun-Hao P. Huang, Michael J. Black
Third, we develop a novel HPS optimization method, SMPLify-XMC, that includes contact constraints and uses the known 3DCP body pose during fitting to create near ground-truth poses for MTP images.
Ranked #79 on 3D Human Pose Estimation on 3DPW (MPJPE metric)
2 code implementations • CVPR 2021 • Shunsuke Saito, Jinlong Yang, Qianli Ma, Michael J. Black
We present SCANimate, an end-to-end trainable framework that takes raw 3D scans of a clothed human and turns them into an animatable avatar.
Ranked #1 on 3D Human Reconstruction on 4D-DRESS
1 code implementation • CVPR 2021 • Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, Michael J. Black
Second, we show that POSA's learned representation of body-scene interaction supports monocular human pose estimation that is consistent with a 3D scene, improving on the state of the art.
Ranked #4 on Contact Detection on BEHAVE
2 code implementations • 7 Dec 2020 • Yao Feng, Haiwen Feng, Michael J. Black, Timo Bolkart
Some methods produce faces that cannot be realistically animated because they do not model how wrinkles vary with expression.
no code implementations • CVPR 2021 • Yan Zhang, Michael J. Black, Siyu Tang
We note that motion prediction methods accumulate errors over time, resulting in joints or markers that diverge from true human bodies.
2 code implementations • ICCV 2021 • Yu Sun, Qian Bao, Wu Liu, Yili Fu, Michael J. Black, Tao Mei
Through a body-center-guided sampling process, the body mesh parameters of all people in the image are easily extracted from the Mesh Parameter map.
Ranked #1 on 3D Multi-Person Mesh Recovery on Relative Human (using extra training data)
2 code implementations • ECCV 2020 • Omid Taheri, Nima Ghorbani, Michael J. Black, Dimitrios Tzionas
Training computers to understand, model, and synthesize human grasping requires a rich dataset containing complex 3D object shapes, detailed contact information, hand pose and shape, and the 3D body motion over time.
1 code implementation • ECCV 2020 • Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, Michael J. Black
To understand how people look, interact, or perform tasks, we need to quickly and accurately capture their 3D body, face, and hands together from an RGB image.
1 code implementation • ECCV 2020 • Ahmed A. A. Osman, Timo Bolkart, Michael J. Black
The SMPL body model is widely used for the estimation, synthesis, and analysis of 3D human pose and shape.
1 code implementation • 16 Aug 2020 • Sergey Prokudin, Michael J. Black, Javier Romero
Recent advances in deep generative models have led to an unprecedented level of realism for synthetically generated images of humans.
1 code implementation • 12 Aug 2020 • Siwei Zhang, Yan Zhang, Qianli Ma, Michael J. Black, Siyu Tang
To synthesize realistic human-scene interactions, it is essential to effectively represent the physical contact and proximity between the body and the world.
no code implementations • 27 Jul 2020 • Yan Zhang, Michael J. Black, Siyu Tang
To address this problem, we propose a model to generate non-deterministic, \textit{ever-changing}, perpetual human motion, in which the global trajectory and the body pose are cross-conditioned.
no code implementations • 13 Jul 2020 • Rahul Tallamraju, Nitin Saini, Elia Bonetto, Michael Pabst, Yu Tang Liu, Michael J. Black, Aamir Ahmad
We focus on vision-based MoCap, where the objective is to estimate the trajectory of body pose and shape of a single moving person using multiple micro aerial vehicles.
no code implementations • 6 Jan 2020 • Nadine Rueegg, Christoph Lassner, Michael J. Black, Konrad Schindler
The goal of many computer vision systems is to transform image pixels into 3D representations.
5 code implementations • CVPR 2020 • Muhammed Kocabas, Nikos Athanasiou, Michael J. Black
Human motion is fundamental to understanding behavior.
Ranked #33 on Monocular 3D Human Pose Estimation on Human3.6M
3 code implementations • CVPR 2020 • Yan Zhang, Mohamed Hassan, Heiko Neumann, Michael J. Black, Siyu Tang
However, this is a challenging task for a computer as solving it requires that (1) the generated human bodies to be semantically plausible within the 3D environment (e. g. people sitting on the sofa or cooking near the stove), and (2) the generated human-scene interaction to be physically feasible such that the human body and scene do not interpenetrate while, at the same time, body-scene contact supports physical interactions.
2 code implementations • 24 Oct 2019 • Anurag Ranjan, David T. Hoffmann, Dimitrios Tzionas, Siyu Tang, Javier Romero, Michael J. Black
Therefore, we develop a dataset of multi-human optical flow and train optical flow networks on this dataset.
1 code implementation • ICCV 2019 • Anurag Ranjan, Joel Janai, Andreas Geiger, Michael J. Black
In this paper, we extend adversarial patch attacks to optical flow networks and show that such attacks can compromise their performance.
1 code implementation • ICCV 2019 • Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, Kostas Daniilidis
Our approach is self-improving by nature, since better network estimates can lead the optimization to better solutions, while more accurate optimization fits provide better supervision for the network.
1 code implementation • ICCV 2019 • Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, Michael J. Black
To motivate this, we show that current 3D human pose estimation methods produce results that are not consistent with the 3D scene.
1 code implementation • ICCV 2019 • Silvia Zuffi, Angjoo Kanazawa, Tanya Berger-Wolf, Michael J. Black
In contrast to research on human pose, shape and texture estimation, training data for endangered species is limited, the animals are in complex natural scenes with occlusion, they are naturally camouflaged, travel in herds, and look similar to each other.
1 code implementation • CVPR 2020 • Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, Michael J. Black
To our knowledge, this is the first generative model that directly dresses 3D human body meshes and generalizes to different poses.
2 code implementations • CVPR 2019 • Soubhik Sanyal, Timo Bolkart, Haiwen Feng, Michael J. Black
The estimation of 3D face shape from a single image must be robust to variations in lighting, head pose, expression, facial hair, makeup, and occlusions.
1 code implementation • CVPR 2019 • Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, Michael J. Black
To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers.
1 code implementation • CVPR 2019 • Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, Michael J. Black
We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild.
Ranked #1 on 3D Human Reconstruction on Expressive hands and faces dataset (EHF) (TR V2V (mm), left hand metric)
3 code implementations • CVPR 2019 • Yana Hasson, Gül Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, Cordelia Schmid
Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation.
Ranked #7 on hand-object pose on DexYCB
4 code implementations • ICCV 2019 • Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, Michael J. Black
We achieve this using a new method, MoSh++, that converts mocap data into realistic 3D human meshes represented by a rigged body model; here we use SMPL [doi:10. 1145/2816795. 2818013], which is widely used and provides a standard skeletal representation as well as a fully rigged surface mesh.
no code implementations • 17 Oct 2018 • Nikolas Hesse, Sergi Pujades, Michael J. Black, Michael Arens, Ulrich G. Hofmann, A. Sebastian Schroeder
To demonstrate the applicability of SMIL, we fit the model to RGB-D sequences of freely moving infants and show, with a case study, that our method captures enough motion detail for General Movements Assessment (GMA), a method used in clinical practice for early detection of neurodevelopmental disorders in infants.
1 code implementation • 10 Oct 2018 • Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J. Black, Otmar Hilliges, Gerard Pons-Moll
To learn from sufficient data, we synthesize IMU data from motion capture datasets.
no code implementations • 21 Sep 2018 • Jonas Wulff, Michael J. Black
The difficulty of annotating training data is a major obstacle to using CNNs for low-level tasks in video.
no code implementations • ECCV 2018 • Timo von Marcard, Roberto Henschel, Michael J. Black, Bodo Rosenhahn, Gerard Pons-Moll
In this work, we propose a method that combines a single hand-held camera and a set of Inertial Measurement Units (IMUs) attached at the body limbs to estimate accurate 3D poses in the wild.
2 code implementations • ECCV 2018 • Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, Michael J. Black
To address this, we introduce a versatile model that learns a non-linear representation of a face using spectral convolutions on a mesh surface.
Ranked #4 on Face Alignment on FaceScape
1 code implementation • 14 Jun 2018 • Anurag Ranjan, Javier Romero, Michael J. Black
Given this, we devise an optical flow algorithm specifically for human motion and show that it is superior to generic flow methods.
no code implementations • CVPR 2018 • Silvia Zuffi, Angjoo Kanazawa, Michael J. Black
Animals are widespread in nature and the analysis of their shape and motion is important in many fields and industries.
no code implementations • 31 May 2018 • Partha Ghosh, Arpan Losalka, Michael J. Black
Our model has the form of a variational autoencoder, with a Gaussian mixture prior on the latent vector.
1 code implementation • CVPR 2019 • Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, Michael J. Black
We address the unsupervised learning of several interconnected problems in low-level vision: single view depth prediction, camera motion estimation, optical flow, and segmentation of a video into the static scene and moving regions.
Ranked #71 on Monocular Depth Estimation on KITTI Eigen split
no code implementations • 22 Dec 2017 • Laura Sevilla-Lara, Yiyi Liao, Fatma Guney, Varun Jampani, Andreas Geiger, Michael J. Black
Here we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better.
9 code implementations • CVPR 2018 • Angjoo Kanazawa, Michael J. Black, David W. Jacobs, Jitendra Malik
The main objective is to minimize the reprojection loss of keypoints, which allow our model to be trained using images in-the-wild that only have ground truth 2D annotations.
Ranked #1 on Weakly-supervised 3D Human Pose Estimation on Human3.6M (3D Annotations metric)
9 code implementations • SIGGRAPH Asia 2017 • Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, Javier Romero
FLAME is low-dimensional but more expressive than the FaceWarehouse model and the Basel Face Model.
Ranked #3 on Face Alignment on FaceScape
no code implementations • 24 Jul 2017 • Yinghao Huang, Federica Bogo, Christoph Lassner, Angjoo Kanazawa, Peter V. Gehler, Ijaz Akhter, Michael J. Black
Existing marker-less motion capture methods often assume known backgrounds, static cameras, and sequence specific motion priors, which narrows its application scenarios.
no code implementations • CVPR 2017 • Federica Bogo, Javier Romero, Gerard Pons-Moll, Michael J. Black
We propose a new mesh registration method that uses both 3D geometry and texture information to register all scans in a sequence to a common reference topology.
no code implementations • CVPR 2017 • Joel Janai, Fatma Guney, Jonas Wulff, Michael J. Black, Andreas Geiger
Existing optical flow datasets are limited in size and variability due to the difficulty of capturing dense ground truth.
no code implementations • CVPR 2017 • Ali Osman Ulusoy, Michael J. Black, Andreas Geiger
Due to its probabilistic nature, the approach is able to cope with the approximate geometry of the 3D models as well as input shapes that are not present in the scene.
8 code implementations • CVPR 2017 • Julieta Martinez, Michael J. Black, Javier Romero
Human motion modelling is a classical problem at the intersection of graphics and computer vision, with applications spanning human-computer interaction, motion synthesis, and motion prediction for virtual and augmented reality.
no code implementations • CVPR 2017 • Jonas Wulff, Laura Sevilla-Lara, Michael J. Black
Existing algorithms typically focus on either recovering motion and structure under the assumption of a purely static world or optical flow for general unconstrained scenes.
Ranked #13 on Optical Flow Estimation on Sintel-clean
no code implementations • 23 Mar 2017 • Timo von Marcard, Bodo Rosenhahn, Michael J. Black, Gerard Pons-Moll
We address the problem of making human motion capture in the wild more practical by using a small set of inertial sensors attached to the body.
2 code implementations • CVPR 2017 • Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael J. Black, Peter V. Gehler
With a comprehensive set of experiments, we show how this data can be used to train discriminative models that produce results with an unprecedented level of detail: our models predict 31 segments and 91 landmark locations on the body.
Ranked #1 on Monocular 3D Human Pose Estimation on Human3.6M (Use Video Sequence metric)
3D human pose and shape estimation Monocular 3D Human Pose Estimation
2 code implementations • CVPR 2017 • Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, Cordelia Schmid
In this work we present SURREAL (Synthetic hUmans foR REAL tasks): a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data.
no code implementations • CVPR 2017 • Silvia Zuffi, Angjoo Kanazawa, David Jacobs, Michael J. Black
The best human body models are learned from thousands of 3D scans of people in specific poses, which is infeasible with live animals.
8 code implementations • CVPR 2017 • Anurag Ranjan, Michael J. Black
We learn to compute optical flow by combining a classical spatial-pyramid formulation with deep learning.
Ranked #8 on Dense Pixel Correspondence Estimation on HPatches
Dense Pixel Correspondence Estimation Optical Flow Estimation
2 code implementations • 27 Jul 2016 • Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, Michael J. Black
We then fit (top-down) a recently published statistical body shape model, called SMPL, to the 2D joints.
Ranked #31 on 3D Human Pose Estimation on HumanEva-I
no code implementations • CVPR 2016 • Yi-Hsuan Tsai, Ming-Hsuan Yang, Michael J. Black
Video object segmentation is challenging due to fast moving objects, deforming shapes, and cluttered backgrounds.
Ranked #74 on Semi-Supervised Video Object Segmentation on DAVIS 2016 (using extra training data)
no code implementations • CVPR 2016 • Huan Fu, Chaohui Wang, DaCheng Tao, Michael J. Black
Occlusion boundaries contain rich perceptual information about the underlying scene structure.
no code implementations • CVPR 2016 • Ali Osman Ulusoy, Michael J. Black, Andreas Geiger
In this paper, we propose a non-local structured prior for volumetric multi-view 3D reconstruction.
no code implementations • CVPR 2016 • Laura Sevilla-Lara, Deqing Sun, Varun Jampani, Michael J. Black
Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow.
no code implementations • ICCV 2015 • Federica Bogo, Michael J. Black, Matthew Loper, Javier Romero
The method then uses geometry and image texture over time to obtain accurate shape, pose, and appearance information despite unconstrained motion, partial views, varying resolution, occlusion, and soft tissue deformation.
no code implementations • ICCV 2015 • Naejin Kong, Michael J. Black
In contrast to raw RGB values, albedo and shading provide a richer, more physical, foundation for depth transfer.
no code implementations • CVPR 2015 • Ijaz Akhter, Michael J. Black
Second, we define a general parametrization of body pose and a new, multi-stage, method to estimate 3D pose from 2D joint locations using an over-complete dictionary of poses.
Ranked #136 on 3D Human Pose Estimation on Human3.6M (PA-MPJPE metric)
no code implementations • CVPR 2015 • Jonas Wulff, Michael J. Black
Given a set of sparse matches, we regress to dense optical flow using a learned set of full-frame basis flow fields.
no code implementations • CVPR 2015 • Silvia Zuffi, Michael J. Black
We propose a new 3D model of the human body that is both realistic and part-based.
no code implementations • CVPR 2014 • Oren Freifeld, Soren Hauberg, Michael J. Black
We demonstrate the approach by transferring PCA and logistic-regression models of real-world data involving 3D shapes and image descriptors.
no code implementations • CVPR 2014 • Federica Bogo, Javier Romero, Matthew Loper, Michael J. Black
We address this with a novel mesh registration technique that combines 3D shape and appearance information to produce high-quality alignments.
no code implementations • CVPR 2014 • Soren Hauberg, Aasa Feragen, Michael J. Black
We exploit that averages can be made robust to formulate the Robust Grassmann Average (RGA) as a form of robust PCA.