Search Results for author: Christian Theobalt

Found 223 papers, 53 papers with code

HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization

no code implementations ECCV 2020 Neng Qian, Jiayi Wang, Franziska Mueller, Florian Bernard, Vladislav Golyanik, Christian Theobalt

3D hand reconstruction from images is a widely-studied problem in computer vision and graphics, and has a particularly high relevance for virtual and augmented reality.

Neural Rendering

Neural Dense Non-Rigid Structure from Motion with Latent Space Constraints

no code implementations ECCV 2020 Vikramjit Sidhu, Edgar Tretschk, Vladislav Golyanik, Antonio Agudo, Christian Theobalt

We introduce the first dense neural non-rigid structure from motion (N-NRSfM) approach, which can be trained end-to-end in an unsupervised manner from 2D point tracks.

3D Shape Reconstruction Decoder

Live2Diff: Live Stream Translation via Uni-directional Attention in Video Diffusion Models

no code implementations11 Jul 2024 Zhening Xing, Gereon Fox, Yanhong Zeng, Xingang Pan, Mohamed Elgharib, Christian Theobalt, Kai Chen

State-of-the-art video diffusion models leverage bi-directional temporal attention to model the correlations between the current frame and all the surrounding (i. e. including future) frames, which hinders them from processing streaming videos.

Denoising Translation

DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image

no code implementations26 Jun 2024 Qingxuan Wu, Zhiyang Dou, Sirui Xu, Soshi Shimada, Chen Wang, Zhengming Yu, YuAn Liu, Cheng Lin, Zeyu Cao, Taku Komura, Vladislav Golyanik, Christian Theobalt, Wenping Wang, Lingjie Liu

The first and only method for hand-face interaction recovery, Decaf, introduces a global fitting optimization guided by contact and deformation estimation networks trained on studio-collected data with 3D annotations.

D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video

no code implementations14 Jun 2024 Moritz Kappel, Florian Hahlbohm, Timon Scholz, Susana Castillo, Christian Theobalt, Martin Eisemann, Vladislav Golyanik, Marcus Magnor

By sampling a discrete point cloud from our model, we can efficiently render high-quality novel views using a fast differentiable rasterizer and neural rendering network.

Dynamic Reconstruction Monocular Depth Estimation +3

Learning Images Across Scales Using Adversarial Training

no code implementations13 Jun 2024 Krzysztof Wolski, Adarsh Djeacoumar, Alireza Javanmardi, Hans-Peter Seidel, Christian Theobalt, Guillaume Cordonnier, Karol Myszkowski, George Drettakis, Xingang Pan, Thomas Leimkühler

We show that our generator can be used as a multiscale generative model, and for reconstructions of scale spaces from unstructured patches.

FaceGPT: Self-supervised Learning to Chat about 3D Human Faces

no code implementations11 Jun 2024 Haoran Wang, Mohit Mendiratta, Christian Theobalt, Adam Kortylewski

We introduce FaceGPT, a self-supervised learning framework for Large Vision-Language Models (VLMs) to reason about 3D human faces from images and text.

3D Face Reconstruction Face Model +2

Neural Gaussian Scale-Space Fields

no code implementations31 May 2024 Felix Mujkanovic, Ntumba Elie Nsampi, Christian Theobalt, Hans-Peter Seidel, Thomas Leimkühler

Our neural Gaussian scale-space fields faithfully capture multiscale representations across a broad range of modalities, and support a diverse set of applications.

Evolutive Rendering Models

no code implementations27 May 2024 Fangneng Zhan, Hanxue Liang, Yifan Wang, Michael Niemeyer, Michael Oechsle, Adam Kortylewski, Cengiz Oztireli, Gordon Wetzstein, Christian Theobalt

Central to this framework is the development of differentiable versions of these rendering elements, allowing for effective gradient backpropagation from the final rendering objectives.

EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams

1 code implementation CVPR 2024 Christen Millerdurai, Hiroyasu Akada, Jian Wang, Diogo Luvizon, Christian Theobalt, Vladislav Golyanik

In response to the existing limitations, this paper 1) introduces a new problem, i. e., 3D human motion capture from an egocentric monocular event camera with a fisheye lens, and 2) proposes the first approach to it called EventEgo3D (EE3D).

3D Pose Estimation 3D Reconstruction +1

MetaCap: Meta-learning Priors from Multi-View Imagery for Sparse-view Human Performance Capture and Rendering

no code implementations27 Mar 2024 Guoxing Sun, Rishabh Dabral, Pascal Fua, Christian Theobalt, Marc Habermann

Our key idea is to meta-learn the radiance field weights solely from potentially sparse multi-view videos, which can serve as a prior when fine-tuning them on sparse imagery depicting the human.

Meta-Learning Novel View Synthesis

Recent Trends in 3D Reconstruction of General Non-Rigid Scenes

no code implementations22 Mar 2024 Raza Yunus, Jan Eric Lenssen, Michael Niemeyer, Yiyi Liao, Christian Rupprecht, Christian Theobalt, Gerard Pons-Moll, Jia-Bin Huang, Vladislav Golyanik, Eddy Ilg

Reconstructing models of the real world, including 3D geometry, appearance, and motion of real scenes, is essential for computer graphics and computer vision.

3D Reconstruction Navigate

StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting

no code implementations12 Mar 2024 Kunhao Liu, Fangneng Zhan, Muyu Xu, Christian Theobalt, Ling Shao, Shijian Lu

We introduce StyleGaussian, a novel 3D style transfer technique that allows instant transfer of any image's style to a 3D scene at 10 frames per second (fps).

Decoder Style Transfer

Blue noise for diffusion models

1 code implementation7 Feb 2024 Xingchang Huang, Corentin Salaün, Cristina Vasconcelos, Christian Theobalt, Cengiz Öztireli, Gurprit Singh

In this paper, we introduce a novel and general class of diffusion models taking correlated noise within and across images into account.


MACS: Mass Conditioned 3D Hand and Object Motion Synthesis

no code implementations22 Dec 2023 Soshi Shimada, Franziska Mueller, Jan Bednarik, Bardia Doosti, Bernd Bickel, Danhang Tang, Vladislav Golyanik, Jonathan Taylor, Christian Theobalt, Thabo Beeler

To improve the naturalness of the synthesized 3D hand object motions, this work proposes MACS the first MAss Conditioned 3D hand and object motion Synthesis approach.

Motion Synthesis Object

3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera

1 code implementation21 Dec 2023 Christen Millerdurai, Diogo Luvizon, Viktor Rudnev, André Jonas, Jiayi Wang, Christian Theobalt, Vladislav Golyanik

3D hand tracking from a monocular video is a very challenging problem due to hand interactions, occlusions, left-right hand ambiguity, and fast motion.

3D Pose Estimation 3D Reconstruction

Relightable Neural Actor with Intrinsic Decomposition and Pose Control

no code implementations18 Dec 2023 Diogo Luvizon, Vladislav Golyanik, Adam Kortylewski, Marc Habermann, Christian Theobalt

Creating a digital human avatar that is relightable, drivable, and photorealistic is a challenging and important problem in Vision and Graphics.

ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering

1 code implementation CVPR 2024 Haokai Pang, Heming Zhu, Adam Kortylewski, Christian Theobalt, Marc Habermann

Real-time rendering of photorealistic and controllable human avatars stands as a cornerstone in Computer Vision and Graphics.

TriHuman : A Real-time and Controllable Tri-plane Representation for Detailed Human Geometry and Appearance Synthesis

1 code implementation8 Dec 2023 Heming Zhu, Fangneng Zhan, Christian Theobalt, Marc Habermann

Creating controllable, photorealistic, and geometrically detailed digital doubles of real humans solely from video data is a key challenge in Computer Graphics and Vision, especially when real-time performance is required.

ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions

no code implementations28 Nov 2023 Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, Philipp Slusallek

Current approaches for 3D human motion synthesis generate high-quality animations of digital humans performing a wide variety of actions and gestures.

Denoising Motion Synthesis

DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields

no code implementations18 Nov 2023 Yu Chi, Fangneng Zhan, Sibo Wu, Christian Theobalt, Adam Kortylewski

The generated data is applicable across various computer vision tasks, including video segmentation and 3D point cloud segmentation.

Decoder Point Cloud Segmentation +3

Wonder3D: Single Image to 3D using Cross-Domain Diffusion

1 code implementation CVPR 2024 Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, YuAn Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, Wenping Wang

In this work, we introduce Wonder3D, a novel method for efficiently generating high-fidelity textured meshes from single-view images. Recent methods based on Score Distillation Sampling (SDS) have shown the potential to recover 3D geometry from 2D diffusion priors, but they typically suffer from time-consuming per-shape optimization and inconsistent geometry.

Image to 3D Single-View 3D Reconstruction

State of the Art on Diffusion Models for Visual Computing

no code implementations11 Oct 2023 Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T. Barron, Amit H. Bermano, Eric Ryan Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, C. Karen Liu, Lingjie Liu, Ben Mildenhall, Matthias Nießner, Björn Ommer, Christian Theobalt, Peter Wonka, Gordon Wetzstein

The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes.

Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering

1 code implementation30 Sep 2023 Linjie Lyu, Ayush Tewari, Marc Habermann, Shunsuke Saito, Michael Zollhöfer, Thomas Leimkühler, Christian Theobalt

We further conduct an extensive comparative study of different priors on illumination used in previous work on inverse rendering.

Denoising Inverse Rendering

Decaf: Monocular Deformation Capture for Face and Hand Interactions

no code implementations28 Sep 2023 Soshi Shimada, Vladislav Golyanik, Patrick Pérez, Christian Theobalt

At the core of our neural approach are a variational auto-encoder supplying the hand-face depth prior and modules that guide the 3D tracking by estimating the contacts and the deformations.

NeuralClothSim: Neural Deformation Fields Meet the Thin Shell Theory

no code implementations24 Aug 2023 Navami Kairanda, Marc Habermann, Christian Theobalt, Vladislav Golyanik

Despite existing 3D cloth simulators producing realistic results, they predominantly operate on discrete surface representations (e. g. points and meshes) with a fixed spatial resolution, which often leads to large memory consumption and resolution-dependent simulations.

ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors

no code implementations24 Aug 2023 Wanyue Zhang, Rishabh Dabral, Thomas Leimkühler, Vladislav Golyanik, Marc Habermann, Christian Theobalt

Given an unseen object and a reference pose-object pair, we optimise for the object-aware pose that is closest in the feature space to the reference pose.

Motion Synthesis Object

VINECS: Video-based Neural Character Skinning

no code implementations CVPR 2024 Zhouyingcheng Liao, Vladislav Golyanik, Marc Habermann, Christian Theobalt

However, the former methods typically predict solely static skinning weights, which perform poorly for highly articulated poses, and the latter ones either require dense 3D character scans in different poses or cannot generate an explicit mesh with vertex correspondence over time.

AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars

no code implementations1 Jun 2023 Mohit Mendiratta, Xingang Pan, Mohamed Elgharib, Kartik Teotia, Mallikarjun B R, Ayush Tewari, Vladislav Golyanik, Adam Kortylewski, Christian Theobalt

Our method edits the full head in a canonical space, and then propagates these edits to remaining time steps via a pretrained deformation network.

Weakly Supervised 3D Open-vocabulary Segmentation

1 code implementation NeurIPS 2023 Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, Shijian Lu

Open-vocabulary segmentation of 3D scenes is a fundamental function of human perception and thus a crucial objective in computer vision research.


Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold

5 code implementations18 May 2023 Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, Christian Theobalt

Synthesizing visual content that meets users' needs often requires flexible and precise controllability of the pose, shape, expression, and layout of the generated objects.

Image Manipulation Point Tracking +1

General Neural Gauge Fields

1 code implementation5 May 2023 Fangneng Zhan, Lingjie Liu, Adam Kortylewski, Christian Theobalt

In this work, we extend this problem to a general paradigm with a taxonomy of discrete \& continuous cases, and develop a learning framework to jointly optimize gauge transformations and neural fields.

Representation Learning

EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors

no code implementations2 May 2023 Xinyu Yi, Yuxiao Zhou, Marc Habermann, Vladislav Golyanik, Shaohua Pan, Christian Theobalt, Feng Xu

We integrate the two techniques together in EgoLocate, a system that simultaneously performs human motion capture (mocap), localization, and mapping in real time from sparse body-mounted sensors, including 6 inertial measurement units (IMUs) and a monocular phone camera.

Simultaneous Localization and Mapping

GVP: Generative Volumetric Primitives

no code implementations31 Mar 2023 Mallikarjun B R, Xingang Pan, Mohamed Elgharib, Christian Theobalt

Advances in 3D-aware generative models have pushed the boundary of image synthesis with explicit camera control.

Image Generation Knowledge Distillation

F$^{2}$-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories

1 code implementation28 Mar 2023 Peng Wang, YuAn Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, Wenping Wang

Based on our analysis, we further propose a novel space-warping method called perspective warping, which allows us to handle arbitrary trajectories in the grid-based NeRF framework.

Novel View Synthesis

Grid-guided Neural Radiance Fields for Large Urban Scenes

no code implementations CVPR 2023 Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, Dahua Lin

An alternative solution is to use a feature grid representation, which is computationally efficient and can naturally scale to a large scene with increased grid resolutions.

Regularized Vector Quantization for Tokenized Image Synthesis

no code implementations CVPR 2023 Jiahui Zhang, Fangneng Zhan, Christian Theobalt, Shijian Lu

The first is a prior distribution regularization which measures the discrepancy between a prior token distribution and the predicted token distribution to avoid codebook collapse and low codebook utilization.

Image Generation Quantization

NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion

no code implementations20 Feb 2023 Jiatao Gu, Alex Trevithick, Kai-En Lin, Josh Susskind, Christian Theobalt, Lingjie Liu, Ravi Ramamoorthi

Novel view synthesis from a single image requires inferring occluded regions of objects and scenes whilst simultaneously maintaining semantic and physical consistency with the input.

Novel View Synthesis

Scene-Aware 3D Multi-Human Motion Capture from a Single Camera

1 code implementation12 Jan 2023 Diogo Luvizon, Marc Habermann, Vladislav Golyanik, Adam Kortylewski, Christian Theobalt

In this work, we consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera.


F2-NeRF: Fast Neural Radiance Field Training With Free Camera Trajectories

no code implementations CVPR 2023 Peng Wang, YuAn Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, Wenping Wang

Existing fast grid-based NeRF training frameworks, like Instant-NGP, Plenoxels, DVGO, or TensoRF, are mainly designed for bounded scenes and rely on space warping to handle unbounded scenes.

Novel View Synthesis

Imitator: Personalized Speech-driven 3D Facial Animation

no code implementations ICCV 2023 Balamurugan Thambiraja, Ikhsanul Habibie, Sadegh Aliakbarian, Darren Cosker, Christian Theobalt, Justus Thies

To address this, we present Imitator, a speech-driven facial expression synthesis method, which learns identity-specific details from a short input video and produces novel facial expressions matching the identity-specific speaking style and facial idiosyncrasies of the target actor.

Scene-aware Egocentric 3D Human Pose Estimation

1 code implementation CVPR 2023 Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, Diogo Luvizon, Christian Theobalt

To this end, we propose an egocentric depth estimation network to predict the scene depth map from a wide-view egocentric fisheye camera while mitigating the occlusion of the human body with a depth-inpainting network.

Ranked #3 on Egocentric Pose Estimation on GlobalEgoMocap Test Dataset (using extra training data)

Depth Estimation Egocentric Pose Estimation

NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction

1 code implementation ICCV 2023 Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, Lingjie Liu

Recent methods for neural surface representation and rendering, for example NeuS, have demonstrated the remarkably high-quality reconstruction of static scenes.

Surface Reconstruction

MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis

no code implementations CVPR 2023 Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, Christian Theobalt

Conventional methods for human motion synthesis are either deterministic or struggle with the trade-off between motion diversity and motion quality.

Denoising Diversity +1

NeuralUDF: Learning Unsigned Distance Fields for Multi-view Reconstruction of Surfaces with Arbitrary Topologies

no code implementations CVPR 2023 Xiaoxiao Long, Cheng Lin, Lingjie Liu, YuAn Liu, Peng Wang, Christian Theobalt, Taku Komura, Wenping Wang

In this paper, we propose to represent surfaces as the Unsigned Distance Function (UDF) and develop a new volume rendering scheme to learn the neural UDF representation.

Neural Rendering

An Implicit Parametric Morphable Dental Model

no code implementations21 Nov 2022 Congyi Zhang, Mohamed Elgharib, Gereon Fox, Min Gu, Christian Theobalt, Wenping Wang

Current dental models use an explicit mesh scene representation and model only the teeth, ignoring the gum.

Batch-based Model Registration for Fast 3D Sherd Reconstruction

no code implementations ICCV 2023 Jiepeng Wang, Congyi Zhang, Peng Wang, Xin Li, Peter J. Cobb, Christian Theobalt, Wenping Wang

In this work, we aim to develop a portable, high-throughput, and accurate reconstruction system for efficient digitization of fragments excavated in archaeological sites.

3D Reconstruction

gCoRF: Generative Compositional Radiance Fields

no code implementations31 Oct 2022 Mallikarjun BR, Ayush Tewari, Xingang Pan, Mohamed Elgharib, Christian Theobalt

We start with a global generative model (GAN) and learn to decompose it into different semantic parts using supervision from 2D segmentation masks.

Image Generation

State of the Art in Dense Monocular Non-Rigid 3D Reconstruction

no code implementations27 Oct 2022 Edith Tretschk, Navami Kairanda, Mallikarjun B R, Rishabh Dabral, Adam Kortylewski, Bernhard Egger, Marc Habermann, Pascal Fua, Christian Theobalt, Vladislav Golyanik

3D reconstruction of deformable (or non-rigid) scenes from a set of monocular 2D image observations is a long-standing and actively researched area of computer vision and graphics.

3D Reconstruction

HDHumans: A Hybrid Approach for High-fidelity Digital Humans

no code implementations21 Oct 2022 Marc Habermann, Lingjie Liu, Weipeng Xu, Gerard Pons-Moll, Michael Zollhoefer, Christian Theobalt

Photo-real digital human avatars are of enormous importance in graphics, as they enable immersive communication over the globe, improve gaming and entertainment experiences, and can be particularly beneficial for AR and VR settings.

Novel View Synthesis Surface Reconstruction +1

HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances

no code implementations11 Oct 2022 Yue Jiang, Marc Habermann, Vladislav Golyanik, Christian Theobalt

Furthermore, we show that HiFECap outperforms the state-of-the-art human performance capture approaches qualitatively and quantitatively while for the first time capturing all aspects of the human.

Vocal Bursts Intensity Prediction

HandFlow: Quantifying View-Dependent 3D Ambiguity in Two-Hand Reconstruction with Normalizing Flow

no code implementations4 Oct 2022 Jiayi Wang, Diogo Luvizon, Franziska Mueller, Florian Bernard, Adam Kortylewski, Dan Casas, Christian Theobalt

Through this, we demonstrate the quality of our probabilistic reconstruction and show that explicit ambiguity modeling is better-suited for this challenging problem.


Voxurf: Voxel-based Efficient and Accurate Neural Surface Reconstruction

1 code implementation26 Aug 2022 Tong Wu, Jiaqi Wang, Xingang Pan, Xudong Xu, Christian Theobalt, Ziwei Liu, Dahua Lin

Previous methods based on neural volume rendering mostly train a fully implicit model with MLPs, which typically require hours of training for a single scene.

Surface Reconstruction

Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors

no code implementations25 Aug 2022 Yiming Wang, Qingzhe Gao, Libin Liu, Lingjie Liu, Christian Theobalt, Baoquan Chen

The learned representation can be used to synthesize novel view images of an arbitrary person from a sparse set of cameras, and further animate them with the user's pose control.


MoCapDeform: Monocular 3D Human Motion Capture in Deformable Scenes

1 code implementation17 Aug 2022 Zhi Li, Soshi Shimada, Bernt Schiele, Christian Theobalt, Vladislav Golyanik

3D human motion capture from monocular RGB images respecting interactions of a subject with complex and possibly deformable environments is a very challenging, ill-posed and under-explored problem.

3D Human Pose Estimation

Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination

no code implementations27 Jul 2022 Linjie Lyu, Ayush Tewari, Thomas Leimkuehler, Marc Habermann, Christian Theobalt

Given a set of images of a scene, the re-rendering of this scene from novel views and lighting conditions is an important and challenging problem in Computer Vision and Graphics.

Disentanglement Novel View Synthesis

NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors

no code implementations27 Jun 2022 Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, Wenping Wang

The key idea of NeuRIS is to integrate estimated normal of indoor scenes as a prior in a neural rendering framework for reconstructing large texture-less shapes and, importantly, to do this in an adaptive manner to also enable the reconstruction of irregular shapes with fine details.

3D Reconstruction Neural Rendering

Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motions

no code implementations25 Jun 2022 Weilin Wan, Lei Yang, Lingjie Liu, Zhuoying Zhang, Ruixing Jia, Yi-King Choi, Jia Pan, Christian Theobalt, Taku Komura, Wenping Wang

We also observe that an object's intrinsic physical properties are useful for the object motion prediction, and thus design a set of object dynamic descriptors to encode such intrinsic properties.

Graph Neural Network Human-Object Interaction Detection +2

EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

no code implementations CVPR 2023 Viktor Rudnev, Mohamed Elgharib, Christian Theobalt, Vladislav Golyanik

Asynchronously operating event cameras find many applications due to their high dynamic range, vanishingly low motion blur, low latency and low data bandwidth.

3D Reconstruction Novel View Synthesis +1

GAN2X: Non-Lambertian Inverse Rendering of Image GANs

no code implementations18 Jun 2022 Xingang Pan, Ayush Tewari, Lingjie Liu, Christian Theobalt

2D images are observations of the 3D physical world depicted with the geometry, material, and illumination components.

3D Face Reconstruction Inverse Rendering

Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data

no code implementations14 Jun 2022 Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans-Peter Seidel, Christian Theobalt, Rhaleb Zayer

With a hybrid architecture that separates static and dynamic contents, fluid interactions with static obstacles are reconstructed for the first time without additional geometry input or human labeling.

HULC: 3D Human Motion Capture with Pose Manifold Sampling and Dense Contact Guidance

no code implementations11 May 2022 Soshi Shimada, Vladislav Golyanik, Zhi Li, Patrick Pérez, Weipeng Xu, Christian Theobalt

Marker-less monocular 3D human motion capture (MoCap) with scene interactions is a challenging research topic relevant for extended reality, robotics and virtual avatar generation.

BEHAVE: Dataset and Method for Tracking Human Object Interactions

1 code implementation CVPR 2022 Bharat Lal Bhatnagar, Xianghui Xie, Ilya A. Petrov, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll

We present BEHAVE dataset, the first full body human- object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them.

Human-Object Interaction Detection Mixed Reality +1

Direct Dense Pose Estimation

no code implementations4 Apr 2022 Liqian Ma, Lingjie Liu, Christian Theobalt, Luc van Gool

In addition, DDP is computationally more efficient than previous dense pose estimation methods, and it reduces jitters when applied to a video sequence, which is a problem plaguing the previous methods.

Action Recognition Pose Estimation +2

Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images

no code implementations CVPR 2022 Ayush Tewari, Mallikarjun B R, Xingang Pan, Ohad Fried, Maneesh Agrawala, Christian Theobalt

Our model can disentangle the geometry and appearance variations in the scene, i. e., we can independently sample from the geometry and appearance spaces of the generative model.


φ-SfT: Shape-from-Template with a Physics-Based Deformation Model

no code implementations22 Mar 2022 Navami Kairanda, Edith Tretschk, Mohamed Elgharib, Christian Theobalt, Vladislav Golyanik

In contrast to previous works, this paper proposes a new SfT approach explaining 2D observations through physical simulations accounting for forces and material properties.

3D Reconstruction Physical Simulations

Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision

no code implementations CVPR 2022 Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, Diogo Luvizon, Christian Theobalt

Specifically, we first generate pseudo labels for the EgoPW dataset with a spatio-temporal optimization method by incorporating the external-view supervision.

Ranked #4 on Egocentric Pose Estimation on GlobalEgoMocap Test Dataset (using extra training data)

Egocentric Pose Estimation

f-SfT: Shape-From-Template With a Physics-Based Deformation Model

no code implementations CVPR 2022 Navami Kairanda, Edith Tretschk, Mohamed Elgharib, Christian Theobalt, Vladislav Golyanik

In contrast to previous works, this paper proposes a new SfT approach explaining 2D observations through physical simulations accounting for forces and material properties.

3D Reconstruction Physical Simulations

Multimodal Image Synthesis and Editing: The Generative AI Era

2 code implementations27 Dec 2021 Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu, Lingjie Liu, Adam Kortylewski, Christian Theobalt, Eric Xing

With superb power in modeling the interaction among multimodal information, multimodal image synthesis and editing has become a hot research topic in recent years.

Image Generation

BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering

no code implementations10 Dec 2021 Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, Dahua Lin

The wide span of viewing positions within these scenes yields multi-scale renderings with very different levels of detail, which poses great challenges to neural radiance field and biases it towards compromised results.

NeRF for Outdoor Scene Relighting

no code implementations9 Dec 2021 Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu, Vladislav Golyanik, Christian Theobalt

Photorealistic editing of outdoor scenes from photographs requires a profound understanding of the image formation process and an accurate estimation of the scene geometry, reflectance and illumination.

EgoRenderer: Rendering Human Avatars from Egocentric Camera Images

no code implementations ICCV 2021 Tao Hu, Kripasindhu Sarkar, Lingjie Liu, Matthias Zwicker, Christian Theobalt

We next combine the target pose image and the textures into a combined feature image, which is transformed into the output color image using a neural image translation network.

Texture Synthesis Translation

A Deeper Look into DeepCap

no code implementations20 Nov 2021 Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt

Human performance capture is a highly important computer vision problem with many applications in movie production and virtual/augmented reality.

Pose Estimation

A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis

1 code implementation NeurIPS 2021 Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, Bo Dai

Motivated by the observation that a 3D object should look realistic from multiple viewpoints, these methods introduce a multi-view constraint as regularization to learn valid 3D radiance fields from 2D images.

3D-Aware Image Synthesis 3D Shape Reconstruction +2

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

1 code implementation ICLR 2022 Jiatao Gu, Lingjie Liu, Peng Wang, Christian Theobalt

We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue.

Image Generation

Neural Rays for Occlusion-aware Image-based Rendering

1 code implementation CVPR 2022 YuAn Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, Wenping Wang

On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction.

Neural Rendering Novel View Synthesis +1

StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN

no code implementations15 Jul 2021 Gereon Fox, Ayush Tewari, Mohamed Elgharib, Christian Theobalt

We demonstrate that it suffices to train our temporal architecture on only 10 minutes of footage of 1 subject for about 6 hours.

Adiabatic Quantum Graph Matching with Permutation Matrix Constraints

no code implementations8 Jul 2021 Marcel Seelbach Benkner, Vladislav Golyanik, Christian Theobalt, Michael Moeller

In this work, we address such problems with emerging quantum computing technology and propose several reformulations of QAPs as unconstrained problems suitable for efficient execution on quantum hardware.

Graph Matching valid

Self-supervised Outdoor Scene Relighting

no code implementations ECCV 2020 Ye Yu, Abhimitra Meka, Mohamed Elgharib, Hans-Peter Seidel, Christian Theobalt, William A. P. Smith

Outdoor scene relighting is a challenging problem that requires good understanding of the scene geometry, illumination and albedo.

NRST: Non-rigid Surface Tracking from Monocular Video

no code implementations6 Jul 2021 Marc Habermann, Weipeng Xu, Helge Rhodin, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt

Our texture term exploits the orientation information in the micro-structures of the objects, e. g., the yarn patterns of fabrics.

HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks

no code implementations2 Jul 2021 Jameel Malik, Soshi Shimada, Ahmed Elhayek, Sk Aziz Ali, Christian Theobalt, Vladislav Golyanik, Didier Stricker

To address the limitations of the existing methods, we develop HandVoxNet++, i. e., a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner.

3D Hand Pose Estimation

RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

no code implementations22 Jun 2021 Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A. Otaduy, Dan Casas, Christian Theobalt

Moreover, we demonstrate that our approach offers previously unseen two-hand tracking performance from RGB, and quantitatively and qualitatively outperforms existing RGB-based methods that were not explicitly designed for two-hand interactions.

3D Reconstruction Sign Language Recognition

Fast Simultaneous Gravitational Alignment of Multiple Point Sets

no code implementations21 Jun 2021 Vladislav Golyanik, Soshi Shimada, Christian Theobalt

The problem of simultaneous rigid alignment of multiple unordered point sets which is unbiased towards any of the inputs has recently attracted increasing interest, and several reliable methods have been newly proposed.

NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

7 code implementations NeurIPS 2021 Peng Wang, Lingjie Liu, YuAn Liu, Christian Theobalt, Taku Komura, Wenping Wang

In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation.

Novel View Synthesis Surface Reconstruction

Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control

no code implementations3 Jun 2021 Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, Christian Theobalt

To address this problem, we utilize a coarse body model as the proxy to unwarp the surrounding 3D space into a canonical pose.

Q-Match: Iterative Shape Matching via Quantum Annealing

no code implementations ICCV 2021 Marcel Seelbach Benkner, Zorah Lähner, Vladislav Golyanik, Christof Wunderlich, Christian Theobalt, Michael Moeller

Finding shape correspondences can be formulated as an NP-hard quadratic assignment problem (QAP) that becomes infeasible for shapes with high sampling density.

Real-time Deep Dynamic Characters

no code implementations4 May 2021 Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt

We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery.

Neural Monocular 3D Human Motion Capture with Physical Awareness

no code implementations3 May 2021 Soshi Shimada, Vladislav Golyanik, Weipeng Xu, Patrick Pérez, Christian Theobalt

We present a new trainable system for physically plausible markerless 3D human motion capture, which achieves state-of-the-art results in a broad range of challenging scenarios.

3D Pose Estimation

Differentiable Event Stream Simulator for Non-Rigid 3D Tracking

no code implementations30 Apr 2021 Jalees Nehvi, Vladislav Golyanik, Franziska Mueller, Hans-Peter Seidel, Mohamed Elgharib, Christian Theobalt

This paper introduces the first differentiable simulator of event streams, i. e., streams of asynchronous brightness change signals recorded by event cameras.

Estimating Egocentric 3D Human Pose in Global Space

1 code implementation ICCV 2021 Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, Christian Theobalt

Furthermore, these methods suffer from limited accuracy and temporal instability due to ambiguities caused by the monocular setup and the severe occlusion in a strongly distorted egocentric perspective.

Ranked #5 on Egocentric Pose Estimation on SceneEgo (using extra training data)

Egocentric Pose Estimation

Synthesis of Compositional Animations from Textual Descriptions

1 code implementation ICCV 2021 Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, Philipp Slusallek

Our model can generate plausible pose sequences for short sentences describing single actions as well as long compositional sentences describing multiple sequential and superimposed actions.

Motion Synthesis Sentence

HumanGAN: A Generative Model of Humans Images

no code implementations11 Mar 2021 Kripasindhu Sarkar, Lingjie Liu, Vladislav Golyanik, Christian Theobalt

We address these limitations and present a generative model for images of dressed humans offering control over pose, local body part appearance and garment style.

Pose Transfer

Style and Pose Control for Image Synthesis of Humans from a Single Monocular View

no code implementations22 Feb 2021 Kripasindhu Sarkar, Vladislav Golyanik, Lingjie Liu, Christian Theobalt

Photo-realistic re-rendering of a human from a single image with explicit control over body pose, shape and appearance enables a wide range of applications, such as human appearance transfer, virtual try-on, motion imitation, and novel view synthesis.

Image Generation Novel View Synthesis +1

Learning Speech-driven 3D Conversational Gestures from Video

no code implementations13 Feb 2021 Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, Christian Theobalt

We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input.

3D Face Animation Generative Adversarial Network +2

Quantum Permutation Synchronization

no code implementations CVPR 2021 Tolga Birdal, Vladislav Golyanik, Christian Theobalt, Leonidas Guibas

We present QuantumSync, the first quantum algorithm for solving a synchronization problem in the context of computer vision.

Neural Re-Rendering of Humans from a Single Image

no code implementations ECCV 2020 Kripasindhu Sarkar, Dushyant Mehta, Weipeng Xu, Vladislav Golyanik, Christian Theobalt

Human re-rendering from a single image is a starkly under-constrained problem, and state-of-the-art algorithms often exhibit undesired artefacts, such as over-smoothing, unrealistic distortions of the body parts and garments, or implausible changes of the texture.


Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video

2 code implementations ICCV 2021 Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, Christian Theobalt

We show that a single handheld consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views, e. g. a `bullet-time' video effect.

Novel View Synthesis Video Editing

High-Fidelity Neural Human Motion Transfer from Monocular Video

1 code implementation CVPR 2021 Moritz Kappel, Vladislav Golyanik, Mohamed Elgharib, Jann-Ole Henningson, Hans-Peter Seidel, Susana Castillo, Christian Theobalt, Marcus Magnor

We address these limitations for the first time in the literature and present a new framework which performs high-fidelity and temporally-consistent human motion transfer with natural pose-dependent non-rigid deformations, for several types of loose garments.

Image Generation Vocal Bursts Intensity Prediction

Monocular Real-time Full Body Capture with Inter-part Correlations

no code implementations CVPR 2021 Yuxiao Zhou, Marc Habermann, Ikhsanul Habibie, Ayush Tewari, Christian Theobalt, Feng Xu

We present the first method for real-time full body capture that estimates shape and motion of body and hands together with a dynamic 3D face model from a single color image.

3D Hand Pose Estimation Computational Efficiency +1

EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream

1 code implementation ICCV 2021 Viktor Rudnev, Vladislav Golyanik, Jiayi Wang, Hans-Peter Seidel, Franziska Mueller, Mohamed Elgharib, Christian Theobalt

Due to the different data modality of event cameras compared to classical cameras, existing methods cannot be directly applied to and re-trained for event streams.

3D Hand Pose Estimation

Pose-Guided Human Animation from a Single Image in the Wild

no code implementations CVPR 2021 Jae Shin Yoon, Lingjie Liu, Vladislav Golyanik, Kripasindhu Sarkar, Hyun Soo Park, Christian Theobalt

We present a new pose transfer method for synthesizing a human animation from a single image of a person controlled by a sequence of body poses.

Pose Transfer

i3DMM: Deep Implicit 3D Morphable Model of Human Heads

1 code implementation CVPR 2021 Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed Elgharib, Daniel Cremers, Christian Theobalt

Our approach has the following favorable properties: (i) It is the first full head morphable model that includes hair.

Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks

1 code implementation CVPR 2021 Xiaoxiao Long, Lingjie Liu, Wei Li, Christian Theobalt, Wenping Wang

We present a novel method for multi-view depth estimation from a single video, which is a critical task in various applications, such as perception, reconstruction and robot navigation.

Depth Estimation Robot Navigation

Deep Physics-aware Inference of Cloth Deformation for Monocular Human Performance Capture

no code implementations25 Nov 2020 Yue Li, Marc Habermann, Bernhard Thomaszewski, Stelian Coros, Thabo Beeler, Christian Theobalt

Recent monocular human performance capture approaches have shown compelling dense tracking results of the full body from a single RGB camera.

LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration

no code implementations NeurIPS 2020 Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll

Formulating this closed loop is not straightforward because it is not trivial to force the output of the NN to be on the surface of the human model - outside this surface the human model is not even defined.

Self-Supervised Learning

Learning Complete 3D Morphable Face Models from Images and Videos

no code implementations CVPR 2021 Mallikarjun B R, Ayush Tewari, Hans-Peter Seidel, Mohamed Elgharib, Christian Theobalt

Our network design and loss functions ensure a disentangled parameterization of not only identity and albedo, but also, for the first time, an expression basis.

3D Face Reconstruction Monocular Reconstruction +1

Fast Gravitational Approach for Rigid Point Set Registration with Ordinary Differential Equations

no code implementations28 Sep 2020 Sk Aziz Ali, Kerem Kahraman, Christian Theobalt, Didier Stricker, Vladislav Golyanik

This article introduces a new physics-based method for rigid point set alignment called Fast Gravitational Approach (FGA).

PIE: Portrait Image Embedding for Semantic Control

no code implementations20 Sep 2020 Ayush Tewari, Mohamed Elgharib, Mallikarjun B R., Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt

We present the first approach for embedding real portrait images in the latent space of StyleGAN, which allows for intuitive editing of the head pose, facial expression, and scene illumination in the image.

Face Model

Monocular Reconstruction of Neural Face Reflectance Fields

no code implementations CVPR 2021 Mallikarjun B R., Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, Christian Theobalt

The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing.

Monocular Reconstruction

PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time

no code implementations20 Aug 2020 Soshi Shimada, Vladislav Golyanik, Weipeng Xu, Christian Theobalt

We, therefore, present PhysCap, the first algorithm for physically plausible, real-time and marker-less human 3D motion capture with a single colour camera at 25 fps.

Face2Face: Real-time Face Capture and Reenactment of RGB Videos

2 code implementations CVPR 2016 Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner

Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.

Neural Sparse Voxel Fields

1 code implementation NeurIPS 2020 Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, Christian Theobalt

We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering.

Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction

1 code implementation ECCV 2020 Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll

In this work, we present methodology that combines detail-rich implicit functions and parametric representations in order to reconstruct 3D models of people that remain controllable and accurate even in the presence of clothing.

3D Human Pose Estimation 3D Human Reconstruction

Vid2Curve: Simultaneous Camera Motion Estimation and Thin Structure Reconstruction from an RGB Video

1 code implementation7 May 2020 Peng Wang, Lingjie Liu, Nenglun Chen, Hung-Kuo Chu, Christian Theobalt, Wenping Wang

We propose the first approach that simultaneously estimates camera motion and reconstructs the geometry of complex 3D thin structures in high quality from a color video captured by a handheld camera.

Camera Pose Estimation Motion Estimation +2

State of the Art on Neural Rendering

no code implementations8 Apr 2020 Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B. Goldman, Michael Zollhöfer

Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e. g., by the integration of differentiable rendering into network training.

BIG-bench Machine Learning Image Generation +2

Occlusion-Aware Depth Estimation with Adaptive Normal Constraints

1 code implementation ECCV 2020 Xiaoxiao Long, Lingjie Liu, Christian Theobalt, Wenping Wang

We present a new learning-based method for multi-frame depth estimation from a color video, which is a fundamental problem in scene understanding, robot navigation or handheld 3D reconstruction.

3D Reconstruction Depth Estimation +2

StyleRig: Rigging StyleGAN for 3D Control over Portrait Images

no code implementations CVPR 2020 Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt

StyleGAN generates photorealistic portrait images of faces with eyes, teeth, hair and context (neck, shoulders, background), but lacks a rig-like control over semantic face parameters that are interpretable in 3D, such as face pose, expressions, and scene illumination.

Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data

2 code implementations CVPR 2020 Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, Feng Xu

We present a novel method for monocular hand shape and pose estimation at unprecedented runtime performance of 100fps and at state-of-the-art accuracy.

Pose Estimation

DeepCap: Monocular Human Performance Capture Using Weak Supervision

no code implementations CVPR 2020 Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt

Human performance capture is a highly important computer vision problem with many applications in movie production and virtual/augmented reality.

Pose Estimation

Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation

no code implementations14 Jan 2020 Lingjie Liu, Weipeng Xu, Marc Habermann, Michael Zollhoefer, Florian Bernard, Hyeongwoo Kim, Wenping Wang, Christian Theobalt

In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space.

Image-to-Image Translation Novel View Synthesis +1

Image-guided Neural Object Rendering

no code implementations ICLR 2020 Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.

Image Generation Object

A Quantum Computational Approach to Correspondence Problems on Point Sets

no code implementations CVPR 2020 Vladislav Golyanik, Christian Theobalt

Modern adiabatic quantum computers (AQC) are already used to solve difficult combinatorial optimisation problems in various domains of science.

Neural Voice Puppetry: Audio-driven Facial Reenactment

1 code implementation ECCV 2020 Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, Matthias Nießner

Neural Voice Puppetry has a variety of use-cases, including audio-driven video avatars, video dubbing, and text-driven video synthesis of a talking head.

Face Model Neural Rendering +2

DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data

1 code implementation9 Dec 2019 Aljaž Božič, Michael Zollhöfer, Christian Theobalt, Matthias Nießner

Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus.

3D Reconstruction RGB-D Reconstruction

Convex Optimisation for Inverse Kinematics

no code implementations24 Oct 2019 Tarun Yenamandra, Florian Bernard, Jiayi Wang, Franziska Mueller, Christian Theobalt

We consider the problem of inverse kinematics (IK), where one wants to find the parameters of a given kinematic skeleton that best explain a set of observed 3D joint locations.

Intrinsic Dynamic Shape Prior for Fast, Sequential and Dense Non-Rigid Structure from Motion with Detection of Temporally-Disjoint Rigidity

no code implementations5 Sep 2019 Vladislav Golyanik, André Jonas, Didier Stricker, Christian Theobalt

The reasons for the slow dissemination are the severe ill-posedness, high sensitivity to motion and deformation cues and the difficulty to obtain reliable point tracks in the vast majority of practical scenarios.

Neural Style-Preserving Visual Dubbing

no code implementations5 Sep 2019 Hyeongwoo Kim, Mohamed Elgharib, Michael Zollhöfer, Hans-Peter Seidel, Thabo Beeler, Christian Richardt, Christian Theobalt

We present a style-preserving visual dubbing approach from single video inputs, which maintains the signature style of target actors when modifying facial expressions, including mouth motions, to match foreign languages.

Generative Adversarial Network

Multi-Garment Net: Learning to Dress 3D People from Images

6 code implementations ICCV 2019 Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, Gerard Pons-Moll

We present Multi-Garment Network (MGN), a method to predict body shape and clothing, layered on top of the SMPL model from a few frames (1-8) of a video.

3D Human Pose Estimation 3D Shape Reconstruction From A Single 2D Image

Real-Time Global Illumination Decomposition of Videos

no code implementations6 Aug 2019 Abhimitra Meka, Mohammad Shafiei, Michael Zollhoefer, Christian Richardt, Christian Theobalt

We propose the first approach for the decomposition of a monocular color video into direct and indirect illumination components in real time.

DispVoxNets: Non-Rigid Point Set Alignment with Supervised Learning Proxies

no code implementations24 Jul 2019 Soshi Shimada, Vladislav Golyanik, Edgar Tretschk, Didier Stricker, Christian Theobalt

We introduce a supervised-learning framework for non-rigid point set alignment of a new kind - Displacements on Voxels Networks (DispVoxNets) - which abstracts away from the point set representation and regresses 3D displacement fields on regularly sampled proxy 3D voxel grids.

XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera

4 code implementations1 Jul 2019 Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, Christian Theobalt

The first stage is a convolutional neural network (CNN) that estimates 2D and 3D pose features along with identity assignments for all visible joints of all individuals. We contribute a new architecture for this CNN, called SelecSLS Net, that uses novel selective long and short range skip connections to improve the information flow allowing for a drastically faster network without compromising accuracy.

3D Multi-Person Human Pose Estimation 3D Multi-Person Pose Estimation +1

Text-based Editing of Talking-head Video

1 code implementation4 Jun 2019 Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B. Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, Maneesh Agrawala

To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material.

Face Model Sentence +3

EgoFace: Egocentric Face Performance Capture and Videorealistic Reenactment

no code implementations26 May 2019 Mohamed Elgharib, Mallikarjun BR, Ayush Tewari, Hyeongwoo Kim, Wentao Liu, Hans-Peter Seidel, Christian Theobalt

Our lightweight setup allows operations in uncontrolled environments, and lends itself to telepresence applications such as video-conferencing from dynamic environments.

Emergence of Implicit Filter Sparsity in Convolutional Neural Networks

no code implementations ICML Workshop Deep_Phenomen 2019 Dushyant Mehta, Kwang In Kim, Christian Theobalt

We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained using adaptive gradient descent techniques with L2 regularization or weight decay.

L2 Regularization