Search Results for author: Christian Theobalt

Found 131 papers, 24 papers with code

HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization

no code implementations ECCV 2020 Neng Qian, Jiayi Wang, Franziska Mueller, Florian Bernard, Vladislav Golyanik, Christian Theobalt

3D hand reconstruction from images is a widely-studied problem in computer vision and graphics, and has a particularly high relevance for virtual and augmented reality.

Neural Rendering

Neural Dense Non-Rigid Structure from Motion with Latent Space Constraints

no code implementations ECCV 2020 Vikramjit Sidhu, Edgar Tretschk, Vladislav Golyanik, Antonio Agudo, Christian Theobalt

We introduce the first dense neural non-rigid structure from motion (N-NRSfM) approach, which can be trained end-to-end in an unsupervised manner from 2D point tracks.

3D Shape Reconstruction Structure from Motion

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

no code implementations18 Oct 2021 Jiatao Gu, Lingjie Liu, Peng Wang, Christian Theobalt

We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue.

Image Generation

Neural Rays for Occlusion-aware Image-based Rendering

no code implementations28 Jul 2021 YuAn Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, Wenping Wang

Experiments demonstrate that NeuRay can quickly generate high-quality novel view images of unseen scenes with little finetuning and can handle complex scenes with severe self-occlusions which previous methods struggle with.

Neural Rendering Novel View Synthesis +1

StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN

no code implementations15 Jul 2021 Gereon Fox, Ayush Tewari, Mohamed Elgharib, Christian Theobalt

We demonstrate that it suffices to train our temporal architecture on only 10 minutes of footage of 1 subject for about 6 hours.

Adiabatic Quantum Graph Matching with Permutation Matrix Constraints

no code implementations8 Jul 2021 Marcel Seelbach Benkner, Vladislav Golyanik, Christian Theobalt, Michael Moeller

In this work, we address such problems with emerging quantum computing technology and propose several reformulations of QAPs as unconstrained problems suitable for efficient execution on quantum hardware.

Graph Matching

Self-supervised Outdoor Scene Relighting

no code implementations ECCV 2020 Ye Yu, Abhimitra Meka, Mohamed Elgharib, Hans-Peter Seidel, Christian Theobalt, William A. P. Smith

Outdoor scene relighting is a challenging problem that requires good understanding of the scene geometry, illumination and albedo.

NRST: Non-rigid Surface Tracking from Monocular Video

no code implementations6 Jul 2021 Marc Habermann, Weipeng Xu, Helge Rhodin, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt

Our texture term exploits the orientation information in the micro-structures of the objects, e. g., the yarn patterns of fabrics.

HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks

no code implementations2 Jul 2021 Jameel Malik, Soshi Shimada, Ahmed Elhayek, Sk Aziz Ali, Christian Theobalt, Vladislav Golyanik, Didier Stricker

To address the limitations of the existing methods, we develop HandVoxNet++, i. e., a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner.

3D Hand Pose Estimation

RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

no code implementations22 Jun 2021 Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A. Otaduy, Dan Casas, Christian Theobalt

Moreover, we demonstrate that our approach offers previously unseen two-hand tracking performance from RGB, and quantitatively and qualitatively outperforms existing RGB-based methods that were not explicitly designed for two-hand interactions.

3D Reconstruction Motion Capture +1

Fast Simultaneous Gravitational Alignment of Multiple Point Sets

no code implementations21 Jun 2021 Vladislav Golyanik, Soshi Shimada, Christian Theobalt

The problem of simultaneous rigid alignment of multiple unordered point sets which is unbiased towards any of the inputs has recently attracted increasing interest, and several reliable methods have been newly proposed.

NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

2 code implementations20 Jun 2021 Peng Wang, Lingjie Liu, YuAn Liu, Christian Theobalt, Taku Komura, Wenping Wang

In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation.

Novel View Synthesis

Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control

no code implementations3 Jun 2021 Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, Christian Theobalt

To address this problem, we utilize a coarse body model as the proxy to unwarp the surrounding 3D space into a canonical pose.

Q-Match: Iterative Shape Matching via Quantum Annealing

no code implementations ICCV 2021 Marcel Seelbach Benkner, Zorah Lähner, Vladislav Golyanik, Christof Wunderlich, Christian Theobalt, Michael Moeller

Finding shape correspondences can be formulated as an NP-hard quadratic assignment problem (QAP) that becomes infeasible for shapes with high sampling density.

Real-time Deep Dynamic Characters

no code implementations4 May 2021 Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt

We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery.

Neural Monocular 3D Human Motion Capture with Physical Awareness

no code implementations3 May 2021 Soshi Shimada, Vladislav Golyanik, Weipeng Xu, Patrick Pérez, Christian Theobalt

We present a new trainable system for physically plausible markerless 3D human motion capture, which achieves state-of-the-art results in a broad range of challenging scenarios.

3D Pose Estimation Motion Capture

Differentiable Event Stream Simulator for Non-Rigid 3D Tracking

no code implementations30 Apr 2021 Jalees Nehvi, Vladislav Golyanik, Franziska Mueller, Hans-Peter Seidel, Mohamed Elgharib, Christian Theobalt

This paper introduces the first differentiable simulator of event streams, i. e., streams of asynchronous brightness change signals recorded by event cameras.

Estimating Egocentric 3D Human Pose in Global Space

no code implementations ICCV 2021 Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, Christian Theobalt

Furthermore, these methods suffer from limited accuracy and temporal instability due to ambiguities caused by the monocular setup and the severe occlusion in a strongly distorted egocentric perspective.

3D Human Pose Estimation Motion Capture

Synthesis of Compositional Animations from Textual Descriptions

2 code implementations ICCV 2021 Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, Philipp Slusallek

Our model can generate plausible pose sequences for short sentences describing single actions as well as long compositional sentences describing multiple sequential and superimposed actions.

motion synthesis

HumanGAN: A Generative Model of Humans Images

no code implementations11 Mar 2021 Kripasindhu Sarkar, Lingjie Liu, Vladislav Golyanik, Christian Theobalt

We address these limitations and present a generative model for images of dressed humans offering control over pose, local body part appearance and garment style.

Pose Transfer

Style and Pose Control for Image Synthesis of Humans from a Single Monocular View

no code implementations22 Feb 2021 Kripasindhu Sarkar, Vladislav Golyanik, Lingjie Liu, Christian Theobalt

Photo-realistic re-rendering of a human from a single image with explicit control over body pose, shape and appearance enables a wide range of applications, such as human appearance transfer, virtual try-on, motion imitation, and novel view synthesis.

Image Generation Novel View Synthesis +1

Learning Speech-driven 3D Conversational Gestures from Video

no code implementations13 Feb 2021 Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, Christian Theobalt

We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input.

Hand Pose Estimation Motion Capture

Quantum Permutation Synchronization

no code implementations CVPR 2021 Tolga Birdal, Vladislav Golyanik, Christian Theobalt, Leonidas Guibas

We present QuantumSync, the first quantum algorithm for solving a synchronization problem in the context of computer vision.

Neural Re-Rendering of Humans from a Single Image

no code implementations ECCV 2020 Kripasindhu Sarkar, Dushyant Mehta, Weipeng Xu, Vladislav Golyanik, Christian Theobalt

Human re-rendering from a single image is a starkly under-constrained problem, and state-of-the-art algorithms often exhibit undesired artefacts, such as over-smoothing, unrealistic distortions of the body parts and garments, or implausible changes of the texture.

Translation

EgoRenderer: Rendering Human Avatars From Egocentric Camera Images

no code implementations ICCV 2021 Tao Hu, Kripasindhu Sarkar, Lingjie Liu, Matthias Zwicker, Christian Theobalt

We next combine the target pose image and the textures into a combined feature image, which is transformed into the output color image using a neural image translation network.

Texture Synthesis Translation

Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video

1 code implementation ICCV 2021 Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, Christian Theobalt

We show that a single handheld consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views, e. g. a `bullet-time' video effect.

Novel View Synthesis Video Editing

High-Fidelity Neural Human Motion Transfer from Monocular Video

1 code implementation CVPR 2021 Moritz Kappel, Vladislav Golyanik, Mohamed Elgharib, Jann-Ole Henningson, Hans-Peter Seidel, Susana Castillo, Christian Theobalt, Marcus Magnor

We address these limitations for the first time in the literature and present a new framework which performs high-fidelity and temporally-consistent human motion transfer with natural pose-dependent non-rigid deformations, for several types of loose garments.

Image Generation

EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream

no code implementations ICCV 2021 Viktor Rudnev, Vladislav Golyanik, Jiayi Wang, Hans-Peter Seidel, Franziska Mueller, Mohamed Elgharib, Christian Theobalt

Due to the different data modality of event cameras compared to classical cameras, existing methods cannot be directly applied to and re-trained for event streams.

3D Hand Pose Estimation

Monocular Real-time Full Body Capture with Inter-part Correlations

no code implementations CVPR 2021 Yuxiao Zhou, Marc Habermann, Ikhsanul Habibie, Ayush Tewari, Christian Theobalt, Feng Xu

We present the first method for real-time full body capture that estimates shape and motion of body and hands together with a dynamic 3D face model from a single color image.

Face Model

Pose-Guided Human Animation from a Single Image in the Wild

no code implementations CVPR 2021 Jae Shin Yoon, Lingjie Liu, Vladislav Golyanik, Kripasindhu Sarkar, Hyun Soo Park, Christian Theobalt

We present a new pose transfer method for synthesizing a human animation from a single image of a person controlled by a sequence of body poses.

Pose Transfer

i3DMM: Deep Implicit 3D Morphable Model of Human Heads

1 code implementation CVPR 2021 Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed Elgharib, Daniel Cremers, Christian Theobalt

Our approach has the following favorable properties: (i) It is the first full head morphable model that includes hair.

Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks

1 code implementation CVPR 2021 Xiaoxiao Long, Lingjie Liu, Wei Li, Christian Theobalt, Wenping Wang

We present a novel method for multi-view depth estimation from a single video, which is a critical task in various applications, such as perception, reconstruction and robot navigation.

Depth Estimation Robot Navigation

Deep Physics-aware Inference of Cloth Deformation for Monocular Human Performance Capture

no code implementations25 Nov 2020 Yue Li, Marc Habermann, Bernhard Thomaszewski, Stelian Coros, Thabo Beeler, Christian Theobalt

Recent monocular human performance capture approaches have shown compelling dense tracking results of the full body from a single RGB camera.

LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration

no code implementations NeurIPS 2020 Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll

Formulating this closed loop is not straightforward because it is not trivial to force the output of the NN to be on the surface of the human model - outside this surface the human model is not even defined.

Self-Supervised Learning

Learning Complete 3D Morphable Face Models from Images and Videos

no code implementations CVPR 2021 Mallikarjun B R, Ayush Tewari, Hans-Peter Seidel, Mohamed Elgharib, Christian Theobalt

Our network design and loss functions ensure a disentangled parameterization of not only identity and albedo, but also, for the first time, an expression basis.

3D Face Reconstruction Self-Supervised Learning

Fast Gravitational Approach for Rigid Point Set Registration with Ordinary Differential Equations

no code implementations28 Sep 2020 Sk Aziz Ali, Kerem Kahraman, Christian Theobalt, Didier Stricker, Vladislav Golyanik

This article introduces a new physics-based method for rigid point set alignment called Fast Gravitational Approach (FGA).

PIE: Portrait Image Embedding for Semantic Control

no code implementations20 Sep 2020 Ayush Tewari, Mohamed Elgharib, Mallikarjun B R., Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt

We present the first approach for embedding real portrait images in the latent space of StyleGAN, which allows for intuitive editing of the head pose, facial expression, and scene illumination in the image.

Face Model

Monocular Reconstruction of Neural Face Reflectance Fields

no code implementations CVPR 2021 Mallikarjun B R., Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, Christian Theobalt

The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing.

PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time

no code implementations20 Aug 2020 Soshi Shimada, Vladislav Golyanik, Weipeng Xu, Christian Theobalt

We, therefore, present PhysCap, the first algorithm for physically plausible, real-time and marker-less human 3D motion capture with a single colour camera at 25 fps.

Motion Capture

Face2Face: Real-time Face Capture and Reenactment of RGB Videos

no code implementations CVPR 2016 Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner

Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.

Neural Sparse Voxel Fields

1 code implementation NeurIPS 2020 Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, Christian Theobalt

We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering.

Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction

1 code implementation ECCV 2020 Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll

In this work, we present methodology that combines detail-rich implicit functions and parametric representations in order to reconstruct 3D models of people that remain controllable and accurate even in the presence of clothing.

3D Human Pose Estimation 3D Human Reconstruction

VideoForensicsHQ: Detecting High-quality Manipulated Face Videos

no code implementations20 May 2020 Gereon Fox, Wentao Liu, Hyeongwoo Kim, Hans-Peter Seidel, Mohamed Elgharib, Christian Theobalt

We introduce a new benchmark dataset for face video forgery detection, of unprecedented quality.

Vid2Curve: Simultaneous Camera Motion Estimation and Thin Structure Reconstruction from an RGB Video

no code implementations7 May 2020 Peng Wang, Lingjie Liu, Nenglun Chen, Hung-Kuo Chu, Christian Theobalt, Wenping Wang

We propose the first approach that simultaneously estimates camera motion and reconstructs the geometry of complex 3D thin structures in high quality from a color video captured by a handheld camera.

Motion Estimation Occlusion Handling +1

State of the Art on Neural Rendering

no code implementations8 Apr 2020 Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B. Goldman, Michael Zollhöfer

Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e. g., by the integration of differentiable rendering into network training.

Image Generation Neural Rendering +1

Occlusion-Aware Depth Estimation with Adaptive Normal Constraints

1 code implementation ECCV 2020 Xiaoxiao Long, Lingjie Liu, Christian Theobalt, Wenping Wang

We present a new learning-based method for multi-frame depth estimation from a color video, which is a fundamental problem in scene understanding, robot navigation or handheld 3D reconstruction.

3D Reconstruction Depth Estimation +2

StyleRig: Rigging StyleGAN for 3D Control over Portrait Images

no code implementations CVPR 2020 Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt

StyleGAN generates photorealistic portrait images of faces with eyes, teeth, hair and context (neck, shoulders, background), but lacks a rig-like control over semantic face parameters that are interpretable in 3D, such as face pose, expressions, and scene illumination.

Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data

1 code implementation CVPR 2020 Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, Feng Xu

We present a novel method for monocular hand shape and pose estimation at unprecedented runtime performance of 100fps and at state-of-the-art accuracy.

Motion Capture Pose Estimation

DeepCap: Monocular Human Performance Capture Using Weak Supervision

no code implementations CVPR 2020 Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt

Human performance capture is a highly important computer vision problem with many applications in movie production and virtual/augmented reality.

Pose Estimation

Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation

no code implementations14 Jan 2020 Lingjie Liu, Weipeng Xu, Marc Habermann, Michael Zollhoefer, Florian Bernard, Hyeongwoo Kim, Wenping Wang, Christian Theobalt

In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space.

Image-to-Image Translation Novel View Synthesis +1

Image-guided Neural Object Rendering

no code implementations ICLR 2020 Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.

Image Generation

A Quantum Computational Approach to Correspondence Problems on Point Sets

no code implementations CVPR 2020 Vladislav Golyanik, Christian Theobalt

Modern adiabatic quantum computers (AQC) are already used to solve difficult combinatorial optimisation problems in various domains of science.

Neural Voice Puppetry: Audio-driven Facial Reenactment

no code implementations ECCV 2020 Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, Matthias Nießner

Neural Voice Puppetry has a variety of use-cases, including audio-driven video avatars, video dubbing, and text-driven video synthesis of a talking head.

Face Model Neural Rendering

DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data

1 code implementation9 Dec 2019 Aljaž Božič, Michael Zollhöfer, Christian Theobalt, Matthias Nießner

Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus.

3D Reconstruction

Convex Optimisation for Inverse Kinematics

no code implementations24 Oct 2019 Tarun Yenamandra, Florian Bernard, Jiayi Wang, Franziska Mueller, Christian Theobalt

We consider the problem of inverse kinematics (IK), where one wants to find the parameters of a given kinematic skeleton that best explain a set of observed 3D joint locations.

Neural Style-Preserving Visual Dubbing

no code implementations5 Sep 2019 Hyeongwoo Kim, Mohamed Elgharib, Michael Zollhöfer, Hans-Peter Seidel, Thabo Beeler, Christian Richardt, Christian Theobalt

We present a style-preserving visual dubbing approach from single video inputs, which maintains the signature style of target actors when modifying facial expressions, including mouth motions, to match foreign languages.

Intrinsic Dynamic Shape Prior for Fast, Sequential and Dense Non-Rigid Structure from Motion with Detection of Temporally-Disjoint Rigidity

no code implementations5 Sep 2019 Vladislav Golyanik, André Jonas, Didier Stricker, Christian Theobalt

The reasons for the slow dissemination are the severe ill-posedness, high sensitivity to motion and deformation cues and the difficulty to obtain reliable point tracks in the vast majority of practical scenarios.

Structure from Motion

Multi-Garment Net: Learning to Dress 3D People from Images

6 code implementations ICCV 2019 Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, Gerard Pons-Moll

We present Multi-Garment Network (MGN), a method to predict body shape and clothing, layered on top of the SMPL model from a few frames (1-8) of a video.

3D Human Pose Estimation 3D Shape Reconstruction From A Single 2D Image

Real-Time Global Illumination Decomposition of Videos

no code implementations6 Aug 2019 Abhimitra Meka, Mohammad Shafiei, Michael Zollhoefer, Christian Richardt, Christian Theobalt

We propose the first approach for the decomposition of a monocular color video into direct and indirect illumination components in real time.

DispVoxNets: Non-Rigid Point Set Alignment with Supervised Learning Proxies

no code implementations24 Jul 2019 Soshi Shimada, Vladislav Golyanik, Edgar Tretschk, Didier Stricker, Christian Theobalt

We introduce a supervised-learning framework for non-rigid point set alignment of a new kind - Displacements on Voxels Networks (DispVoxNets) - which abstracts away from the point set representation and regresses 3D displacement fields on regularly sampled proxy 3D voxel grids.

XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera

4 code implementations1 Jul 2019 Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, Christian Theobalt

The first stage is a convolutional neural network (CNN) that estimates 2D and 3D pose features along with identity assignments for all visible joints of all individuals. We contribute a new architecture for this CNN, called SelecSLS Net, that uses novel selective long and short range skip connections to improve the information flow allowing for a drastically faster network without compromising accuracy.

Monocular 3D Human Pose Estimation Motion Capture

Text-based Editing of Talking-head Video

no code implementations4 Jun 2019 Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B. Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, Maneesh Agrawala

To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material.

Face Model Translation +1

EgoFace: Egocentric Face Performance Capture and Videorealistic Reenactment

no code implementations26 May 2019 Mohamed Elgharib, Mallikarjun BR, Ayush Tewari, Hyeongwoo Kim, Wentao Liu, Hans-Peter Seidel, Christian Theobalt

Our lightweight setup allows operations in uncontrolled environments, and lends itself to telepresence applications such as video-conferencing from dynamic environments.

Implicit Filter Sparsification In Convolutional Neural Networks

no code implementations13 May 2019 Dushyant Mehta, Kwang In Kim, Christian Theobalt

We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained with adaptive gradient descent techniques and L2 regularization or weight decay.

L2 Regularization

IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D Reconstruction

no code implementations27 Apr 2019 Soshi Shimada, Vladislav Golyanik, Christian Theobalt, Didier Stricker

The majority of the existing methods for non-rigid 3D surface regression from monocular 2D images require an object template or point tracks over multiple frames as an input, and are still far from real-time processing rates.

3D Reconstruction

Tex2Shape: Detailed Full Human Body Geometry From a Single Image

1 code implementation ICCV 2019 Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, Marcus Magnor

From a partial texture, we estimate detailed normal and vector displacement maps, which can be applied to a low-resolution smooth body model to add detail and clothing.

Image-to-Image Translation Translation

Learning to Reconstruct People in Clothing from a Single RGB Camera

1 code implementation CVPR 2019 Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, Gerard Pons-Moll

We present a learning-based model to infer the personalized 3D shape of people from a few frames (1-8) of a monocular video in which the person is moving, in less than 10 seconds with a reconstruction accuracy of 5mm.

Learning a Disentangled Embedding for Monocular 3D Shape Retrieval and Pose Estimation

1 code implementation24 Dec 2018 Kyaw Zaw Lin, Weipeng Xu, Qianru Sun, Christian Theobalt, Tat-Seng Chua

We propose a novel approach to jointly perform 3D shape retrieval and pose estimation from monocular images. In order to make the method robust to real-world image variations, e. g. complex textures and backgrounds, we learn an embedding space from 3D data that only includes the relevant information, namely the shape and pose.

3D Object Retrieval 3D Shape Classification +2

FML: Face Model Learning from Videos

no code implementations CVPR 2019 Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj, Mohamed Elgharib, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt

In contrast, we propose multi-frame video-based self-supervised training of a deep network that (i) learns a face identity model both in shape and appearance while (ii) jointly learning to reconstruct 3D faces.

3D Reconstruction Face Model

On Implicit Filter Level Sparsity in Convolutional Neural Networks

no code implementations CVPR 2019 Dushyant Mehta, Kwang In Kim, Christian Theobalt

We investigate filter level sparsity that emerges in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained with adaptive gradient descent techniques and L2 regularization or weight decay.

L2 Regularization

Higher-order Projected Power Iterations for Scalable Multi-Matching

no code implementations26 Nov 2018 Florian Bernard, Johan Thunberg, Paul Swoboda, Christian Theobalt

The matching of multiple objects (e. g. shapes or images) is a fundamental problem in vision and graphics.

IGNOR: Image-guided Neural Object Rendering

no code implementations26 Nov 2018 Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.

Image Generation Novel View Synthesis

Neural Rendering and Reenactment of Human Actor Videos

no code implementations11 Sep 2018 Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Marc Habermann, Wenping Wang, Christian Theobalt

In contrast to conventional human character rendering, we do not require the availability of a production-quality photo-realistic 3D model of the human, but instead rely on a video sequence in conjunction with a (medium-quality) controllable 3D template model of the person.

Image Generation Neural Rendering

Detailed Human Avatars from Monocular Video

1 code implementation3 Aug 2018 Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, Gerard Pons-Moll

We present a novel method for high detail-preserving human avatar creation from monocular video.

Deep Video Portraits

no code implementations29 May 2018 Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, Christian Theobalt

In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network -- thus taking full control of the target.

Face Model

HeadOn: Real-time Reenactment of Human Portrait Videos

no code implementations29 May 2018 Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze.

A Hybrid Model for Identity Obfuscation by Face Replacement

no code implementations ECCV 2018 Qianru Sun, Ayush Tewari, Weipeng Xu, Mario Fritz, Christian Theobalt, Bernt Schiele

As more and more personal photos are shared and tagged in social media, avoiding privacy risks such as unintended recognition becomes increasingly challenging.

Face Generation

Synchronisation of Partial Multi-Matchings via Non-negative Factorisations

no code implementations16 Mar 2018 Florian Bernard, Johan Thunberg, Jorge Goncalves, Christian Theobalt

In order to deal with the inherent non-convexity of the permutation synchronisation problem, we use an initialisation procedure based on a novel rotation scheme applied to the solution of the spectral relaxation.

Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera

no code implementations15 Mar 2018 Weipeng Xu, Avishek Chatterjee, Michael Zollhoefer, Helge Rhodin, Pascal Fua, Hans-Peter Seidel, Christian Theobalt

We tackle these challenges based on a novel lightweight setup that converts a standard baseball cap to a device for high-quality pose estimation based on a single cap-mounted fisheye camera.

3D Pose Estimation Motion Capture

Video Based Reconstruction of 3D People Models

1 code implementation CVPR 2018 Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, Gerard Pons-Moll

This paper describes how to obtain accurate 3D body models and texture of arbitrary people from a single, monocular video in which a person is moving.

3D Reconstruction Virtual Try-on

LIME: Live Intrinsic Material Estimation

no code implementations CVPR 2018 Abhimitra Meka, Maxim Maximov, Michael Zollhoefer, Avishek Chatterjee, Hans-Peter Seidel, Christian Richardt, Christian Theobalt

We present the first end to end approach for real time material estimation for general object shapes with uniform material that only requires a single color image as input.

Image-to-Image Translation Mixed Reality +1

Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB

6 code implementations9 Dec 2017 Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, Christian Theobalt

Our approach uses novel occlusion-robust pose-maps (ORPM) which enable full body pose inference even under strong partial occlusions by other people and objects in the scene.

3D Pose Estimation

Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz

no code implementations CVPR 2018 Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, Christian Theobalt

To alleviate this problem, we present the first approach that jointly learns 1) a regressor for face shape, expression, reflectance and illumination on the basis of 2) a concurrently learned parametric face model.

Face Model

MonoPerfCap: Human Performance Capture from Monocular Video

no code implementations7 Aug 2017 Weipeng Xu, Avishek Chatterjee, Michael Zollhöfer, Helge Rhodin, Dushyant Mehta, Hans-Peter Seidel, Christian Theobalt

Reconstruction from monocular video alone is drastically more challenging, since strong occlusions and the inherent depth ambiguity lead to a highly ill-posed reconstruction problem.

Pose Estimation Video Editing

Criteria Sliders: Learning Continuous Database Criteria via Interactive Ranking

no code implementations12 Jun 2017 James Tompkin, Kwang In Kim, Hanspeter Pfister, Christian Theobalt

Large databases are often organized by hand-labeled metadata, or criteria, which are expensive to collect.

VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera

1 code implementation3 May 2017 Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, Christian Theobalt

A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton.

3D Human Pose Estimation

InverseFaceNet: Deep Monocular Inverse Face Rendering

no code implementations CVPR 2018 Hyeongwoo Kim, Michael Zollhöfer, Ayush Tewari, Justus Thies, Christian Richardt, Christian Theobalt

In contrast, we propose to recover high-quality facial pose, shape, expression, reflectance and illumination using a deep neural network that is trained using a large, synthetically created training corpus.

Face Reconstruction

MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction

no code implementations ICCV 2017 Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Pérez, Christian Theobalt

In this work we propose a novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image.

Face Reconstruction

EgoCap: Egocentric Marker-less Motion Capture with Two Fisheye Cameras (Extended Abstract)

no code implementations31 Dec 2016 Helge Rhodin, Christian Richardt, Dan Casas, Eldar Insafutdinov, Mohammad Shafiei, Hans-Peter Seidel, Bernt Schiele, Christian Theobalt

Marker-based and marker-less optical skeletal motion-capture methods use an outside-in arrangement of cameras placed around a scene, with viewpoints converging on the center.

Motion Capture Pose Estimation +1

Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision

no code implementations29 Nov 2016 Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, Christian Theobalt

We propose a CNN-based approach for 3D human body pose estimation from single RGB images that addresses the issue of limited generalizability of models trained solely on the starkly limited publicly available 3D pose data.

Monocular 3D Human Pose Estimation Motion Capture +1

Real-time Halfway Domain Reconstruction of Motion and Geometry

no code implementations23 Oct 2016 Lucas Thies, Michael Zollhöfer, Christian Richardt, Christian Theobalt, Günther Greiner

Our extensive experiments and evaluations show that our approach produces high-quality dense reconstructions of 3D geometry and scene flow at real-time frame rates, and compares favorably to the state of the art.

Model-based Outdoor Performance Capture

no code implementations21 Oct 2016 Nadia Robertini, Dan Casas, Helge Rhodin, Hans-Peter Seidel, Christian Theobalt

We propose a new model-based method to accurately reconstruct human performances captured outdoors in a multi-camera setup.

Edge Detection

Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input

no code implementations16 Oct 2016 Srinath Sridhar, Franziska Mueller, Michael Zollhöfer, Dan Casas, Antti Oulasvirta, Christian Theobalt

However, due to difficult occlusions, fast motions, and uniform hand appearance, jointly tracking hand and object pose is more challenging than tracking either of the two separately.

Object Tracking

Video Depth-From-Defocus

no code implementations12 Oct 2016 Hyeongwoo Kim, Christian Richardt, Christian Theobalt

Many compelling video post-processing effects, in particular aesthetic focus editing and refocusing effects, are feasible if per-frame depth information is available.

FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality

no code implementations11 Oct 2016 Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner

Based on reenactment of a prerecorded stereo video of the person without the HMD, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances.

Eye Tracking Motion Capture +1

EgoCap: Egocentric Marker-less Motion Capture with Two Fisheye Cameras

no code implementations23 Sep 2016 Helge Rhodin, Christian Richardt, Dan Casas, Eldar Insafutdinov, Mohammad Shafiei, Hans-Peter Seidel, Bernt Schiele, Christian Theobalt

We therefore propose a new method for real-time, marker-less and egocentric motion capture which estimates the full-body skeleton pose from a lightweight stereo pair of fisheye cameras that are attached to a helmet or virtual reality headset.

Motion Capture Pose Estimation +1

Dense Wide-Baseline Scene Flow From Two Handheld Video Cameras

no code implementations16 Sep 2016 Christian Richardt, Hyeongwoo Kim, Levi Valgaerts, Christian Theobalt

We finally refine the computed correspondence fields in a variational scene flow formulation.

Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging

no code implementations22 Apr 2016 Zachary DeVito, Michael Mara, Michael Zollhöfer, Gilbert Bernstein, Jonathan Ragan-Kelley, Christian Theobalt, Pat Hanrahan, Matthew Fisher, Matthias Nießner

Many graphics and vision problems can be expressed as non-linear least squares optimizations of objective functions over visual data, such as images and meshes.

BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration

no code implementations5 Apr 2016 Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Christian Theobalt

Our approach estimates globally optimized (i. e., bundle adjusted) poses in real-time, supports robust tracking with recovery from gross tracking failures (i. e., relocalization), and re-estimates the 3D model in real-time to ensure global consistency; all within a single framework.

3D Reconstruction Mixed Reality +1

VolumeDeform: Real-time Volumetric Non-rigid Reconstruction

no code implementations27 Mar 2016 Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, Marc Stamminger

We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints.

Context-guided diffusion for label propagation on graphs

no code implementations ICCV 2015 Kwang In Kim, James Tompkin, Hanspeter Pfister, Christian Theobalt

Existing approaches for diffusion on graphs, e. g., for label propagation, are mainly focused on isotropic diffusion, which is induced by the commonly-used graph Laplacian regularizer.

Fast and Robust Hand Tracking Using Detection-Guided Optimization

no code implementations CVPR 2015 Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, Christian Theobalt

In the optimization step, a novel objective function combines the detected part labels and a Gaussian mixture representation of the depth to estimate a pose that best fits the depth.

Pose Estimation

Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model

no code implementations11 Feb 2016 Srinath Sridhar, Helge Rhodin, Hans-Peter Seidel, Antti Oulasvirta, Christian Theobalt

In this paper, we propose a new approach that tracks the full skeleton motion of the hand from multiple RGB cameras in real-time.

Semi-supervised Learning with Explicit Relationship Regularization

no code implementations CVPR 2015 Kwang In Kim, James Tompkin, Hanspeter Pfister, Christian Theobalt

In many learning tasks, the structure of the target space of a function holds rich information about the relationships between evaluations of functions on different data points.

Dimensionality Reduction General Classification

A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation

no code implementations ICCV 2015 Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, Christian Theobalt

Generative reconstruction methods compute the 3D configuration (such as pose and/or geometry) of a shape by optimizing the overlap of the projected 3D shape model with images.

Motion Capture Occlusion Handling +1

Local High-order Regularization on Data Manifolds

no code implementations CVPR 2015 Kwang In Kim, James Tompkin, Hanspeter Pfister, Christian Theobalt

The iterated graph Laplacian enables high-order regularization, but it has a high computational complexity and so cannot be applied to large problems.

Dimensionality Reduction

Automatic Face Reenactment

no code implementations CVPR 2014 Pablo Garrido, Levi Valgaerts, Ole Rehmsen, Thorsten Thormaehlen, Patrick Perez, Christian Theobalt

We propose an image-based, facial reenactment system that replaces the face of an actor in an existing target video with the face of a user from a source video, while preserving the original target performance.

Face Model Face Reenactment +2

Efficient Multi-view Performance Capture of Fine-Scale Surface Detail

no code implementations5 Feb 2016 Nadia Robertini, Edilson de Aguiar, Thomas Helten, Christian Theobalt

We present a new effective way for performance capture of deforming meshes with fine-scale time-varying surface detail from multi-view video.

Occlusion Handling

Building Statistical Shape Spaces for 3D Human Modeling

no code implementations19 Mar 2015 Leonid Pishchulin, Stefanie Wuhrer, Thomas Helten, Christian Theobalt, Bernt Schiele

Statistical models of 3D human shape and pose learned from scan databases have developed into valuable tools to solve a variety of vision and graphics problems.

Cannot find the paper you are looking for? You can Submit a new open access paper.