Search Results for author: Matthias Nießner

Found 107 papers, 55 papers with code

Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors

no code implementations28 Feb 2023 Ji Hou, Xiaoliang Dai, Zijian He, Angela Dai, Matthias Nießner

Current popular backbones in computer vision, such as Vision Transformers (ViT) and ResNets are trained to perceive the world from 2D images.

Contrastive Learning Instance Segmentation +5

S4R: Self-Supervised Semantic Scene Reconstruction from RGB-D Scans

no code implementations7 Feb 2023 Junwen Huang, Alexey Artemov, Yujin Chen, Shuaifeng Zhi, Kai Xu, Matthias Nießner

To this end, we design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images, fusing cross-domain features into volumetric embeddings to predict complete 3D geometry, color, and semantics.

Colorization Semantic Segmentation

Panoptic Lifting for 3D Scene Understanding with Neural Fields

no code implementations19 Dec 2022 Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthias Nießner, Angela Dai, Peter Kontschieder

We propose Panoptic Lifting, a novel approach for learning panoptic 3D volumetric representations from images of in-the-wild scenes.

Panoptic Segmentation Scene Understanding

SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images

1 code implementation18 Dec 2022 Abdullah Hamdi, Bernard Ghanem, Matthias Nießner

SuRFNet employs partial SRFs from few/one images and a specialized SRF loss to learn to generate high-quality sparse voxel radiance fields that can be rendered from novel views.

Novel View Synthesis

Learning Neural Parametric Head Models

no code implementations6 Dec 2022 Simon Giebenhain, Tobias Kirschstein, Markos Georgopoulos, Martin Rünz, Lourdes Agapito, Matthias Nießner

We propose a novel 3D morphable model for complete human heads based on hybrid neural fields.

ObjectMatch: Robust Registration using Canonical Object Correspondences

no code implementations5 Dec 2022 Can Gümeli, Angela Dai, Matthias Nießner

In a pairwise setting, our method improves registration recall of state-of-the-art feature matching from 77% to 87% overall and from 21% to 52% in pairs with 10% or less inter-frame overlap.

Pose Estimation

DiffRF: Rendering-Guided 3D Radiance Field Diffusion

no code implementations2 Dec 2022 Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Matthias Nießner

We introduce DiffRF, a novel approach for 3D radiance field synthesis based on denoising diffusion probabilistic models.


ClipFace: Text-guided Editing of Textured 3D Morphable Models

no code implementations2 Dec 2022 Shivangi Aneja, Justus Thies, Angela Dai, Matthias Nießner

Controllable editing and manipulation are given by language prompts to adapt texture and expression of the 3D morphable model.

Texture Synthesis

UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding

no code implementations1 Dec 2022 Dave Zhenyu Chen, Ronghang Hu, Xinlei Chen, Matthias Nießner, Angel X. Chang

Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships.

3D dense captioning Dense Captioning +1

Learning 3D Scene Priors with 2D Supervision

no code implementations25 Nov 2022 Yinyu Nie, Angela Dai, Xiaoguang Han, Matthias Nießner

Holistic 3D scene understanding entails estimation of both layout configuration and object geometry in a 3D environment.

Scene Understanding

Neural Poisson: Indicator Functions for Neural Fields

no code implementations25 Nov 2022 Angela Dai, Matthias Nießner

Implicit neural field generating signed distance field representations (SDFs) of 3D shapes have shown remarkable progress in 3D shape reconstruction and generation.

3D Shape Reconstruction Surface Reconstruction

How to Boost Face Recognition with StyleGAN?

1 code implementation18 Oct 2022 Artem Sevastopolsky, Yury Malkov, Nikita Durasov, Luisa Verdoliva, Matthias Nießner

Our self-supervised strategy is the most useful with limited amounts of labeled training data, which can be beneficial for more tailored face recognition tasks and when facing privacy concerns.

Face Recognition

Neural Shape Deformation Priors

no code implementations11 Oct 2022 Jiapeng Tang, Lev Markhasin, Bi Wang, Justus Thies, Matthias Nießner

To this end, we introduce transformer-based deformation networks that represent a shape deformation as a composition of local surface deformations.

3D Multi-Object Tracking with Differentiable Pose Estimation

no code implementations28 Jun 2022 Dominik Schmauser, Zeju Qiu, Norman Müller, Matthias Nießner

We propose a novel approach for joint 3D multi-object tracking and reconstruction from RGB-D sequences in indoor environments.

3D Multi-Object Tracking Pose Estimation

Fast Dynamic Radiance Fields with Time-Aware Neural Voxels

1 code implementation30 May 2022 Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, Qi Tian

A multi-distance interpolation method is proposed and applied on voxel features to model both small and large motions.

3DILG: Irregular Latent Grids for 3D Generative Modeling

1 code implementation27 May 2022 Biao Zhang, Matthias Nießner, Peter Wonka

All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.

3D Shape Modeling

End2End Multi-View Feature Matching using Differentiable Pose Optimization

no code implementations3 May 2022 Barbara Roessle, Matthias Nießner

Training feature matching with gradients from pose optimization naturally learns to down-weight outliers and boosts pose estimation on image pairs compared to SuperGlue by 6. 7% on ScanNet.

Graph Attention Pose Estimation

AutoRF: Learning 3D Object Radiance Fields from Single View Observations

no code implementations CVPR 2022 Norman Müller, Andrea Simonelli, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, Peter Kontschieder

We introduce AutoRF - a new approach for learning neural 3D object representations where each object in the training set is observed by only a single view.

Novel View Synthesis

Audio-Visual Person-of-Interest DeepFake Detection

1 code implementation6 Apr 2022 Davide Cozzolino, Matthias Nießner, Luisa Verdoliva

The aim of this work is to propose a deepfake detector that can cope with the wide variety of manipulation methods and scenarios encountered in the real world.

Contrastive Learning DeepFake Detection +1

Texturify: Generating Textures on 3D Shape Surfaces

no code implementations5 Apr 2022 Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, Angela Dai

Texture cues on 3D objects are key to compelling visual representations, with the possibility to create high visual fidelity with inherent spatial consistency across different views.

3D Equivariant Graph Implicit Functions

no code implementations31 Mar 2022 Yunlu Chen, Basura Fernando, Hakan Bilen, Matthias Nießner, Efstratios Gavves

In this work, we address two key limitations of such representations, in failing to capture local 3D geometric fine details, and to learn from and generalize to shapes with unseen 3D transformations.

RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering

1 code implementation8 Mar 2022 Di Chang, Aljaž Božič, Tong Zhang, Qingsong Yan, Yingcong Chen, Sabine Süsstrunk, Matthias Nießner

Finding accurate correspondences among different views is the Achilles' heel of unsupervised Multi-View Stereo (MVS).

Neural Rendering

4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding

no code implementations6 Dec 2021 Yujin Chen, Matthias Nießner, Angela Dai

We present a new approach to instill 4D dynamic object priors into learned 3D representations by unsupervised pre-training.

Ranked #14 on 3D Instance Segmentation on ScanNet(v2) (mAP @ 50 metric)

3D Instance Segmentation 3D Semantic Segmentation +7

Neural Head Avatars from Monocular RGB Videos

no code implementations CVPR 2022 Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, Justus Thies

We present Neural Head Avatars, a novel neural representation that explicitly models the surface geometry and appearance of an animatable human avatar that can be used for teleconferencing in AR/VR or other applications in the movie or games industry that rely on a digital human.

Novel View Synthesis

Pose2Room: Understanding 3D Scenes from Human Activities

no code implementations1 Dec 2021 Yinyu Nie, Angela Dai, Xiaoguang Han, Matthias Nießner

To this end, we propose P2R-Net to learn a probabilistic 3D model of the objects in a scene characterized by their class categories and oriented 3D bounding boxes, based on an input observed human trajectory in the environment.

Panoptic 3D Scene Reconstruction From a Single RGB Image

1 code implementation NeurIPS 2021 Manuel Dahnert, Ji Hou, Matthias Nießner, Angela Dai

Inspired by 2D panoptic segmentation, we propose to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction - from a single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations.

3D Instance Segmentation 3D Scene Reconstruction +4

4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface

1 code implementation ICCV 2021 Yang Li, Hikari Takehara, Takafumi Taketomi, Bo Zheng, Matthias Nießner

Tracking non-rigidly deforming scenes using range sensors has numerous applications including computer vision, AR/VR, and robotics.

Motion Estimation

Pri3D: Can 3D Priors Help 2D Representation Learning?

1 code implementation ICCV 2021 Ji Hou, Saining Xie, Benjamin Graham, Angela Dai, Matthias Nießner

Inspired by these advances in geometric understanding, we aim to imbue image-based perception with representations learned under geometric constraints.

Contrastive Learning Instance Segmentation +4

Neural RGB-D Surface Reconstruction

2 code implementations CVPR 2022 Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, Justus Thies

Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR.

Image Generation Mixed Reality +2

Dynamic Surface Function Networks for Clothed Human Bodies

1 code implementation ICCV 2021 Andrei Burov, Matthias Nießner, Justus Thies

To this end, we explicitly model the surface of the person using a multi-layer perceptron (MLP) which is embedded into the canonical space of the SMPL body model.

NPMs: Neural Parametric Models for 3D Deformable Shapes

1 code implementation ICCV 2021 Pablo Palafox, Aljaž Božič, Justus Thies, Matthias Nießner, Angela Dai

Crucially, once learned, our neural parametric models of shape and pose enable optimization over the learned spaces to fit to new observations, similar to the fitting of a traditional parametric model, e. g., SMPL.

Pose Transfer

RetrievalFuse: Neural 3D Scene Reconstruction with a Database

1 code implementation ICCV 2021 Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, Angela Dai

3D reconstruction of large scenes is a challenging problem due to the high-complexity nature of the solution space, in particular for generative neural networks.

3D Reconstruction 3D Scene Reconstruction +3

SceneFormer: Indoor Scene Generation with Transformers

2 code implementations17 Dec 2020 Xinpeng Wang, Chandan Yeshwanth, Matthias Nießner

In contrast, we do not use any appearance information, and implicitly learn object relations using the self-attention mechanism of transformers.

Scene Generation

Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts

1 code implementation CVPR 2021 Ji Hou, Benjamin Graham, Matthias Nießner, Saining Xie

The rapid progress in 3D scene understanding has come with growing demand for data; however, collecting and annotating 3D scenes (e. g. point clouds) are notoriously hard.

3D Semantic Segmentation Instance Segmentation +1

Seeing Behind Objects for 3D Multi-Object Tracking in RGB-D Sequences

no code implementations CVPR 2021 Norman Müller, Yu-Shiang Wong, Niloy J. Mitra, Angela Dai, Matthias Nießner

From a sequence of RGB-D frames, we detect objects in each frame and learn to predict their complete object geometry as well as a dense correspondence mapping into a canonical space.

3D Multi-Object Tracking

Vid2CAD: CAD Model Alignment using Multi-View Constraints from Videos

1 code implementation8 Dec 2020 Kevis-Kokitsi Maninis, Stefan Popov, Matthias Nießner, Vittorio Ferrari

We address the task of aligning CAD models to a video sequence of a complex scene containing multiple objects.

ID-Reveal: Identity-aware DeepFake Video Detection

1 code implementation ICCV 2021 Davide Cozzolino, Andreas Rössler, Justus Thies, Matthias Nießner, Luisa Verdoliva

A major challenge in DeepFake forgery detection is that state-of-the-art algorithms are mostly trained to detect a specific fake method.

Face Swapping Metric Learning

Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction

1 code implementation CVPR 2021 Aljaž Božič, Pablo Palafox, Michael Zollhöfer, Justus Thies, Angela Dai, Matthias Nießner

We introduce Neural Deformation Graphs for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

3D Reconstruction

RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction

1 code implementation CVPR 2021 Yinyu Nie, Ji Hou, Xiaoguang Han, Matthias Nießner

In this work, we introduce RfD-Net that jointly detects and reconstructs dense object surfaces directly from raw point clouds.

object-detection Object Detection +3

Face2Face: Real-time Face Capture and Reenactment of RGB Videos

no code implementations CVPR 2016 Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner

Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.

Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition

no code implementations29 Jun 2020 Hassan Abu Alhaija, Siva Karthik Mustikovela, Justus Thies, Varun Jampani, Matthias Nießner, Andreas Geiger, Carsten Rother

Neural rendering techniques promise efficient photo-realistic image synthesis while at the same time providing rich control over scene parameters by learning the physical image formation process.

Image-to-Image Translation Intrinsic Image Decomposition +1

SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans

1 code implementation CVPR 2021 Angela Dai, Yawar Siddiqui, Justus Thies, Julien Valentin, Matthias Nießner

We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion.

3D Reconstruction Scene Generation

Neural Non-Rigid Tracking

1 code implementation NeurIPS 2020 Aljaž Božič, Pablo Palafox, Michael Zollhöfer, Angela Dai, Justus Thies, Matthias Nießner

We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction by a learned robust optimization.

Generalized Zero and Few-Shot Transfer for Facial Forgery Detection

no code implementations21 Jun 2020 Shivangi Aneja, Matthias Nießner

We propose Deep Distribution Transfer(DDT), a new transfer learning approach to address the problem of zero and few-shot transfer in the context of facial forgery detection.

Domain Adaptation Few-Shot Learning +1

State of the Art on Neural Rendering

no code implementations8 Apr 2020 Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B. Goldman, Michael Zollhöfer

Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e. g., by the integration of differentiable rendering into network training.

BIG-bench Machine Learning Image Generation +2

3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation

1 code implementation30 Mar 2020 Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner

We show that grouping proposals improves over NMS and outperforms previous state-of-the-art methods on the tasks of 3D object detection and semantic instance segmentation on the ScanNetV2 benchmark and the S3DIS dataset.

3D Instance Segmentation 3D Object Detection +2

Learning to Optimize Non-Rigid Tracking

no code implementations CVPR 2020 Yang Li, Aljaž Božič, Tianwei Zhang, Yanli Ji, Tatsuya Harada, Matthias Nießner

One of the widespread solutions for non-rigid tracking has a nested-loop structure: with Gauss-Newton to minimize a tracking objective in the outer loop, and Preconditioned Conjugate Gradient (PCG) to solve a sparse linear system in the inner loop.

SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans

no code implementations ECCV 2020 Armen Avetisyan, Tatiana Khanova, Christopher Choy, Denver Dash, Angela Dai, Matthias Nießner

We present a novel approach to reconstructing lightweight, CAD-based representations of scanned 3D environments from commodity RGB-D sensors.

Modeling 3D Shapes by Reinforcement Learning

2 code implementations ECCV 2020 Cheng Lin, Tingxiang Fan, Wenping Wang, Matthias Nießner

We explore how to enable machines to model 3D shapes like human modelers using deep reinforcement learning (RL).

Imitation Learning reinforcement-learning +1

Local Implicit Grid Representations for 3D Scenes

1 code implementation19 Mar 2020 Chiyu Max Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser

Then, we use the decoder as a component in a shape optimization that solves for a set of latent codes on a regular grid of overlapping crops such that an interpolation of the decoded local shapes matches a partial or noisy observation.

3D Shape Representation Surface Reconstruction

Adversarial Texture Optimization from RGB-D Scans

1 code implementation CVPR 2020 Jingwei Huang, Justus Thies, Angela Dai, Abhijit Kundu, Chiyu Max Jiang, Leonidas Guibas, Matthias Nießner, Thomas Funkhouser

In this work, we present a novel approach for color texture generation using a conditional adversarial loss obtained from weakly-supervised views.

Surface Reconstruction Texture Synthesis

Image-guided Neural Object Rendering

no code implementations ICLR 2020 Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.

Image Generation

Neural Voice Puppetry: Audio-driven Facial Reenactment

1 code implementation ECCV 2020 Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, Matthias Nießner

Neural Voice Puppetry has a variety of use-cases, including audio-driven video avatars, video dubbing, and text-driven video synthesis of a talking head.

Face Model Neural Rendering +2

DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data

1 code implementation9 Dec 2019 Aljaž Božič, Michael Zollhöfer, Christian Theobalt, Matthias Nießner

Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus.

3D Reconstruction RGB-D Reconstruction

SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans

2 code implementations CVPR 2020 Angela Dai, Christian Diller, Matthias Nießner

We present a novel approach that converts partial and noisy RGB-D scans into high-quality 3D scene reconstructions by inferring unobserved scene geometry.

3D Reconstruction

SpoC: Spoofing Camera Fingerprints

no code implementations27 Nov 2019 Davide Cozzolino, Justus Thies, Andreas Rössler, Matthias Nießner, Luisa Verdoliva

Given a GAN-generated image, we insert the traces of a specific camera model into it and deceive state-of-the-art detectors into believing the image was acquired by that model.

Demosaicking Misinformation

ViewAL: Active Learning with Viewpoint Entropy for Semantic Segmentation

1 code implementation CVPR 2020 Yawar Siddiqui, Julien Valentin, Matthias Nießner

To incorporate this uncertainty measure, we introduce a new viewpoint entropy formulation, which is the basis of our active learning strategy.

Active Learning Semantic Segmentation +1

Joint Embedding of 3D Scan and CAD Objects

1 code implementation ICCV 2019 Manuel Dahnert, Angela Dai, Leonidas Guibas, Matthias Nießner

We propose a novel approach to learn a joint embedding space between scan and CAD geometry, where semantically similar objects from both domains lie close together.


RIO: 3D Object Instance Re-Localization in Changing Indoor Environments

1 code implementation ICCV 2019 Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, Matthias Nießner

In this work, we introduce the task of 3D object instance re-localization (RIO): given one or multiple objects in an RGB-D scan, we want to estimate their corresponding 6DoF poses in another 3D scan of the same environment taken at a later point in time.

Scene Understanding

End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans

1 code implementation ICCV 2019 Armen Avetisyan, Angela Dai, Matthias Nießner

We present a novel, end-to-end approach to align CAD models to an 3D scan of a scene, enabling transformation of a noisy, incomplete 3D scan to a compact, CAD reconstruction with clean, complete object geometry.


Deferred Neural Rendering: Image Synthesis using Neural Textures

3 code implementations28 Apr 2019 Justus Thies, Michael Zollhöfer, Matthias Nießner

Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline.

Image Generation Neural Rendering +1

RevealNet: Seeing Behind Objects in RGB-D Scans

no code implementations CVPR 2020 Ji Hou, Angela Dai, Matthias Nießner

Thus, we introduce the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances and infer their complete object geometry.

3D Reconstruction 3D Semantic Instance Segmentation +1

Inverse Path Tracing for Joint Material and Lighting Estimation

no code implementations17 Mar 2019 Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, Matthias Nießner

We introduce Inverse Path Tracing, a novel approach to jointly estimate the material properties of objects and light sources in indoor scenes by using an invertible light transport simulation.

Lighting Estimation Retrieval

Homogeneous Linear Inequality Constraints for Neural Network Activations

1 code implementation5 Feb 2019 Thomas Frerix, Matthias Nießner, Daniel Cremers

One way to achieve this task is by means of a projection step at test time after unconstrained training.

FaceForensics++: Learning to Detect Manipulated Facial Images

16 code implementations25 Jan 2019 Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner

In particular, the benchmark is based on DeepFakes, Face2Face, FaceSwap and NeuralTextures as prominent representatives for facial manipulations at random compression level and size.

DeepFake Detection Face Swapping +2

RegNet: Learning the Optimization of Direct Image-to-Image Pose Registration

no code implementations26 Dec 2018 Lei Han, Mengqi Ji, Lu Fang, Matthias Nießner

Direct image-to-image alignment that relies on the optimization of photometric error metrics suffers from limited convergence range and sensitivity to lighting conditions.

ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection

no code implementations6 Dec 2018 Davide Cozzolino, Justus Thies, Andreas Rössler, Christian Riess, Matthias Nießner, Luisa Verdoliva

We devise a learning-based forensic detector which adapts well to new domains, i. e., novel manipulation methods and can handle scenarios where only a handful of fake examples are available during training.

Domain Adaptation

DeepVoxels: Learning Persistent 3D Feature Embeddings

1 code implementation CVPR 2019 Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer

In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis.

3D Reconstruction Novel View Synthesis

TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes

1 code implementation CVPR 2019 Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkhouser, Matthias Nießner, Leonidas Guibas

We introduce, TextureNet, a neural network architecture designed to extract features from high-resolution signals associated with 3D surface meshes (e. g., color texture maps).

3D Semantic Segmentation

Scan2CAD: Learning CAD Model Alignment in RGB-D Scans

2 code implementations CVPR 2019 Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, Matthias Nießner

For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry.

3D Reconstruction

IGNOR: Image-guided Neural Object Rendering

no code implementations26 Nov 2018 Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.

Image Generation Novel View Synthesis

Scan2Mesh: From Unstructured Range Scans to 3D Meshes

1 code implementation CVPR 2019 Angela Dai, Matthias Nießner

We introduce Scan2Mesh, a novel data-driven generative approach which transforms an unstructured and potentially incomplete range scan into a structured 3D mesh representation.

Deep Video Portraits

no code implementations29 May 2018 Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, Christian Theobalt

In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network -- thus taking full control of the target.

Face Model

HeadOn: Real-time Reenactment of Human Portrait Videos

no code implementations29 May 2018 Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze.

3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation

1 code implementation ECCV 2018 Angela Dai, Matthias Nießner

We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network.

Scene Segmentation

ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans

no code implementations CVPR 2018 Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, Matthias Nießner

We introduce ScanComplete, a novel data-driven approach for taking an incomplete 3D scan of a scene as input and predicting a complete 3D model along with per-voxel semantic labels.

Semantic Segmentation

Multiframe Scene Flow with Piecewise Rigid Motion

no code implementations5 Oct 2017 Vladislav Golyanik, Kihwan Kim, Robert Maier, Matthias Nießner, Didier Stricker, Jan Kautz

We introduce a novel multiframe scene flow approach that jointly optimizes the consistency of the patch appearances and their local rigid motions from RGB-D image sequences.

Scene Flow Estimation

Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction

no code implementations25 May 2017 Benjamin Hepp, Matthias Nießner, Otmar Hilliges

We introduce a new method that efficiently computes a set of viewpoints and trajectories for high-quality 3D reconstructions in outdoor environments.

A Lightweight Approach for On-the-Fly Reflectance Estimation

no code implementations ICCV 2017 Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, Jan Kautz

In addition, we have created a large synthetic dataset, SynBRDF, which comprises a total of $500$K RGBD images rendered with a physically-based ray tracer under a variety of natural illumination, covering $5000$ materials and $5000$ shapes.

Color Constancy

Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis

2 code implementations CVPR 2017 Angela Dai, Charles Ruizhongtai Qi, Matthias Nießner

Although our 3D-EPN outperforms state-of-the-art completion method, the main contribution in our work lies in the combination of a data-driven shape predictor and analytic 3D shape synthesis.

3D Shape Generation

FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality

no code implementations11 Oct 2016 Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner

Based on reenactment of a prerecorded stereo video of the person without the HMD, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances.

Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging

no code implementations22 Apr 2016 Zachary DeVito, Michael Mara, Michael Zollhöfer, Gilbert Bernstein, Jonathan Ragan-Kelley, Christian Theobalt, Pat Hanrahan, Matthew Fisher, Matthias Nießner

Many graphics and vision problems can be expressed as non-linear least squares optimizations of objective functions over visual data, such as images and meshes.

BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration

no code implementations5 Apr 2016 Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Christian Theobalt

Our approach estimates globally optimized (i. e., bundle adjusted) poses in real-time, supports robust tracking with recovery from gross tracking failures (i. e., relocalization), and re-estimates the 3D model in real-time to ensure global consistency; all within a single framework.

3D Reconstruction Mixed Reality +1

VolumeDeform: Real-time Volumetric Non-rigid Reconstruction

no code implementations27 Mar 2016 Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, Marc Stamminger

We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints.

3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

2 code implementations CVPR 2017 Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, Thomas Funkhouser

To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions.

3D Reconstruction Point Cloud Registration

Cannot find the paper you are looking for? You can Submit a new open access paper.