Search Results for author: Matthias Nießner

Found 137 papers, 71 papers with code

Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation

no code implementations28 Mar 2024 Yujin Chen, Yinyu Nie, Benjamin Ummenhofer, Reiner Birkl, Michael Paulitsch, Matthias Müller, Matthias Nießner

In Mesh2NeRF, we propose an analytic solution to directly obtain ground-truth radiance fields from 3D meshes, characterizing the density field with an occupancy function featuring a defined surface thickness, and determining view-dependent color through a reflection function considering both the mesh and environment lighting.

3D Generation

DeepMIF: Deep Monotonic Implicit Fields for Large-Scale LiDAR 3D Mapping

no code implementations26 Mar 2024 Kutay Yılmaz, Matthias Nießner, Anastasiia Kornilova, Alexey Artemov

Recently, significant progress has been achieved in sensing real large-scale outdoor 3D environments, particularly by using modern acquisition equipment such as LiDAR sensors.

AutoInst: Automatic Instance-Based Segmentation of LiDAR 3D Scans

no code implementations24 Mar 2024 Cedric Perauer, Laurenz Adrian Heidrich, Haifan Zhang, Matthias Nießner, Anastasiia Kornilova, Alexey Artemov

To this end, we construct a learning framework consisting of two components: (1) a pseudo-annotation scheme for generating initial unsupervised pseudo-labels; and (2) a self-training algorithm for instance segmentation to fit robust, accurate instances from initial noisy proposals.

3D Instance Segmentation Scene Understanding +1

LightIt: Illumination Modeling and Control for Diffusion Models

no code implementations15 Mar 2024 Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, Yannick Hold-Geoffroy

Our method is the first that enables the generation of images with controllable, consistent lighting and performs on par with specialized relighting state-of-the-art methods.

Image Generation

ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models

1 code implementation4 Mar 2024 Lukas Höllein, Aljaž Božič, Norman Müller, David Novotny, Hung-Yu Tseng, Christian Richardt, Michael Zollhöfer, Matthias Nießner

In this paper, we present a method that leverages pretrained text-to-image models as a prior, and learn to generate multi-view images in a single denoising process from real-world data.

Denoising Image Generation +1

Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking

no code implementations12 Jan 2024 Wei Cao, Chang Luo, Biao Zhang, Matthias Nießner, Jiapeng Tang

To address these challenges, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations.

4D reconstruction Denoising +2

Intrinsic Image Diffusion for Indoor Single-view Material Estimation

1 code implementation19 Dec 2023 Peter Kocsis, Vincent Sitzmann, Matthias Nießner

We present Intrinsic Image Diffusion, a generative model for appearance decomposition of indoor scenes.

PolyDiff: Generating 3D Polygonal Meshes with Diffusion Models

no code implementations18 Dec 2023 Antonio Alliegro, Yawar Siddiqui, Tatiana Tommasi, Matthias Nießner

In contrast to methods that use alternate 3D shape representations (e. g. implicit representations), our approach is a discrete denoising diffusion probabilistic model that operates natively on the polygonal mesh data structure.

Avg Denoising

FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models

1 code implementation13 Dec 2023 Shivangi Aneja, Justus Thies, Angela Dai, Matthias Nießner

We propose a new latent diffusion model for this task, operating in the expression space of neural parametric head models, to synthesize audio-driven realistic head sequences.

3D Face Animation Audio Synthesis +1

Fast Training of Diffusion Transformer with Extreme Masking for 3D Point Clouds Generation

no code implementations12 Dec 2023 Shentong Mo, Enze Xie, Yue Wu, Junsong Chen, Matthias Nießner, Zhenguo Li

Motivated by the inherent redundancy of 3D compared to 2D, we propose FastDiT-3D, a novel masked diffusion transformer tailored for efficient 3D point cloud generation, which greatly reduces training costs.

3D Generation Denoising +1

GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians

1 code implementation4 Dec 2023 Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, Matthias Nießner

We introduce GaussianAvatars, a new method to create photorealistic head avatars that are fully controllable in terms of expression, pose, and viewpoint.

Face Model

DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars

no code implementations30 Nov 2023 Tobias Kirschstein, Simon Giebenhain, Matthias Nießner

DiffusionAvatars synthesizes a high-fidelity 3D head avatar of a person, offering intuitive control over both pose and expression.

PRS: Sharp Feature Priors for Resolution-Free Surface Remeshing

no code implementations30 Nov 2023 Natalia Soboleva, Olga Gorbunova, Maria Ivanova, Evgeny Burnaev, Matthias Nießner, Denis Zorin, Alexey Artemov

We define and learn a collection of surface-based fields to (1) capture sharp geometric features in the shape with an implicit vertexwise model and (2) approximate improvements in normals alignment obtained by applying edge-flips with an edgewise model.

Surface Reconstruction

Raising the Bar of AI-generated Image Detection with CLIP

no code implementations30 Nov 2023 Davide Cozzolino, Giovanni Poggi, Riccardo Corvi, Matthias Nießner, Luisa Verdoliva

Aim of this work is to explore the potential of pre-trained vision-language models (VLMs) for universal detection of AI-generated images.

SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors

no code implementations28 Nov 2023 Dave Zhenyu Chen, Haoxuan Li, Hsin-Ying Lee, Sergey Tulyakov, Matthias Nießner

We propose SceneTex, a novel method for effectively generating high-quality and style-consistent textures for indoor scenes using depth-to-image diffusion priors.

Texture Synthesis

MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers

2 code implementations27 Nov 2023 Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, Matthias Nießner

We introduce MeshGPT, a new approach for generating triangle meshes that reflects the compactness typical of artist-created meshes, in contrast to dense triangle meshes extracted by iso-surfacing methods from neural fields.

State of the Art on Diffusion Models for Visual Computing

no code implementations11 Oct 2023 Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T. Barron, Amit H. Bermano, Eric Ryan Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, C. Karen Liu, Lingjie Liu, Ben Mildenhall, Matthias Nießner, Björn Ommer, Christian Theobalt, Peter Wonka, Gordon Wetzstein

The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes.

ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes

no code implementations ICCV 2023 Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, Angela Dai

Each scene is captured with a high-end laser scanner at sub-millimeter resolution, along with registered 33-megapixel images from a DSLR camera, and RGB-D streams from an iPhone.

Novel View Synthesis Scene Understanding

Estimating Generic 3D Room Structures from 2D Annotations

1 code implementation NeurIPS 2023 Denys Rozumnyi, Stefan Popov, Kevis-Kokitsi Maninis, Matthias Nießner, Vittorio Ferrari

Based on these 2D annotations, we automatically reconstruct 3D plane equations for the structural elements and their spatial extent in the scene, and connect adjacent elements at the appropriate contact edges.

Scene Understanding

CAD-Estate: Large-scale CAD Model Annotation in RGB Videos

1 code implementation ICCV 2023 Kevis-Kokitsi Maninis, Stefan Popov, Matthias Nießner, Vittorio Ferrari

We propose a method for annotating videos of complex multi-object scenes with a globally-consistent 3D representation of the objects.

3D Object Reconstruction Object +1

GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields

no code implementations9 Jun 2023 Barbara Roessle, Norman Müller, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Matthias Nießner

Neural Radiance Fields (NeRF) have shown impressive novel view synthesis results; nonetheless, even thorough recordings yield imperfections in reconstructions, for instance due to poorly observed areas or minor lighting changes.

3D Scene Reconstruction Novel View Synthesis

HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion

1 code implementation10 May 2023 Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, Matthias Nießner

To close the gap to production-level quality, we introduce HumanRF, a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input, and enables playback from novel, unseen viewpoints.

Motion Synthesis Novel View Synthesis +1

NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads

no code implementations4 May 2023 Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, Matthias Nießner

We focus on reconstructing high-fidelity radiance fields of human heads, capturing their animations over time, and synthesizing re-renderings from novel viewpoints at arbitrary time steps.

HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion

1 code implementation ICCV 2023 Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, Angela Dai

HyperDiffusion operates directly on MLP weights and generates new neural implicit fields encoded by synthesized MLP parameters.

DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis

no code implementations24 Mar 2023 Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, Matthias Nießner

We introduce a diffusion network to synthesize a collection of 3D indoor objects by denoising a set of unordered object attributes.

Denoising Indoor Scene Synthesis +1

TriPlaneNet: An Encoder for EG3D Inversion

no code implementations23 Mar 2023 Ananta R. Bhattarai, Matthias Nießner, Artem Sevastopolsky

Recent progress in NeRF-based GANs has introduced a number of approaches for high-resolution and high-fidelity generative modeling of human heads with a possibility for novel view rendering.

Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models

1 code implementation ICCV 2023 Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, Matthias Nießner

We present Text2Room, a method for generating room-scale textured 3D meshes from a given text prompt as input.

Text to 3D

Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors

no code implementations CVPR 2023 Ji Hou, Xiaoliang Dai, Zijian He, Angela Dai, Matthias Nießner

Current popular backbones in computer vision, such as Vision Transformers (ViT) and ResNets are trained to perceive the world from 2D images.

Contrastive Learning Instance Segmentation +6

SSR-2D: Semantic 3D Scene Reconstruction from 2D Images

no code implementations7 Feb 2023 Junwen Huang, Alexey Artemov, Yujin Chen, Shuaifeng Zhi, Kai Xu, Matthias Nießner

In this work, we explore a central 3D scene modeling task, namely, semantic scene reconstruction without using any 3D annotations.

3D Scene Reconstruction Colorization +1

SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images

1 code implementation18 Dec 2022 Abdullah Hamdi, Bernard Ghanem, Matthias Nießner

SuRFNet employs partial SRFs from few/one images and a specialized SRF loss to learn to generate high-quality sparse voxel radiance fields that can be rendered from novel views.

Novel View Synthesis

ClipFace: Text-guided Editing of Textured 3D Morphable Models

1 code implementation2 Dec 2022 Shivangi Aneja, Justus Thies, Angela Dai, Matthias Nießner

Controllable editing and manipulation are given by language prompts to adapt texture and expression of the 3D morphable model.

Texture Synthesis

DiffRF: Rendering-Guided 3D Radiance Field Diffusion

no code implementations CVPR 2023 Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Matthias Nießner

We introduce DiffRF, a novel approach for 3D radiance field synthesis based on denoising diffusion probabilistic models.


Learning 3D Scene Priors with 2D Supervision

no code implementations CVPR 2023 Yinyu Nie, Angela Dai, Xiaoguang Han, Matthias Nießner

Holistic 3D scene understanding entails estimation of both layout configuration and object geometry in a 3D environment.

Scene Understanding

Neural Poisson: Indicator Functions for Neural Fields

no code implementations25 Nov 2022 Angela Dai, Matthias Nießner

Implicit neural field generating signed distance field representations (SDFs) of 3D shapes have shown remarkable progress in 3D shape reconstruction and generation.

3D Shape Reconstruction Surface Reconstruction

How to Boost Face Recognition with StyleGAN?

1 code implementation ICCV 2023 Artem Sevastopolsky, Yury Malkov, Nikita Durasov, Luisa Verdoliva, Matthias Nießner

We show that a simple approach based on fine-tuning pSp encoder for StyleGAN allows us to improve upon the state-of-the-art facial recognition and performs better compared to training on synthetic face identities.

Face Recognition

Neural Shape Deformation Priors

no code implementations11 Oct 2022 Jiapeng Tang, Lev Markhasin, Bi Wang, Justus Thies, Matthias Nießner

To this end, we introduce transformer-based deformation networks that represent a shape deformation as a composition of local surface deformations.

3D Multi-Object Tracking with Differentiable Pose Estimation

no code implementations28 Jun 2022 Dominik Schmauser, Zeju Qiu, Norman Müller, Matthias Nießner

We propose a novel approach for joint 3D multi-object tracking and reconstruction from RGB-D sequences in indoor environments.

3D Multi-Object Tracking Object +1

Fast Dynamic Radiance Fields with Time-Aware Neural Voxels

1 code implementation30 May 2022 Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, Qi Tian

A multi-distance interpolation method is proposed and applied on voxel features to model both small and large motions.

3DILG: Irregular Latent Grids for 3D Generative Modeling

1 code implementation27 May 2022 Biao Zhang, Matthias Nießner, Peter Wonka

All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.

3D Shape Modeling

End2End Multi-View Feature Matching with Differentiable Pose Optimization

no code implementations ICCV 2023 Barbara Roessle, Matthias Nießner

Training feature matching with gradients from pose optimization naturally learns to down-weight outliers and boosts pose estimation on image pairs compared to SuperGlue by 6. 7% on ScanNet.

Graph Attention Pose Estimation

AutoRF: Learning 3D Object Radiance Fields from Single View Observations

no code implementations CVPR 2022 Norman Müller, Andrea Simonelli, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, Peter Kontschieder

We introduce AutoRF - a new approach for learning neural 3D object representations where each object in the training set is observed by only a single view.

Novel View Synthesis Object

Audio-Visual Person-of-Interest DeepFake Detection

1 code implementation6 Apr 2022 Davide Cozzolino, Alessandro Pianese, Matthias Nießner, Luisa Verdoliva

The aim of this work is to propose a deepfake detector that can cope with the wide variety of manipulation methods and scenarios encountered in the real world.

Contrastive Learning DeepFake Detection +1

Texturify: Generating Textures on 3D Shape Surfaces

no code implementations5 Apr 2022 Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, Angela Dai

Texture cues on 3D objects are key to compelling visual representations, with the possibility to create high visual fidelity with inherent spatial consistency across different views.

3D Equivariant Graph Implicit Functions

no code implementations31 Mar 2022 Yunlu Chen, Basura Fernando, Hakan Bilen, Matthias Nießner, Efstratios Gavves

In this work, we address two key limitations of such representations, in failing to capture local 3D geometric fine details, and to learn from and generalize to shapes with unseen 3D transformations.

RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering

1 code implementation8 Mar 2022 Di Chang, Aljaž Božič, Tong Zhang, Qingsong Yan, Yingcong Chen, Sabine Süsstrunk, Matthias Nießner

Finding accurate correspondences among different views is the Achilles' heel of unsupervised Multi-View Stereo (MVS).

Neural Rendering

Neural Head Avatars from Monocular RGB Videos

no code implementations CVPR 2022 Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, Justus Thies

We present Neural Head Avatars, a novel neural representation that explicitly models the surface geometry and appearance of an animatable human avatar that can be used for teleconferencing in AR/VR or other applications in the movie or games industry that rely on a digital human.

Novel View Synthesis

Pose2Room: Understanding 3D Scenes from Human Activities

no code implementations1 Dec 2021 Yinyu Nie, Angela Dai, Xiaoguang Han, Matthias Nießner

To this end, we propose P2R-Net to learn a probabilistic 3D model of the objects in a scene characterized by their class categories and oriented 3D bounding boxes, based on an input observed human trajectory in the environment.


Panoptic 3D Scene Reconstruction From a Single RGB Image

1 code implementation NeurIPS 2021 Manuel Dahnert, Ji Hou, Matthias Nießner, Angela Dai

Inspired by 2D panoptic segmentation, we propose to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction - from a single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations.

3D Instance Segmentation 3D Scene Reconstruction +5

4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface

1 code implementation ICCV 2021 Yang Li, Hikari Takehara, Takafumi Taketomi, Bo Zheng, Matthias Nießner

Tracking non-rigidly deforming scenes using range sensors has numerous applications including computer vision, AR/VR, and robotics.

Motion Estimation

Pri3D: Can 3D Priors Help 2D Representation Learning?

1 code implementation ICCV 2021 Ji Hou, Saining Xie, Benjamin Graham, Angela Dai, Matthias Nießner

Inspired by these advances in geometric understanding, we aim to imbue image-based perception with representations learned under geometric constraints.

Contrastive Learning Instance Segmentation +5

Neural RGB-D Surface Reconstruction

2 code implementations CVPR 2022 Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, Justus Thies

Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR.

Image Generation Mixed Reality +2

Dynamic Surface Function Networks for Clothed Human Bodies

1 code implementation ICCV 2021 Andrei Burov, Matthias Nießner, Justus Thies

To this end, we explicitly model the surface of the person using a multi-layer perceptron (MLP) which is embedded into the canonical space of the SMPL body model.

NPMs: Neural Parametric Models for 3D Deformable Shapes

1 code implementation ICCV 2021 Pablo Palafox, Aljaž Božič, Justus Thies, Matthias Nießner, Angela Dai

Crucially, once learned, our neural parametric models of shape and pose enable optimization over the learned spaces to fit to new observations, similar to the fitting of a traditional parametric model, e. g., SMPL.

Pose Transfer

RetrievalFuse: Neural 3D Scene Reconstruction with a Database

1 code implementation ICCV 2021 Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, Angela Dai

3D reconstruction of large scenes is a challenging problem due to the high-complexity nature of the solution space, in particular for generative neural networks.

3D Reconstruction 3D Scene Reconstruction +3

SceneFormer: Indoor Scene Generation with Transformers

2 code implementations17 Dec 2020 Xinpeng Wang, Chandan Yeshwanth, Matthias Nießner

In contrast, we do not use any appearance information, and implicitly learn object relations using the self-attention mechanism of transformers.

Scene Generation

Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts

2 code implementations CVPR 2021 Ji Hou, Benjamin Graham, Matthias Nießner, Saining Xie

The rapid progress in 3D scene understanding has come with growing demand for data; however, collecting and annotating 3D scenes (e. g. point clouds) are notoriously hard.

3D Semantic Segmentation Instance Segmentation +2

Seeing Behind Objects for 3D Multi-Object Tracking in RGB-D Sequences

no code implementations CVPR 2021 Norman Müller, Yu-Shiang Wong, Niloy J. Mitra, Angela Dai, Matthias Nießner

From a sequence of RGB-D frames, we detect objects in each frame and learn to predict their complete object geometry as well as a dense correspondence mapping into a canonical space.

3D Multi-Object Tracking Object

Vid2CAD: CAD Model Alignment using Multi-View Constraints from Videos

1 code implementation8 Dec 2020 Kevis-Kokitsi Maninis, Stefan Popov, Matthias Nießner, Vittorio Ferrari

We address the task of aligning CAD models to a video sequence of a complex scene containing multiple objects.

ID-Reveal: Identity-aware DeepFake Video Detection

1 code implementation ICCV 2021 Davide Cozzolino, Andreas Rössler, Justus Thies, Matthias Nießner, Luisa Verdoliva

A major challenge in DeepFake forgery detection is that state-of-the-art algorithms are mostly trained to detect a specific fake method.

Face Swapping Metric Learning

RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction

1 code implementation CVPR 2021 Yinyu Nie, Ji Hou, Xiaoguang Han, Matthias Nießner

In this work, we introduce RfD-Net that jointly detects and reconstructs dense object surfaces directly from raw point clouds.

Object object-detection +4

Face2Face: Real-time Face Capture and Reenactment of RGB Videos

2 code implementations CVPR 2016 Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner

Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.

Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition

no code implementations29 Jun 2020 Hassan Abu Alhaija, Siva Karthik Mustikovela, Justus Thies, Varun Jampani, Matthias Nießner, Andreas Geiger, Carsten Rother

Neural rendering techniques promise efficient photo-realistic image synthesis while at the same time providing rich control over scene parameters by learning the physical image formation process.

Image-to-Image Translation Intrinsic Image Decomposition +1

SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans

1 code implementation CVPR 2021 Angela Dai, Yawar Siddiqui, Justus Thies, Julien Valentin, Matthias Nießner

We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion.

3D Reconstruction Scene Generation

Neural Non-Rigid Tracking

1 code implementation NeurIPS 2020 Aljaž Božič, Pablo Palafox, Michael Zollhöfer, Angela Dai, Justus Thies, Matthias Nießner

We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction by a learned robust optimization.

Generalized Zero and Few-Shot Transfer for Facial Forgery Detection

no code implementations21 Jun 2020 Shivangi Aneja, Matthias Nießner

We propose Deep Distribution Transfer(DDT), a new transfer learning approach to address the problem of zero and few-shot transfer in the context of facial forgery detection.

Domain Adaptation Few-Shot Learning +1

State of the Art on Neural Rendering

no code implementations8 Apr 2020 Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B. Goldman, Michael Zollhöfer

Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e. g., by the integration of differentiable rendering into network training.

BIG-bench Machine Learning Image Generation +2

3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation

1 code implementation30 Mar 2020 Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner

We show that grouping proposals improves over NMS and outperforms previous state-of-the-art methods on the tasks of 3D object detection and semantic instance segmentation on the ScanNetV2 benchmark and the S3DIS dataset.

3D Instance Segmentation 3D Object Detection +3

Modeling 3D Shapes by Reinforcement Learning

2 code implementations ECCV 2020 Cheng Lin, Tingxiang Fan, Wenping Wang, Matthias Nießner

We explore how to enable machines to model 3D shapes like human modelers using deep reinforcement learning (RL).

Imitation Learning reinforcement-learning +1

SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans

no code implementations ECCV 2020 Armen Avetisyan, Tatiana Khanova, Christopher Choy, Denver Dash, Angela Dai, Matthias Nießner

We present a novel approach to reconstructing lightweight, CAD-based representations of scanned 3D environments from commodity RGB-D sensors.


Learning to Optimize Non-Rigid Tracking

no code implementations CVPR 2020 Yang Li, Aljaž Božič, Tianwei Zhang, Yanli Ji, Tatsuya Harada, Matthias Nießner

One of the widespread solutions for non-rigid tracking has a nested-loop structure: with Gauss-Newton to minimize a tracking objective in the outer loop, and Preconditioned Conjugate Gradient (PCG) to solve a sparse linear system in the inner loop.

Local Implicit Grid Representations for 3D Scenes

1 code implementation19 Mar 2020 Chiyu Max Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser

Then, we use the decoder as a component in a shape optimization that solves for a set of latent codes on a regular grid of overlapping crops such that an interpolation of the decoded local shapes matches a partial or noisy observation.

3D Shape Representation Surface Reconstruction

Adversarial Texture Optimization from RGB-D Scans

1 code implementation CVPR 2020 Jingwei Huang, Justus Thies, Angela Dai, Abhijit Kundu, Chiyu Max Jiang, Leonidas Guibas, Matthias Nießner, Thomas Funkhouser

In this work, we present a novel approach for color texture generation using a conditional adversarial loss obtained from weakly-supervised views.

Surface Reconstruction Texture Synthesis

Image-guided Neural Object Rendering

no code implementations ICLR 2020 Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.

Image Generation Object

Neural Voice Puppetry: Audio-driven Facial Reenactment

1 code implementation ECCV 2020 Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, Matthias Nießner

Neural Voice Puppetry has a variety of use-cases, including audio-driven video avatars, video dubbing, and text-driven video synthesis of a talking head.

Face Model Neural Rendering +2

DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data

1 code implementation9 Dec 2019 Aljaž Božič, Michael Zollhöfer, Christian Theobalt, Matthias Nießner

Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus.

3D Reconstruction RGB-D Reconstruction

SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans

2 code implementations CVPR 2020 Angela Dai, Christian Diller, Matthias Nießner

We present a novel approach that converts partial and noisy RGB-D scans into high-quality 3D scene reconstructions by inferring unobserved scene geometry.

3D Reconstruction

SpoC: Spoofing Camera Fingerprints

no code implementations27 Nov 2019 Davide Cozzolino, Justus Thies, Andreas Rössler, Matthias Nießner, Luisa Verdoliva

Given a GAN-generated image, we insert the traces of a specific camera model into it and deceive state-of-the-art detectors into believing the image was acquired by that model.

Demosaicking Misinformation

ViewAL: Active Learning with Viewpoint Entropy for Semantic Segmentation

1 code implementation CVPR 2020 Yawar Siddiqui, Julien Valentin, Matthias Nießner

To incorporate this uncertainty measure, we introduce a new viewpoint entropy formulation, which is the basis of our active learning strategy.

Active Learning Semantic Segmentation +1

Joint Embedding of 3D Scan and CAD Objects

1 code implementation ICCV 2019 Manuel Dahnert, Angela Dai, Leonidas Guibas, Matthias Nießner

We propose a novel approach to learn a joint embedding space between scan and CAD geometry, where semantically similar objects from both domains lie close together.


RIO: 3D Object Instance Re-Localization in Changing Indoor Environments

1 code implementation ICCV 2019 Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, Matthias Nießner

In this work, we introduce the task of 3D object instance re-localization (RIO): given one or multiple objects in an RGB-D scan, we want to estimate their corresponding 6DoF poses in another 3D scan of the same environment taken at a later point in time.

Object Scene Understanding

End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans

1 code implementation ICCV 2019 Armen Avetisyan, Angela Dai, Matthias Nießner

We present a novel, end-to-end approach to align CAD models to an 3D scan of a scene, enabling transformation of a noisy, incomplete 3D scan to a compact, CAD reconstruction with clean, complete object geometry.

CAD Reconstruction Object +1

Deferred Neural Rendering: Image Synthesis using Neural Textures

4 code implementations28 Apr 2019 Justus Thies, Michael Zollhöfer, Matthias Nießner

Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline.

Image Generation Neural Rendering +1

RevealNet: Seeing Behind Objects in RGB-D Scans

no code implementations CVPR 2020 Ji Hou, Angela Dai, Matthias Nießner

Thus, we introduce the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances and infer their complete object geometry.

3D Reconstruction 3D Semantic Instance Segmentation +2

Inverse Path Tracing for Joint Material and Lighting Estimation

no code implementations17 Mar 2019 Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, Matthias Nießner

We introduce Inverse Path Tracing, a novel approach to jointly estimate the material properties of objects and light sources in indoor scenes by using an invertible light transport simulation.

Lighting Estimation Retrieval

Homogeneous Linear Inequality Constraints for Neural Network Activations

1 code implementation5 Feb 2019 Thomas Frerix, Matthias Nießner, Daniel Cremers

One way to achieve this task is by means of a projection step at test time after unconstrained training.

FaceForensics++: Learning to Detect Manipulated Facial Images

14 code implementations25 Jan 2019 Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner

In particular, the benchmark is based on DeepFakes, Face2Face, FaceSwap and NeuralTextures as prominent representatives for facial manipulations at random compression level and size.

DeepFake Detection Face Swapping +2

RegNet: Learning the Optimization of Direct Image-to-Image Pose Registration

1 code implementation26 Dec 2018 Lei Han, Mengqi Ji, Lu Fang, Matthias Nießner

Direct image-to-image alignment that relies on the optimization of photometric error metrics suffers from limited convergence range and sensitivity to lighting conditions.

ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection

no code implementations6 Dec 2018 Davide Cozzolino, Justus Thies, Andreas Rössler, Christian Riess, Matthias Nießner, Luisa Verdoliva

We devise a learning-based forensic detector which adapts well to new domains, i. e., novel manipulation methods and can handle scenarios where only a handful of fake examples are available during training.

Domain Adaptation

DeepVoxels: Learning Persistent 3D Feature Embeddings

1 code implementation CVPR 2019 Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer

In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis.

3D Reconstruction Novel View Synthesis

TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes

1 code implementation CVPR 2019 Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkhouser, Matthias Nießner, Leonidas Guibas

We introduce, TextureNet, a neural network architecture designed to extract features from high-resolution signals associated with 3D surface meshes (e. g., color texture maps).

3D Semantic Segmentation

Scan2CAD: Learning CAD Model Alignment in RGB-D Scans

2 code implementations CVPR 2019 Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, Matthias Nießner

For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry.

3D Reconstruction

IGNOR: Image-guided Neural Object Rendering

no code implementations26 Nov 2018 Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.

Image Generation Novel View Synthesis +1

Scan2Mesh: From Unstructured Range Scans to 3D Meshes

1 code implementation CVPR 2019 Angela Dai, Matthias Nießner

We introduce Scan2Mesh, a novel data-driven generative approach which transforms an unstructured and potentially incomplete range scan into a structured 3D mesh representation.

Deep Video Portraits

no code implementations29 May 2018 Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, Christian Theobalt

In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network -- thus taking full control of the target.

Face Model

HeadOn: Real-time Reenactment of Human Portrait Videos

no code implementations29 May 2018 Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias Nießner

We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze.

3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation

1 code implementation ECCV 2018 Angela Dai, Matthias Nießner

We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network.

3D Architecture Scene Segmentation +1

ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans

no code implementations CVPR 2018 Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, Matthias Nießner

We introduce ScanComplete, a novel data-driven approach for taking an incomplete 3D scan of a scene as input and predicting a complete 3D model along with per-voxel semantic labels.

Semantic Segmentation

Multiframe Scene Flow with Piecewise Rigid Motion

no code implementations5 Oct 2017 Vladislav Golyanik, Kihwan Kim, Robert Maier, Matthias Nießner, Didier Stricker, Jan Kautz

We introduce a novel multiframe scene flow approach that jointly optimizes the consistency of the patch appearances and their local rigid motions from RGB-D image sequences.

Scene Flow Estimation

Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction

no code implementations25 May 2017 Benjamin Hepp, Matthias Nießner, Otmar Hilliges

We introduce a new method that efficiently computes a set of viewpoints and trajectories for high-quality 3D reconstructions in outdoor environments.

A Lightweight Approach for On-the-Fly Reflectance Estimation

no code implementations ICCV 2017 Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, Jan Kautz

In addition, we have created a large synthetic dataset, SynBRDF, which comprises a total of $500$K RGBD images rendered with a physically-based ray tracer under a variety of natural illumination, covering $5000$ materials and $5000$ shapes.

Color Constancy

Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis

2 code implementations CVPR 2017 Angela Dai, Charles Ruizhongtai Qi, Matthias Nießner

Although our 3D-EPN outperforms state-of-the-art completion method, the main contribution in our work lies in the combination of a data-driven shape predictor and analytic 3D shape synthesis.

3D Shape Generation

FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality

no code implementations11 Oct 2016 Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner

Based on reenactment of a prerecorded stereo video of the person without the HMD, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances.

Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging

no code implementations22 Apr 2016 Zachary DeVito, Michael Mara, Michael Zollhöfer, Gilbert Bernstein, Jonathan Ragan-Kelley, Christian Theobalt, Pat Hanrahan, Matthew Fisher, Matthias Nießner

Many graphics and vision problems can be expressed as non-linear least squares optimizations of objective functions over visual data, such as images and meshes.

BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration

1 code implementation5 Apr 2016 Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Christian Theobalt

Our approach estimates globally optimized (i. e., bundle adjusted) poses in real-time, supports robust tracking with recovery from gross tracking failures (i. e., relocalization), and re-estimates the 3D model in real-time to ensure global consistency; all within a single framework.

3D Reconstruction Mixed Reality +1

VolumeDeform: Real-time Volumetric Non-rigid Reconstruction

no code implementations27 Mar 2016 Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, Marc Stamminger

We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints.

3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

2 code implementations CVPR 2017 Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, Thomas Funkhouser

To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions.

3D Reconstruction Point Cloud Registration

Cannot find the paper you are looking for? You can Submit a new open access paper.