Search Results for author:
Found 159 papers, 82 papers with code
TUM2TWIN: Introducing the Large-Scale Multimodal Urban Digital Twin Benchmark Dataset
By ensuring georeferenced indoor-outdoor acquisition, high accuracy, and multimodal data integration, the benchmark supports robust analysis of sensors and the development of advanced reconstruction methods.
QuickSplat: Fast 3D Surface Reconstruction via Learned Gaussian Initialization
Surface reconstruction is fundamental to computer vision and graphics, enabling applications in 3D modeling, mixed reality, robotics, and more.
Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction
To evaluate our method, we introduce a new benchmark for single-image face reconstruction, which features high diversity facial expressions, viewing angles, and ethnicities.
SHeaP: Self-Supervised Head Geometry Predictor Learned via 2D Gaussians
Given a source image, we predict a 3DMM mesh and a set of Gaussians that are rigged to this mesh.
Animating the Uncaptured: Humanoid Mesh Animation with Video Diffusion Models
From an input static 3D humanoid mesh and a text prompt describing the desired animation, we synthesize a corresponding video conditioned on a rendered image of the 3D mesh.
MTGS: Multi-Traversal Gaussian Splatting
To address these issues, we propose Multi-Traversal Gaussian Splatting (MTGS), a novel approach that reconstructs high-quality driving scenes from arbitrarily collected multi-traversal data by modeling a shared static geometry while separately handling dynamic elements and appearance variations.
MeshPad: Interactive Sketch-Conditioned Artist-Designed Mesh Generation and Editing
To this end, we focus on enabling consistent edits by decomposing editing into 'deletion' of regions of a mesh, followed by 'addition' of new mesh geometry.
Avat3r: Large Animatable Gaussian Reconstruction Model for High-fidelity 3D Head Avatars
Finally, we increase robustness by feeding input images with different expressions to our model during training, enabling the reconstruction of 3D head avatars from inconsistent inputs, e. g., an imperfect phone capture with accidental movement, or frames from a monocular video.
LinPrim: Linear Primitives for Differentiable Volumetric Rendering
Volumetric rendering has become central to modern novel view synthesis methods, which use differentiable rendering to optimize 3D scene representations directly from observed views.
Coherent 3D Scene Diffusion From a Single RGB Image
Motivated by the ill-posed nature of the task and to obtain consistent scene reconstruction results, we learn a generative scene prior by conditioning on all scene objects simultaneously to capture the scene context and by allowing the model to learn inter-object relationships throughout the diffusion process.
PrEditor3D: Fast and Precise 3D Shape Editing
The edited 3D mesh aligns well with the prompts, and remains identical for regions that are not intended to be altered.
GaussianSpeech: Audio-Driven Gaussian Avatars
We introduce GaussianSpeech, a novel approach that synthesizes high-fidelity animation sequences of photo-realistic, personalized 3D human head avatars from spoken audio.
L3DG: Latent 3D Gaussian Diffusion
We propose L3DG, the first approach for generative 3D modeling of 3D Gaussians through a latent 3D Gaussian diffusion formulation.
Zero-Shot Detection of AI-Generated Images
Detecting AI-generated images has become an extraordinarily difficult challenge as new generative architectures emerge on a daily basis with more and more capabilities and unprecedented realism.
3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-Marquardt
We present 3DGS-LM, a new method that accelerates the reconstruction of 3D Gaussian Splatting (3DGS) by replacing its ADAM optimizer with a tailored Levenberg-Marquardt (LM).
LT3SD: Latent Trees for 3D Scene Diffusion
We present LT3SD, a novel latent diffusion model for large-scale 3D scene generation.
G3DST: Generalizing 3D Style Transfer with Neural Radiance Fields across Scenes and Styles
Existing methods for NeRF-based 3D style transfer need extensive per-scene optimization for single or multiple styles, limiting the applicability and efficiency of 3D style transfer.
Robust 3D Gaussian Splatting for Novel View Synthesis in Presence of Distractors
Our goal is to identify and ignore such distractors during the 3D Gaussian optimization to obtain a clean reconstruction.
MultiDiff: Consistent Novel View Synthesis from a Single Image
We introduce MultiDiff, a novel approach for consistent novel view synthesis of scenes from a single RGB image.
GGHead: Fast and Generalizable 3D Gaussian Heads
Unfortunately, existing 3D GANs struggle to scale to generate samples at high resolutions due to their relatively slow train and render speeds, and typically have to rely on 2D superresolution networks at the expense of global 3D consistency.
NPGA: Neural Parametric Gaussian Avatars
In this work, we propose Neural Parametric Gaussian Avatars (NPGA), a data-driven approach to create high-fidelity, controllable avatars from multi-view video recordings.
When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
Hence, with this paper, we aim to chart a course for future research that explores and expands the capabilities of 3D-LLMs in understanding and interacting with the complex 3D world.
Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation
In Mesh2NeRF, we propose an analytic solution to directly obtain ground-truth radiance fields from 3D meshes, characterizing the density field with an occupancy function featuring a defined surface thickness, and determining view-dependent color through a reflection function considering both the mesh and environment lighting.
DeepMIF: Deep Monotonic Implicit Fields for Large-Scale LiDAR 3D Mapping
Recently, significant progress has been achieved in sensing real large-scale outdoor 3D environments, particularly by using modern acquisition equipment such as LiDAR sensors.
AutoInst: Automatic Instance-Based Segmentation of LiDAR 3D Scans
To this end, we construct a learning framework consisting of two components: (1) a pseudo-annotation scheme for generating initial unsupervised pseudo-labels; and (2) a self-training algorithm for instance segmentation to fit robust, accurate instances from initial noisy proposals.
LightIt: Illumination Modeling and Control for Diffusion Models
Our method is the first that enables the generation of images with controllable, consistent lighting and performs on par with specialized relighting state-of-the-art methods.
ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models
In this paper, we present a method that leverages pretrained text-to-image models as a prior, and learn to generate multi-view images in a single denoising process from real-world data.
Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking
To address these challenges, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations.
Intrinsic Image Diffusion for Indoor Single-view Material Estimation
We present Intrinsic Image Diffusion, a generative model for appearance decomposition of indoor scenes.
PolyDiff: Generating 3D Polygonal Meshes with Diffusion Models
In contrast to methods that use alternate 3D shape representations (e. g. implicit representations), our approach is a discrete denoising diffusion probabilistic model that operates natively on the polygonal mesh data structure.
FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models
We propose a new latent diffusion model for this task, operating in the expression space of neural parametric head models, to synthesize audio-driven realistic head sequences.
Fast Training of Diffusion Transformer with Extreme Masking for 3D Point Clouds Generation
Motivated by the inherent redundancy of 3D compared to 2D, we propose FastDiT-3D, a novel masked diffusion transformer tailored for efficient 3D point cloud generation, which greatly reduces training costs.
MonoNPHM: Dynamic Head Reconstruction from Monocular Videos
We present Monocular Neural Parametric Head Models (MonoNPHM) for dynamic 3D head reconstructions from monocular RGB videos.
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians
We introduce GaussianAvatars, a new method to create photorealistic head avatars that are fully controllable in terms of expression, pose, and viewpoint.
PRS: Sharp Feature Priors for Resolution-Free Surface Remeshing
We define and learn a collection of surface-based fields to (1) capture sharp geometric features in the shape with an implicit vertexwise model and (2) approximate improvements in normals alignment obtained by applying edge-flips with an edgewise model.
DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars
DiffusionAvatars synthesizes a high-fidelity 3D head avatar of a person, offering intuitive control over both pose and expression.
Raising the Bar of AI-generated Image Detection with CLIP
The aim of this work is to explore the potential of pre-trained vision-language models (VLMs) for universal detection of AI-generated images.
SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors
We propose SceneTex, a novel method for effectively generating high-quality and style-consistent textures for indoor scenes using depth-to-image diffusion priors.
MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers
We introduce MeshGPT, a new approach for generating triangle meshes that reflects the compactness typical of artist-created meshes, in contrast to dense triangle meshes extracted by iso-surfacing methods from neural fields.
State of the Art on Diffusion Models for Visual Computing
The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes.
ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes
Each scene is captured with a high-end laser scanner at sub-millimeter resolution, along with registered 33-megapixel images from a DSLR camera, and RGB-D streams from an iPhone.
DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
Recent Diffusion Transformers (e. g., DiT) have demonstrated their powerful effectiveness in generating high-quality 2D images.
Ranked #1 on
Point Cloud Generation
on ShapeNet Car
CAD-Estate: Large-scale CAD Model Annotation in RGB Videos
We propose a method for annotating videos of complex multi-object scenes with a globally-consistent 3D representation of the objects.
Estimating Generic 3D Room Structures from 2D Annotations
Based on these 2D annotations, we automatically reconstruct 3D plane equations for the structural elements and their spatial extent in the scene, and connect adjacent elements at the appropriate contact edges.
GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
Neural Radiance Fields (NeRF) have shown impressive novel view synthesis results; nonetheless, even thorough recordings yield imperfections in reconstructions, for instance due to poorly observed areas or minor lighting changes.
HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
To close the gap to production-level quality, we introduce HumanRF, a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input, and enables playback from novel, unseen viewpoints.
NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads
We focus on reconstructing high-fidelity radiance fields of human heads, capturing their animations over time, and synthesizing re-renderings from novel viewpoints at arbitrary time steps.
HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion
HyperDiffusion operates directly on MLP weights and generates new neural implicit fields encoded by synthesized MLP parameters.
DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis
We introduce a diffusion network to synthesize a collection of 3D indoor objects by denoising a set of unordered object attributes.
TriPlaneNet: An Encoder for EG3D Inversion
Recent progress in NeRF-based GANs has introduced a number of approaches for high-resolution and high-fidelity generative modeling of human heads with a possibility for novel view rendering.
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models
We present Text2Room, a method for generating room-scale textured 3D meshes from a given text prompt as input.
Text2Tex: Text-driven Texture Synthesis via Diffusion Models
We present Text2Tex, a novel method for generating high-quality textures for 3D meshes from the given text prompts.
Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors
Current popular backbones in computer vision, such as Vision Transformers (ViT) and ResNets are trained to perceive the world from 2D images.
SSR-2D: Semantic 3D Scene Reconstruction from 2D Images
Most deep learning approaches to comprehensive semantic modeling of 3D indoor spaces require costly dense annotations in the 3D domain.
Panoptic Lifting for 3D Scene Understanding with Neural Fields
We propose Panoptic Lifting, a novel approach for learning panoptic 3D volumetric representations from images of in-the-wild scenes.
SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images
SuRFNet employs partial SRFs from few/one images and a specialized SRF loss to learn to generate high-quality sparse voxel radiance fields that can be rendered from novel views.
Learning Neural Parametric Head Models
We propose a novel 3D morphable model for complete human heads based on hybrid neural fields.
ObjectMatch: Robust Registration using Canonical Object Correspondences
We present ObjectMatch, a semantic and object-centric camera pose estimator for RGB-D SLAM pipelines.
ClipFace: Text-guided Editing of Textured 3D Morphable Models
Controllable editing and manipulation are given by language prompts to adapt texture and expression of the 3D morphable model.
DiffRF: Rendering-Guided 3D Radiance Field Diffusion
We introduce DiffRF, a novel approach for 3D radiance field synthesis based on denoising diffusion probabilistic models.
UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding
Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships.
Learning 3D Scene Priors with 2D Supervision
Holistic 3D scene understanding entails estimation of both layout configuration and object geometry in a 3D environment.
Neural Poisson: Indicator Functions for Neural Fields
Implicit neural field generating signed distance field representations (SDFs) of 3D shapes have shown remarkable progress in 3D shape reconstruction and generation.
How to Boost Face Recognition with StyleGAN?
We show that a simple approach based on fine-tuning pSp encoder for StyleGAN allows us to improve upon the state-of-the-art facial recognition and performs better compared to training on synthetic face identities.
Neural Shape Deformation Priors
To this end, we introduce transformer-based deformation networks that represent a shape deformation as a composition of local surface deformations.
The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes
This allows us to improve the generalization of a CNN-based model without any increase in the number of weights at test time.
3D Multi-Object Tracking with Differentiable Pose Estimation
We propose a novel approach for joint 3D multi-object tracking and reconstruction from RGB-D sequences in indoor environments.
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels
A multi-distance interpolation method is proposed and applied on voxel features to model both small and large motions.
3DILG: Irregular Latent Grids for 3D Generative Modeling
All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.
End2End Multi-View Feature Matching with Differentiable Pose Optimization
Training feature matching with gradients from pose optimization naturally learns to down-weight outliers and boosts pose estimation on image pairs compared to SuperGlue by 6. 7% on ScanNet.
AutoRF: Learning 3D Object Radiance Fields from Single View Observations
We introduce AutoRF - a new approach for learning neural 3D object representations where each object in the training set is observed by only a single view.
Audio-Visual Person-of-Interest DeepFake Detection
The aim of this work is to propose a deepfake detector that can cope with the wide variety of manipulation methods and scenarios encountered in the real world.
Texturify: Generating Textures on 3D Shape Surfaces
Texture cues on 3D objects are key to compelling visual representations, with the possibility to create high visual fidelity with inherent spatial consistency across different views.
3D Equivariant Graph Implicit Functions
In this work, we address two key limitations of such representations, in failing to capture local 3D geometric fine details, and to learn from and generalize to shapes with unseen 3D transformations.
RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering
Finding accurate correspondences among different views is the Achilles' heel of unsupervised Multi-View Stereo (MVS).
Dense Depth Priors for Neural Radiance Fields from Sparse Input Views
To this end, we leverage dense depth priors in order to constrain the NeRF optimization.
4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding
We present a new approach to instill 4D dynamic object priors into learned 3D representations by unsupervised pre-training.
Ranked #22 on
3D Instance Segmentation
on ScanNet(v2)
ROCA: Robust CAD Model Retrieval and Alignment from a Single Image
We present ROCA, a novel end-to-end approach that retrieves and aligns 3D CAD models from a shape database to a single input image.
3D Dense Shape Correspondence
3D Object Detection From Monocular Images
+2
D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding
Our D3Net unifies dense captioning and visual grounding in 3D in a self-critical manner.
Ranked #8 on
3D dense captioning
on Nr3D
Neural Head Avatars from Monocular RGB Videos
We present Neural Head Avatars, a novel neural representation that explicitly models the surface geometry and appearance of an animatable human avatar that can be used for teleconferencing in AR/VR or other applications in the movie or games industry that rely on a digital human.
StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions
Style transfer typically operates on 2D images, making stylization of a mesh challenging.
Pose2Room: Understanding 3D Scenes from Human Activities
To this end, we propose P2R-Net to learn a probabilistic 3D model of the objects in a scene characterized by their class categories and oriented 3D bounding boxes, based on an input observed human trajectory in the environment.
Panoptic 3D Scene Reconstruction From a Single RGB Image
Inspired by 2D panoptic segmentation, we propose to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction - from a single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations.
Egocentric Videoconferencing
Even holding a mobile phone camera in the front of the face while sitting for a long duration is not convenient.
TransformerFusion: Monocular RGB Scene Reconstruction using Transformers
We introduce TransformerFusion, a transformer-based 3D scene reconstruction approach.
4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface
Tracking non-rigidly deforming scenes using range sensors has numerous applications including computer vision, AR/VR, and robotics.
Pri3D: Can 3D Priors Help 2D Representation Learning?
Inspired by these advances in geometric understanding, we aim to imbue image-based perception with representations learned under geometric constraints.
Neural RGB-D Surface Reconstruction
Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR.
Dynamic Surface Function Networks for Clothed Human Bodies
To this end, we explicitly model the surface of the person using a multi-layer perceptron (MLP) which is embedded into the canonical space of the SMPL body model.
NPMs: Neural Parametric Models for 3D Deformable Shapes
Crucially, once learned, our neural parametric models of shape and pose enable optimization over the learned spaces to fit to new observations, similar to the fitting of a traditional parametric model, e. g., SMPL.
RetrievalFuse: Neural 3D Scene Reconstruction with a Database
3D reconstruction of large scenes is a challenging problem due to the high-complexity nature of the solution space, in particular for generative neural networks.
COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning
We propose a self-supervised training strategy where we only need a set of captioned images.
SceneFormer: Indoor Scene Generation with Transformers
In contrast, we do not use any appearance information, and implicitly learn object relations using the self-attention mechanism of transformers.
Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts
The rapid progress in 3D scene understanding has come with growing demand for data; however, collecting and annotating 3D scenes (e. g. point clouds) are notoriously hard.
Ranked #15 on
3D Semantic Segmentation
on ScanNet200
Seeing Behind Objects for 3D Multi-Object Tracking in RGB-D Sequences
From a sequence of RGB-D frames, we detect objects in each frame and learn to predict their complete object geometry as well as a dense correspondence mapping into a canonical space.
Vid2CAD: CAD Model Alignment using Multi-View Constraints from Videos
We address the task of aligning CAD models to a video sequence of a complex scene containing multiple objects.
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction
We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face.
ID-Reveal: Identity-aware DeepFake Video Detection
A major challenge in DeepFake forgery detection is that state-of-the-art algorithms are mostly trained to detect a specific fake method.
Scan2Cap: Context-aware Dense Captioning in RGB-D Scans
We introduce the task of dense captioning in 3D scans from commodity RGB-D sensors.
Ranked #9 on
3D dense captioning
on ScanRefer Dataset
Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction
We introduce Neural Deformation Graphs for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.
RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction
In this work, we introduce RfD-Net that jointly detects and reconstructs dense object surfaces directly from raw point clouds.
Face2Face: Real-time Face Capture and Reenactment of RGB Videos
Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.
Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition
Neural rendering techniques promise efficient photo-realistic image synthesis while at the same time providing rich control over scene parameters by learning the physical image formation process.
SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans
We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion.
Neural Non-Rigid Tracking
We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction by a learned robust optimization.
Generalized Zero and Few-Shot Transfer for Facial Forgery Detection
We propose Deep Distribution Transfer(DDT), a new transfer learning approach to address the problem of zero and few-shot transfer in the context of facial forgery detection.
Explicitly Modeled Attention Maps for Image Classification
Self-attention networks have shown remarkable progress in computer vision tasks such as image classification.
State of the Art on Neural Rendering
Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e. g., by the integration of differentiable rendering into network training.
3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation
We show that grouping proposals improves over NMS and outperforms previous state-of-the-art methods on the tasks of 3D object detection and semantic instance segmentation on the ScanNetV2 benchmark and the S3DIS dataset.
Ranked #1 on
3D Semantic Instance Segmentation
on ScanNetV2
Learning to Optimize Non-Rigid Tracking
One of the widespread solutions for non-rigid tracking has a nested-loop structure: with Gauss-Newton to minimize a tracking objective in the outer loop, and Preconditioned Conjugate Gradient (PCG) to solve a sparse linear system in the inner loop.
SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans
We present a novel approach to reconstructing lightweight, CAD-based representations of scanned 3D environments from commodity RGB-D sensors.
Modeling 3D Shapes by Reinforcement Learning
We explore how to enable machines to model 3D shapes like human modelers using deep reinforcement learning (RL).
Local Implicit Grid Representations for 3D Scenes
Then, we use the decoder as a component in a shape optimization that solves for a set of latent codes on a regular grid of overlapping crops such that an interpolation of the decoded local shapes matches a partial or noisy observation.
Adversarial Texture Optimization from RGB-D Scans
In this work, we present a novel approach for color texture generation using a conditional adversarial loss obtained from weakly-supervised views.
Image-guided Neural Object Rendering
Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language
We introduce the task of 3D object localization in RGB-D scans using natural language descriptions.
Neural Voice Puppetry: Audio-driven Facial Reenactment
Neural Voice Puppetry has a variety of use-cases, including audio-driven video avatars, video dubbing, and text-driven video synthesis of a talking head.
DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data
Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus.
SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans
We present a novel approach that converts partial and noisy RGB-D scans into high-quality 3D scene reconstructions by inferring unobserved scene geometry.
SpoC: Spoofing Camera Fingerprints
Given a GAN-generated image, we insert the traces of a specific camera model into it and deceive state-of-the-art detectors into believing the image was acquired by that model.
ViewAL: Active Learning with Viewpoint Entropy for Semantic Segmentation
To incorporate this uncertainty measure, we introduce a new viewpoint entropy formulation, which is the basis of our active learning strategy.
Joint Embedding of 3D Scan and CAD Objects
We propose a novel approach to learn a joint embedding space between scan and CAD geometry, where semantically similar objects from both domains lie close together.
RIO: 3D Object Instance Re-Localization in Changing Indoor Environments
In this work, we introduce the task of 3D object instance re-localization (RIO): given one or multiple objects in an RGB-D scan, we want to estimate their corresponding 6DoF poses in another 3D scan of the same environment taken at a later point in time.
End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans
We present a novel, end-to-end approach to align CAD models to an 3D scan of a scene, enabling transformation of a noisy, incomplete 3D scan to a compact, CAD reconstruction with clean, complete object geometry.
Deferred Neural Rendering: Image Synthesis using Neural Textures
Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline.
RevealNet: Seeing Behind Objects in RGB-D Scans
Thus, we introduce the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances and infer their complete object geometry.
Inverse Path Tracing for Joint Material and Lighting Estimation
We introduce Inverse Path Tracing, a novel approach to jointly estimate the material properties of objects and light sources in indoor scenes by using an invertible light transport simulation.
Homogeneous Linear Inequality Constraints for Neural Network Activations
One way to achieve this task is by means of a projection step at test time after unconstrained training.
DDSL: Deep Differentiable Simplex Layer for Learning Geometric Signals
We present a Deep Differentiable Simplex Layer (DDSL) for neural networks for geometric deep learning.
FaceForensics++: Learning to Detect Manipulated Facial Images
In particular, the benchmark is based on DeepFakes, Face2Face, FaceSwap and NeuralTextures as prominent representatives for facial manipulations at random compression level and size.
Ranked #1 on
DeepFake Detection
on FaceForensics
Convolutional Neural Networks on non-uniform geometrical signals using Euclidean spectral transformation
It has been challenging to analyze signals with mixed topologies (for example, point cloud with surface mesh).
RegNet: Learning the Optimization of Direct Image-to-Image Pose Registration
Direct image-to-image alignment that relies on the optimization of photometric error metrics suffers from limited convergence range and sensitivity to lighting conditions.
3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans
We introduce 3D-SIS, a novel neural network architecture for 3D semantic instance segmentation in commodity RGB-D scans.
Ranked #3 on
3D Semantic Instance Segmentation
on ScanNetV2
ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection
We devise a learning-based forensic detector which adapts well to new domains, i. e., novel manipulation methods and can handle scenarios where only a handful of fake examples are available during training.
DeepVoxels: Learning Persistent 3D Feature Embeddings
In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis.
TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes
We introduce, TextureNet, a neural network architecture designed to extract features from high-resolution signals associated with 3D surface meshes (e. g., color texture maps).
Ranked #26 on
Semantic Segmentation
on ScanNet
(test mIoU metric)
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry.
Ranked #1 on
3D Reconstruction
on Scan2CAD
Scan2Mesh: From Unstructured Range Scans to 3D Meshes
We introduce Scan2Mesh, a novel data-driven generative approach which transforms an unstructured and potentially incomplete range scan into a structured 3D mesh representation.
IGNOR: Image-guided Neural Object Rendering
Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.
Deep Video Portraits
In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network -- thus taking full control of the target.
HeadOn: Real-time Reenactment of Human Portrait Videos
We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze.
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation
We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network.
Ranked #1 on
Scene Segmentation
on ScanNet
FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
Research on the detection of face manipulations has been seriously hampered by the lack of adequate datasets.
ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
We introduce ScanComplete, a novel data-driven approach for taking an incomplete 3D scan of a scene as input and predicting a complete 3D model along with per-voxel semantic labels.
IMU2Face: Real-time Gesture-driven Facial Reenactment
We present IMU2Face, a gesture-driven facial reenactment system.
Multiframe Scene Flow with Piecewise Rigid Motion
We introduce a novel multiframe scene flow approach that jointly optimizes the consistency of the patch appearances and their local rigid motions from RGB-D image sequences.
Matterport3D: Learning from RGB-D Data in Indoor Environments
Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms.
Calipso: Physics-based Image and Video Editing through CAD Model Proxies
We present Calipso, an interactive method for editing images and videos in a physically-coherent manner.
Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting
We introduce a novel method to obtain high-quality 3D reconstructions from consumer RGB-D sensors.
Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction
We introduce a new method that efficiently computes a set of viewpoints and trajectories for high-quality 3D reconstructions in outdoor environments.
A Lightweight Approach for On-the-Fly Reflectance Estimation
In addition, we have created a large synthetic dataset, SynBRDF, which comprises a total of $500$K RGBD images rendered with a physically-based ray tracer under a variety of natural illumination, covering $5000$ materials and $5000$ shapes.
ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets.
Ranked #11 on
Semantic Segmentation
on ScanNetV2
Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
Although our 3D-EPN outperforms state-of-the-art completion method, the main contribution in our work lies in the combination of a data-driven shape predictor and analytic 3D shape synthesis.
FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality
Based on reenactment of a prerecorded stereo video of the person without the HMD, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances.
Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging
Many graphics and vision problems can be expressed as non-linear least squares optimizations of objective functions over visual data, such as images and meshes.
BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration
Our approach estimates globally optimized (i. e., bundle adjusted) poses in real-time, supports robust tracking with recovery from gross tracking failures (i. e., relocalization), and re-estimates the 3D model in real-time to ensure global consistency; all within a single framework.
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions.
Ranked #2 on
3D Reconstruction
on Scan2CAD
VolumeDeform: Real-time Volumetric Non-rigid Reconstruction
We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints.
Learning to Navigate the Energy Landscape
We demonstrate the efficacy of our approach on the challenging problem of RGB Camera Relocalization.
