Search Results for author:
Found 107 papers, 55 papers with code
Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors
Current popular backbones in computer vision, such as Vision Transformers (ViT) and ResNets are trained to perceive the world from 2D images.
S4R: Self-Supervised Semantic Scene Reconstruction from RGB-D Scans
To this end, we design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images, fusing cross-domain features into volumetric embeddings to predict complete 3D geometry, color, and semantics.
Panoptic Lifting for 3D Scene Understanding with Neural Fields
We propose Panoptic Lifting, a novel approach for learning panoptic 3D volumetric representations from images of in-the-wild scenes.
SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images
SuRFNet employs partial SRFs from few/one images and a specialized SRF loss to learn to generate high-quality sparse voxel radiance fields that can be rendered from novel views.
Learning Neural Parametric Head Models
We propose a novel 3D morphable model for complete human heads based on hybrid neural fields.
ObjectMatch: Robust Registration using Canonical Object Correspondences
In a pairwise setting, our method improves registration recall of state-of-the-art feature matching from 77% to 87% overall and from 21% to 52% in pairs with 10% or less inter-frame overlap.
DiffRF: Rendering-Guided 3D Radiance Field Diffusion
We introduce DiffRF, a novel approach for 3D radiance field synthesis based on denoising diffusion probabilistic models.
ClipFace: Text-guided Editing of Textured 3D Morphable Models
Controllable editing and manipulation are given by language prompts to adapt texture and expression of the 3D morphable model.
UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding
Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships.
Learning 3D Scene Priors with 2D Supervision
Holistic 3D scene understanding entails estimation of both layout configuration and object geometry in a 3D environment.
Neural Poisson: Indicator Functions for Neural Fields
Implicit neural field generating signed distance field representations (SDFs) of 3D shapes have shown remarkable progress in 3D shape reconstruction and generation.
How to Boost Face Recognition with StyleGAN?
Our self-supervised strategy is the most useful with limited amounts of labeled training data, which can be beneficial for more tailored face recognition tasks and when facing privacy concerns.
Neural Shape Deformation Priors
To this end, we introduce transformer-based deformation networks that represent a shape deformation as a composition of local surface deformations.
The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes
This allows us to improve the generalization of a CNN-based model without any increase in the number of weights at test time.
3D Multi-Object Tracking with Differentiable Pose Estimation
We propose a novel approach for joint 3D multi-object tracking and reconstruction from RGB-D sequences in indoor environments.
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels
A multi-distance interpolation method is proposed and applied on voxel features to model both small and large motions.
3DILG: Irregular Latent Grids for 3D Generative Modeling
All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.
End2End Multi-View Feature Matching using Differentiable Pose Optimization
Training feature matching with gradients from pose optimization naturally learns to down-weight outliers and boosts pose estimation on image pairs compared to SuperGlue by 6. 7% on ScanNet.
AutoRF: Learning 3D Object Radiance Fields from Single View Observations
We introduce AutoRF - a new approach for learning neural 3D object representations where each object in the training set is observed by only a single view.
Audio-Visual Person-of-Interest DeepFake Detection
The aim of this work is to propose a deepfake detector that can cope with the wide variety of manipulation methods and scenarios encountered in the real world.
Texturify: Generating Textures on 3D Shape Surfaces
Texture cues on 3D objects are key to compelling visual representations, with the possibility to create high visual fidelity with inherent spatial consistency across different views.
3D Equivariant Graph Implicit Functions
In this work, we address two key limitations of such representations, in failing to capture local 3D geometric fine details, and to learn from and generalize to shapes with unseen 3D transformations.
RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering
Finding accurate correspondences among different views is the Achilles' heel of unsupervised Multi-View Stereo (MVS).
4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding
We present a new approach to instill 4D dynamic object priors into learned 3D representations by unsupervised pre-training.
Ranked #14 on
3D Instance Segmentation
on ScanNet(v2)
(mAP @ 50 metric)
Dense Depth Priors for Neural Radiance Fields from Sparse Input Views
To this end, we leverage dense depth priors in order to constrain the NeRF optimization.
ROCA: Robust CAD Model Retrieval and Alignment from a Single Image
We present ROCA, a novel end-to-end approach that retrieves and aligns 3D CAD models from a shape database to a single input image.
3D Dense Shape Correspondence
3D Object Detection From Monocular Images
+2
StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions
Style transfer typically operates on 2D images, making stylization of a mesh challenging.
D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding
Our D3Net unifies dense captioning and visual grounding in 3D in a self-critical manner.
Neural Head Avatars from Monocular RGB Videos
We present Neural Head Avatars, a novel neural representation that explicitly models the surface geometry and appearance of an animatable human avatar that can be used for teleconferencing in AR/VR or other applications in the movie or games industry that rely on a digital human.
Pose2Room: Understanding 3D Scenes from Human Activities
To this end, we propose P2R-Net to learn a probabilistic 3D model of the objects in a scene characterized by their class categories and oriented 3D bounding boxes, based on an input observed human trajectory in the environment.
Panoptic 3D Scene Reconstruction From a Single RGB Image
Inspired by 2D panoptic segmentation, we propose to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction - from a single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations.
Egocentric Videoconferencing
Even holding a mobile phone camera in the front of the face while sitting for a long duration is not convenient.
TransformerFusion: Monocular RGB Scene Reconstruction using Transformers
We introduce TransformerFusion, a transformer-based 3D scene reconstruction approach.
4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface
Tracking non-rigidly deforming scenes using range sensors has numerous applications including computer vision, AR/VR, and robotics.
Pri3D: Can 3D Priors Help 2D Representation Learning?
Inspired by these advances in geometric understanding, we aim to imbue image-based perception with representations learned under geometric constraints.
Neural RGB-D Surface Reconstruction
Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR.
Dynamic Surface Function Networks for Clothed Human Bodies
To this end, we explicitly model the surface of the person using a multi-layer perceptron (MLP) which is embedded into the canonical space of the SMPL body model.
NPMs: Neural Parametric Models for 3D Deformable Shapes
Crucially, once learned, our neural parametric models of shape and pose enable optimization over the learned spaces to fit to new observations, similar to the fitting of a traditional parametric model, e. g., SMPL.
RetrievalFuse: Neural 3D Scene Reconstruction with a Database
3D reconstruction of large scenes is a challenging problem due to the high-complexity nature of the solution space, in particular for generative neural networks.
COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning
We propose a self-supervised training strategy where we only need a set of captioned images.
SceneFormer: Indoor Scene Generation with Transformers
In contrast, we do not use any appearance information, and implicitly learn object relations using the self-attention mechanism of transformers.
Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts
The rapid progress in 3D scene understanding has come with growing demand for data; however, collecting and annotating 3D scenes (e. g. point clouds) are notoriously hard.
Ranked #2 on
3D Semantic Segmentation
on ScanNet200
Seeing Behind Objects for 3D Multi-Object Tracking in RGB-D Sequences
From a sequence of RGB-D frames, we detect objects in each frame and learn to predict their complete object geometry as well as a dense correspondence mapping into a canonical space.
Vid2CAD: CAD Model Alignment using Multi-View Constraints from Videos
We address the task of aligning CAD models to a video sequence of a complex scene containing multiple objects.
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction
We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face.
ID-Reveal: Identity-aware DeepFake Video Detection
A major challenge in DeepFake forgery detection is that state-of-the-art algorithms are mostly trained to detect a specific fake method.
Scan2Cap: Context-aware Dense Captioning in RGB-D Scans
We introduce the task of dense captioning in 3D scans from commodity RGB-D sensors.
Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction
We introduce Neural Deformation Graphs for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.
RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction
In this work, we introduce RfD-Net that jointly detects and reconstructs dense object surfaces directly from raw point clouds.
Face2Face: Real-time Face Capture and Reenactment of RGB Videos
Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.
Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition
Neural rendering techniques promise efficient photo-realistic image synthesis while at the same time providing rich control over scene parameters by learning the physical image formation process.
SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans
We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion.
Neural Non-Rigid Tracking
We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction by a learned robust optimization.
Generalized Zero and Few-Shot Transfer for Facial Forgery Detection
We propose Deep Distribution Transfer(DDT), a new transfer learning approach to address the problem of zero and few-shot transfer in the context of facial forgery detection.
Explicitly Modeled Attention Maps for Image Classification
Self-attention networks have shown remarkable progress in computer vision tasks such as image classification.
State of the Art on Neural Rendering
Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e. g., by the integration of differentiable rendering into network training.
3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation
We show that grouping proposals improves over NMS and outperforms previous state-of-the-art methods on the tasks of 3D object detection and semantic instance segmentation on the ScanNetV2 benchmark and the S3DIS dataset.
Ranked #1 on
3D Semantic Instance Segmentation
on ScanNetV2
Learning to Optimize Non-Rigid Tracking
One of the widespread solutions for non-rigid tracking has a nested-loop structure: with Gauss-Newton to minimize a tracking objective in the outer loop, and Preconditioned Conjugate Gradient (PCG) to solve a sparse linear system in the inner loop.
SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans
We present a novel approach to reconstructing lightweight, CAD-based representations of scanned 3D environments from commodity RGB-D sensors.
Modeling 3D Shapes by Reinforcement Learning
We explore how to enable machines to model 3D shapes like human modelers using deep reinforcement learning (RL).
Local Implicit Grid Representations for 3D Scenes
Then, we use the decoder as a component in a shape optimization that solves for a set of latent codes on a regular grid of overlapping crops such that an interpolation of the decoded local shapes matches a partial or noisy observation.
Adversarial Texture Optimization from RGB-D Scans
In this work, we present a novel approach for color texture generation using a conditional adversarial loss obtained from weakly-supervised views.
Image-guided Neural Object Rendering
Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language
We introduce the task of 3D object localization in RGB-D scans using natural language descriptions.
Neural Voice Puppetry: Audio-driven Facial Reenactment
Neural Voice Puppetry has a variety of use-cases, including audio-driven video avatars, video dubbing, and text-driven video synthesis of a talking head.
DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data
Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus.
SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans
We present a novel approach that converts partial and noisy RGB-D scans into high-quality 3D scene reconstructions by inferring unobserved scene geometry.
SpoC: Spoofing Camera Fingerprints
Given a GAN-generated image, we insert the traces of a specific camera model into it and deceive state-of-the-art detectors into believing the image was acquired by that model.
ViewAL: Active Learning with Viewpoint Entropy for Semantic Segmentation
To incorporate this uncertainty measure, we introduce a new viewpoint entropy formulation, which is the basis of our active learning strategy.
Joint Embedding of 3D Scan and CAD Objects
We propose a novel approach to learn a joint embedding space between scan and CAD geometry, where semantically similar objects from both domains lie close together.
RIO: 3D Object Instance Re-Localization in Changing Indoor Environments
In this work, we introduce the task of 3D object instance re-localization (RIO): given one or multiple objects in an RGB-D scan, we want to estimate their corresponding 6DoF poses in another 3D scan of the same environment taken at a later point in time.
End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans
We present a novel, end-to-end approach to align CAD models to an 3D scan of a scene, enabling transformation of a noisy, incomplete 3D scan to a compact, CAD reconstruction with clean, complete object geometry.
Deferred Neural Rendering: Image Synthesis using Neural Textures
Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline.
RevealNet: Seeing Behind Objects in RGB-D Scans
Thus, we introduce the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances and infer their complete object geometry.
Inverse Path Tracing for Joint Material and Lighting Estimation
We introduce Inverse Path Tracing, a novel approach to jointly estimate the material properties of objects and light sources in indoor scenes by using an invertible light transport simulation.
Homogeneous Linear Inequality Constraints for Neural Network Activations
One way to achieve this task is by means of a projection step at test time after unconstrained training.
DDSL: Deep Differentiable Simplex Layer for Learning Geometric Signals
We present a Deep Differentiable Simplex Layer (DDSL) for neural networks for geometric deep learning.
FaceForensics++: Learning to Detect Manipulated Facial Images
In particular, the benchmark is based on DeepFakes, Face2Face, FaceSwap and NeuralTextures as prominent representatives for facial manipulations at random compression level and size.
Ranked #1 on
DeepFake Detection
on FaceForensics
Convolutional Neural Networks on non-uniform geometrical signals using Euclidean spectral transformation
It has been challenging to analyze signals with mixed topologies (for example, point cloud with surface mesh).
RegNet: Learning the Optimization of Direct Image-to-Image Pose Registration
Direct image-to-image alignment that relies on the optimization of photometric error metrics suffers from limited convergence range and sensitivity to lighting conditions.
3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans
We introduce 3D-SIS, a novel neural network architecture for 3D semantic instance segmentation in commodity RGB-D scans.
Ranked #3 on
3D Semantic Instance Segmentation
on ScanNetV2
ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection
We devise a learning-based forensic detector which adapts well to new domains, i. e., novel manipulation methods and can handle scenarios where only a handful of fake examples are available during training.
DeepVoxels: Learning Persistent 3D Feature Embeddings
In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis.
TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes
We introduce, TextureNet, a neural network architecture designed to extract features from high-resolution signals associated with 3D surface meshes (e. g., color texture maps).
Ranked #12 on
Semantic Segmentation
on ScanNet
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry.
Ranked #1 on
3D Reconstruction
on Scan2CAD
IGNOR: Image-guided Neural Object Rendering
Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.
Scan2Mesh: From Unstructured Range Scans to 3D Meshes
We introduce Scan2Mesh, a novel data-driven generative approach which transforms an unstructured and potentially incomplete range scan into a structured 3D mesh representation.
Deep Video Portraits
In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network -- thus taking full control of the target.
HeadOn: Real-time Reenactment of Human Portrait Videos
We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze.
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation
We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network.
Ranked #1 on
Scene Segmentation
on ScanNet
FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
Research on the detection of face manipulations has been seriously hampered by the lack of adequate datasets.
ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
We introduce ScanComplete, a novel data-driven approach for taking an incomplete 3D scan of a scene as input and predicting a complete 3D model along with per-voxel semantic labels.
IMU2Face: Real-time Gesture-driven Facial Reenactment
We present IMU2Face, a gesture-driven facial reenactment system.
Multiframe Scene Flow with Piecewise Rigid Motion
We introduce a novel multiframe scene flow approach that jointly optimizes the consistency of the patch appearances and their local rigid motions from RGB-D image sequences.
Matterport3D: Learning from RGB-D Data in Indoor Environments
Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms.
Calipso: Physics-based Image and Video Editing through CAD Model Proxies
We present Calipso, an interactive method for editing images and videos in a physically-coherent manner.
Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting
We introduce a novel method to obtain high-quality 3D reconstructions from consumer RGB-D sensors.
Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction
We introduce a new method that efficiently computes a set of viewpoints and trajectories for high-quality 3D reconstructions in outdoor environments.
A Lightweight Approach for On-the-Fly Reflectance Estimation
In addition, we have created a large synthetic dataset, SynBRDF, which comprises a total of $500$K RGBD images rendered with a physically-based ray tracer under a variety of natural illumination, covering $5000$ materials and $5000$ shapes.
ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets.
Ranked #9 on
Semantic Segmentation
on ScanNetV2
Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
Although our 3D-EPN outperforms state-of-the-art completion method, the main contribution in our work lies in the combination of a data-driven shape predictor and analytic 3D shape synthesis.
FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality
Based on reenactment of a prerecorded stereo video of the person without the HMD, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances.
Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging
Many graphics and vision problems can be expressed as non-linear least squares optimizations of objective functions over visual data, such as images and meshes.
BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration
Our approach estimates globally optimized (i. e., bundle adjusted) poses in real-time, supports robust tracking with recovery from gross tracking failures (i. e., relocalization), and re-estimates the 3D model in real-time to ensure global consistency; all within a single framework.
VolumeDeform: Real-time Volumetric Non-rigid Reconstruction
We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints.
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions.
Ranked #2 on
3D Reconstruction
on Scan2CAD
Learning to Navigate the Energy Landscape
We demonstrate the efficacy of our approach on the challenging problem of RGB Camera Relocalization.
