Search Results for author: Marc Pollefeys

Found 183 papers, 76 papers with code

Handcrafted Outlier Detection Revisited

1 code implementation ECCV 2020 Luca Cavalli, Viktor Larsson, Martin Ralf Oswald, Torsten Sattler, Marc Pollefeys

As a result, outlier detection is a fundamental problem in computer vision and a wide range of approaches, from simple checks based on descriptor similarity to geometric verification, have been proposed over the last decades.

Outlier Detection Visual Localization

Calibration-free Structure-from-Motion with Calibrated Radial Trifocal Tensors

no code implementations ECCV 2020 Viktor Larsson, Nicolas Zobernig, Kasim Taskin, Marc Pollefeys

In this paper we consider the problem of Structure-from-Motion from images with unknown intrinsic calibration.

CaSAR: Contact-aware Skeletal Action Recognition

no code implementations17 Sep 2023 Junan Lin, Zhichao Sun, Enjie Cao, Taein Kwon, Mahdi Rad, Marc Pollefeys

Skeletal Action recognition from an egocentric view is important for applications such as interfaces in AR/VR glasses and human-robot interaction, where the device has limited resources.

Action Recognition

Learning Disentangled Avatars with Hybrid 3D Representations

no code implementations12 Sep 2023 Yao Feng, Weiyang Liu, Timo Bolkart, Jinlong Yang, Marc Pollefeys, Michael J. Black

Towards this end, both explicit and implicit 3D representations are heavily studied for a holistic modeling and capture of the whole human (e. g., body, clothing, face and hair), but neither representation is an optimal choice in terms of representation efficacy since different parts of the human avatar have different modeling desiderata.


ResFields: Residual Neural Fields for Spatiotemporal Signals

1 code implementation6 Sep 2023 Marko Mihajlovic, Sergey Prokudin, Marc Pollefeys, Siyu Tang

Neural fields, a category of neural networks trained to represent high-frequency signals, have gained significant attention in recent years due to their impressive performance in modeling complex 3D data, especially large neural signed distance (SDFs) or radiance fields (NeRFs) via a single multi-layer perceptron (MLP).

4D reconstruction Neural Rendering

Vanishing Point Estimation in Uncalibrated Images with Prior Gravity Direction

1 code implementation21 Aug 2023 Rémi Pautrat, Shaohui Liu, Petr Hruby, Marc Pollefeys, Daniel Barath

We tackle the problem of estimating a Manhattan frame, i. e. three orthogonal vanishing points, and the unknown focal length of the camera, leveraging a prior vertical direction.

RLSAC: Reinforcement Learning enhanced Sample Consensus for End-to-End Robust Estimation

1 code implementation10 Aug 2023 Chang Nie, Guangming Wang, Zhe Liu, Luca Cavalli, Marc Pollefeys, Hesheng Wang

Therefore, RLSAC can avoid differentiating to learn the features and the feedback of downstream tasks for end-to-end robust estimation.


Quantification of Predictive Uncertainty via Inference-Time Sampling

no code implementations3 Aug 2023 Katarína Tóthová, Ľubor Ladický, Daniel Thul, Marc Pollefeys, Ender Konukoglu

Predictive variability due to data ambiguities has typically been addressed via construction of dedicated models with built-in probabilistic capabilities that are trained to predict uncertainty estimates as variables of interest.

AffineGlue: Joint Matching and Robust Estimation

no code implementations28 Jul 2023 Daniel Barath, Dmytro Mishkin, Luca Cavalli, Paul-Edouard Sarlin, Petr Hruby, Marc Pollefeys

Moreover, we derive a new minimal solver for homography estimation, requiring only a single affine correspondence (AC) and a gravity prior.

Homography Estimation

Consensus-Adaptive RANSAC

1 code implementation26 Jul 2023 Luca Cavalli, Daniel Barath, Marc Pollefeys, Viktor Larsson

The proposed attention mechanism and one-step transformer provide an adaptive behavior that enhances the performance of RANSAC, making it a more effective tool for robust estimation.

Lazy Visual Localization via Motion Averaging

no code implementations19 Jul 2023 Siyan Dong, Shaohui Liu, Hengkai Guo, Baoquan Chen, Marc Pollefeys

Visual (re)localization is critical for various applications in computer vision and robotics.

Visual Localization

OpenMask3D: Open-Vocabulary 3D Instance Segmentation

no code implementations23 Jun 2023 Ayça Takmaz, Elisabetta Fedele, Robert W. Sumner, Marc Pollefeys, Federico Tombari, Francis Engelmann

In this work, we address this limitation, and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D instance segmentation.

3D Instance Segmentation Scene Understanding +1

LightGlue: Local Feature Matching at Light Speed

2 code implementations23 Jun 2023 Philipp Lindenberger, Paul-Edouard Sarlin, Marc Pollefeys

We introduce LightGlue, a deep neural network that learns to match local features across images.

3D Reconstruction Homography Estimation +3

SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding

no code implementations8 Jun 2023 Paul-Edouard Sarlin, Eduard Trulls, Marc Pollefeys, Jan Hosang, Simon Lynen

Semantic 2D maps are commonly used by humans and machines for navigation purposes, whether it's walking or driving.

Scene Understanding

Next-generation Surgical Navigation: Multi-view Marker-less 6DoF Pose Estimation of Surgical Instruments

no code implementations5 May 2023 Jonas Hein, Nicola Cavalcanti, Daniel Suter, Lukas Zingg, Fabio Carrillo, Mazda Farshad, Marc Pollefeys, Nassir Navab, Philipp Fürnstahl

A particular focus in computer-assisted surgery is to replace marker-based tracking systems for instrument localization with pure image-based 6DoF pose estimation.

Pose Estimation

Learning-based Relational Object Matching Across Views

no code implementations3 May 2023 Cathrin Elich, Iro Armeni, Martin R. Oswald, Marc Pollefeys, Joerg Stueckler

Our approach compares favorably to previous state-of-the-art object-level matching approaches and achieves improved performance over a pure keypoint-based approach for large view-point changes.

Image Retrieval Retrieval +1

SGAligner : 3D Scene Alignment with Scene Graphs

1 code implementation28 Apr 2023 Sayan Deb Sarkar, Ondrej Miksik, Marc Pollefeys, Daniel Barath, Iro Armeni

We propose SGAligner, the first method for aligning pairs of 3D scene graphs that is robust to in-the-wild scenarios (ie, unknown overlap -- if any -- and changes in the environment).

3D Scene Graph Alignment Contrastive Learning +2

Tracking by 3D Model Estimation of Unknown Objects in Videos

no code implementations13 Apr 2023 Denys Rozumnyi, Jiri Matas, Marc Pollefeys, Vittorio Ferrari, Martin R. Oswald

We argue that this representation is limited and instead propose to guide and improve 2D tracking with an explicit object representation, namely the textured 3D shape and 6DoF pose in each video frame.

Visual Object Tracking

Human from Blur: Human Pose Tracking from Blurry Images

no code implementations30 Mar 2023 Yiming Zhao, Denys Rozumnyi, Jie Song, Otmar Hilliges, Marc Pollefeys, Martin R. Oswald

The key idea is to tackle the inverse problem of image deblurring by modeling the forward problem with a 3D human model, a texture map, and a sequence of poses to describe human motion.

Deblurring Image Deblurring +2

3D Line Mapping Revisited

1 code implementation CVPR 2023 Shaohui Liu, Yifan Yu, Rémi Pautrat, Marc Pollefeys, Viktor Larsson

In contrast to sparse keypoints, a handful of line segments can concisely encode the high-level scene layout, as they often delineate the main structural elements.

Visual Localization

RegFormer: An Efficient Projection-Aware Transformer Network for Large-Scale Point Cloud Registration

1 code implementation22 Mar 2023 Jiuming Liu, Guangming Wang, Zhe Liu, Chaokang Jiang, Marc Pollefeys, Hesheng Wang

Specifically, a projection-aware hierarchical transformer is proposed to capture long-range dependencies and filter outliers by extracting point features globally.

Point Cloud Registration

NICER-SLAM: Neural Implicit Scene Encoding for RGB SLAM

no code implementations7 Feb 2023 Zihan Zhu, Songyou Peng, Viktor Larsson, Zhaopeng Cui, Martin R. Oswald, Andreas Geiger, Marc Pollefeys

Neural implicit representations have recently become popular in simultaneous localization and mapping (SLAM), especially in dense visual SLAM.

3D Scene Reconstruction Novel View Synthesis +2

DeepLSD: Line Segment Detection and Refinement with Deep Image Gradients

1 code implementation CVPR 2023 Rémi Pautrat, Daniel Barath, Viktor Larsson, Martin R. Oswald, Marc Pollefeys

Their learned counterparts are more repeatable and can handle challenging images, but at the cost of a lower accuracy and a bias towards wireframe lines.

Line Detection Line Segment Detection

VolRecon: Volume Rendering of Signed Ray Distance Functions for Generalizable Multi-View Reconstruction

1 code implementation CVPR 2023 Yufan Ren, Fangjinhua Wang, Tong Zhang, Marc Pollefeys, Sabine Süsstrunk

The success of the Neural Radiance Fields (NeRF) in novel view synthesis has inspired researchers to propose neural implicit scene reconstruction.

Novel View Synthesis

LaMAR: Benchmarking Localization and Mapping for Augmented Reality

no code implementations19 Oct 2022 Paul-Edouard Sarlin, Mihai Dusmanu, Johannes L. Schönberger, Pablo Speciale, Lukas Gruber, Viktor Larsson, Ondrej Miksik, Marc Pollefeys

To close this gap, we introduce LaMAR, a new benchmark with a comprehensive capture and GT pipeline that co-registers realistic trajectories and sensor streams captured by heterogeneous AR devices in large, unconstrained scenes.


NeuralMeshing: Differentiable Meshing of Implicit Neural Representations

no code implementations5 Oct 2022 Mathias Vetsch, Sandro Lombardi, Marc Pollefeys, Martin R. Oswald

The generation of triangle meshes from point clouds, i. e. meshing, is a core task in computer graphics and computer vision.

Capturing and Animation of Body and Clothing from Monocular Video

1 code implementation4 Oct 2022 Yao Feng, Jinlong Yang, Marc Pollefeys, Michael J. Black, Timo Bolkart

Building on this insight, we propose SCARF (Segmented Clothed Avatar Radiance Field), a hybrid model combining a mesh-based body with a neural radiance field.

Virtual Try-on

IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis

1 code implementation2 Oct 2022 Weicai Ye, Shuo Chen, Chong Bao, Hujun Bao, Marc Pollefeys, Zhaopeng Cui, Guofeng Zhang

Existing inverse rendering combined with neural rendering methods can only perform editable novel view synthesis on object-specific scenes, while we present intrinsic neural radiance fields, dubbed IntrinsicNeRF, which introduce intrinsic decomposition into the NeRF-based neural rendering method and can extend its application to room-scale scenes.

Clustering Inverse Rendering +2

Learning-Based Dimensionality Reduction for Computing Compact and Effective Local Feature Descriptors

1 code implementation27 Sep 2022 Hao Dong, Xieyuanli Chen, Mihai Dusmanu, Viktor Larsson, Marc Pollefeys, Cyrill Stachniss

A distinctive representation of image patches in form of features is a key component of many computer vision and robotics tasks, such as image matching, image retrieval, and visual localization.

Dimensionality Reduction Image Retrieval +2

Visual Localization via Few-Shot Scene Region Classification

1 code implementation14 Aug 2022 Siyan Dong, Shuzhe Wang, Yixin Zhuang, Juho Kannala, Marc Pollefeys, Baoquan Chen

Visual (re)localization addresses the problem of estimating the 6-DoF (Degree of Freedom) camera pose of a query image captured in a known scene, which is a key building block of many computer vision and robotics applications.

General Classification Memorization +2

CompNVS: Novel View Synthesis with Scene Completion

no code implementations23 Jul 2022 Zuoyue Li, Tianxing Fan, Zhenqiang Li, Zhaopeng Cui, Yoichi Sato, Marc Pollefeys, Martin R. Oswald

We introduce a scalable framework for novel view synthesis from RGB-D images with largely incomplete scene coverage.

Novel View Synthesis Scene Understanding

NeFSAC: Neurally Filtered Minimal Samples

1 code implementation16 Jul 2022 Luca Cavalli, Marc Pollefeys, Daniel Barath

We tested NeFSAC on more than 100k image pairs from three publicly available real-world datasets and found that it leads to one order of magnitude speed-up, while often finding more accurate results than USAC alone.

Autonomous Driving Pose Estimation

Context-Aware Sequence Alignment using 4D Skeletal Augmentation

1 code implementation CVPR 2022 Taein Kwon, Bugra Tekin, Siyu Tang, Marc Pollefeys

Temporal alignment of fine-grained human actions in videos is important for numerous applications in computer vision, robotics, and mixed reality.

Hand Pose Estimation Mixed Reality +1

Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together

no code implementations3 Feb 2022 Jeffrey Delmerico, Roi Poranne, Federica Bogo, Helen Oleynikova, Eric Vollenweider, Stelian Coros, Juan Nieto, Marc Pollefeys

Spatial computing -- the ability of devices to be aware of their surroundings and to represent this digitally -- offers novel capabilities in human-robot interaction.

Mixed Reality

Camera Pose Estimation Using Implicit Distortion Models

no code implementations CVPR 2022 Linfei Pan, Marc Pollefeys, Viktor Larsson

Low-dimensional parametric models are the de-facto standard in computer vision for intrinsic camera calibration.

Camera Calibration Pose Estimation +1

Learning To Find Good Models in RANSAC

no code implementations CVPR 2022 Daniel Barath, Luca Cavalli, Marc Pollefeys

We propose the Model Quality Network, MQ-Net in short, for predicting the quality, e. g. the pose error of essential matrices, of models generated inside RANSAC.

Pose Estimation

NICE-SLAM: Neural Implicit Scalable Encoding for SLAM

1 code implementation CVPR 2022 Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, Marc Pollefeys

Neural implicit representations have recently shown encouraging results in various domains, including promising progress in simultaneous localization and mapping (SLAM).

Simultaneous Localization and Mapping

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

1 code implementation14 Dec 2021 Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, Siyu Tang

Key to reasoning about interactions is to understand the body pose and motion of the interaction partner from the egocentric view.

Motion Estimation

LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies

no code implementations30 Nov 2021 Sandro Lombardi, Bangbang Yang, Tianxing Fan, Hujun Bao, Guofeng Zhang, Marc Pollefeys, Zhaopeng Cui

In this work, we propose a novel neural implicit representation for the human body, which is fully differentiable and optimizable with disentangled shape and pose latent spaces.

3D Reconstruction motion retargeting +1

Learning to Align Sequential Actions in the Wild

no code implementations CVPR 2022 Weizhe Liu, Bugra Tekin, Huseyin Coskun, Vibhav Vineet, Pascal Fua, Marc Pollefeys

To this end, we propose an approach to enforce temporal priors on the optimal transport matrix, which leverages temporal consistency, while allowing for variations in the order of actions.

Representation Learning

Non-local Recurrent Regularization Networks for Multi-view Stereo

no code implementations13 Oct 2021 Qingshan Xu, Martin R. Oswald, Wenbing Tao, Marc Pollefeys, Zhaopeng Cui

However, existing recurrent methods only model the local dependencies in the depth domain, which greatly limits the capability of capturing the global scene context along the depth dimension.

Depth Estimation

Reconstructing and grounding narrated instructional videos in 3D

no code implementations9 Sep 2021 Dimitri Zhukov, Ignacio Rocco, Ivan Laptev, Josef Sivic, Johannes L. Schönberger, Bugra Tekin, Marc Pollefeys

Contrary to the standard scenario of instance-level 3D reconstruction, where identical objects or scenes are present in all views, objects in different instructional videos may have large appearance variations given varying conditions and versions of the same product.

3D Reconstruction

Learning Motion Priors for 4D Human Body Capture in 3D Scenes

no code implementations ICCV 2021 Siwei Zhang, Yan Zhang, Federica Bogo, Marc Pollefeys, Siyu Tang

To prove the effectiveness of the proposed motion priors, we combine them into a novel pipeline for 4D human body capture in 3D scenes.


Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects

1 code implementation NeurIPS 2021 Denys Rozumnyi, Martin R. Oswald, Vittorio Ferrari, Marc Pollefeys

We address the novel task of jointly reconstructing the 3D shape, texture, and motion of an object from a single motion-blurred image.

Deblurring Super-Resolution +1

Shape As Points: A Differentiable Poisson Solver

1 code implementation NeurIPS 2021 Songyou Peng, Chiyu "Max" Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, Andreas Geiger

However, the implicit nature of neural implicit representations results in slow inference time and requires careful initialization.

3D Reconstruction Surface Reconstruction

Towards Efficient Graph Convolutional Networks for Point Cloud Handling

no code implementations ICCV 2021 Yawei Li, He Chen, Zhaopeng Cui, Radu Timofte, Marc Pollefeys, Gregory Chirikjian, Luc van Gool

In this paper, we aim at improving the computational efficiency of graph convolutional networks (GCNs) for learning on point clouds.

Back to the Feature: Learning Robust Camera Localization from Pixels to Pose

1 code implementation CVPR 2021 Paul-Edouard Sarlin, Ajaykumar Unagar, Måns Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, Torsten Sattler

In this paper, we go Back to the Feature: we argue that deep networks should focus on learning robust and invariant visual features, while the geometric estimation should be left to principled algorithms.

Camera Localization Metric Learning +1

Holistic 3D Scene Understanding from a Single Image with Implicit Representation

1 code implementation CVPR 2021 Cheng Zhang, Zhaopeng Cui, yinda zhang, Bing Zeng, Marc Pollefeys, Shuaicheng Liu

We not only propose an image-based local structured implicit network to improve the object shape estimation, but also refine the 3D object pose and scene layout via a novel implicit scene graph neural network that exploits the implicit local object features.

3D Shape Reconstruction Monocular 3D Object Detection +3

Localizing Unsynchronized Sensors with Unknown Sources

1 code implementation6 Feb 2021 Dalia El Badawy, Viktor Larsson, Marc Pollefeys, Ivan Dokmanić

We look at the general case where neither the emission times of the sources nor the reference time frames of the receivers are known.

The Card Shuffling Hypotheses: Building a Time and Memory Efficient Graph Convolutional Network

no code implementations1 Jan 2021 Yawei Li, He Chen, Zhaopeng Cui, Radu Timofte, Marc Pollefeys, Gregory Chirikjian, Luc van Gool

State-of-the-art GCNs adopt $K$-nearest neighbor (KNN) searches for local feature aggregation and feature extraction operations from layer to layer.

3D Classification Point Cloud Classification +1

Orthographic-Perspective Epipolar Geometry

no code implementations ICCV 2021 Viktor Larsson, Marc Pollefeys, Magnus Oskarsson

In this paper we consider the epipolar geometry between orthographic and perspective cameras.

Camera Calibration

DeepSurfels: Learning Online Appearance Fusion

1 code implementation CVPR 2021 Marko Mihajlovic, Silvan Weder, Marc Pollefeys, Martin R. Oswald

We present DeepSurfels, a novel hybrid scene representation for geometry and appearance information.

CodeVIO: Visual-Inertial Odometry with Learned Optimizable Dense Depth

no code implementations18 Dec 2020 Xingxing Zuo, Nathaniel Merrill, Wei Li, Yong liu, Marc Pollefeys, Guoquan Huang

In this work, we present a lightweight, tightly-coupled deep depth network and visual-inertial odometry (VIO) system, which can provide accurate state estimates and dense depth maps of the immediate surroundings.

Depth Estimation Depth Prediction +1

FMODetect: Robust Detection of Fast Moving Objects

1 code implementation ICCV 2021 Denys Rozumnyi, Jiri Matas, Filip Sroubek, Marc Pollefeys, Martin R. Oswald

Compared to other methods, such as deblatting, the inference is of several orders of magnitude faster and allows applications such as real-time fast moving object detection and retrieval in large video collections.

Deblurring Image Matting +3

Sat2Vid: Street-view Panoramic Video Synthesis from a Single Satellite Image

no code implementations ICCV 2021 Zuoyue Li, Zhenqiang Li, Zhaopeng Cui, Rongjun Qin, Marc Pollefeys, Martin R. Oswald

For geometrical and temporal consistency, our approach explicitly creates a 3D point cloud representation of the scene and maintains dense 3D-2D correspondences across frames that reflect the geometric scene configuration inferred from the satellite view.

Image Generation

DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion

1 code implementation CVPR 2021 Arda Düzçeker, Silvano Galliani, Christoph Vogel, Pablo Speciale, Mihai Dusmanu, Marc Pollefeys

We propose an online multi-view depth prediction approach on posed video streams, where the scene geometry information computed in the previous time steps is propagated to the current time step in an efficient and geometrically plausible way.

Depth Estimation Depth Prediction

Cross-Descriptor Visual Localization and Mapping

1 code implementation ICCV 2021 Mihai Dusmanu, Ondrej Miksik, Johannes L. Schönberger, Marc Pollefeys

Visual localization and mapping is the key technology underlying the majority of mixed reality and robotics systems.

Mixed Reality Visual Localization

DeFMO: Deblurring and Shape Recovery of Fast Moving Objects

5 code implementations CVPR 2021 Denys Rozumnyi, Martin R. Oswald, Vittorio Ferrari, Jiri Matas, Marc Pollefeys

We propose a method that, given a single image with its estimated background, outputs the object's appearance and position in a series of sub-frames as if captured by a high-speed camera (i. e. temporal super-resolution).

Deblurring Object Tracking +1

NeuralFusion: Online Depth Fusion in Latent Space

1 code implementation CVPR 2021 Silvan Weder, Johannes L. Schönberger, Marc Pollefeys, Martin R. Oswald

We present a novel online depth map fusion approach that learns depth map aggregation in a latent feature space.

Freetures: Localization in Signed Distance Function Maps

no code implementations19 Oct 2020 Alexander Millane, Helen Oleynikova, Christian Lanegger, Jeff Delmerico, Juan Nieto, Roland Siegwart, Marc Pollefeys, Cesar Cadena

Localization of a robotic system within a previously mapped environment is important for reducing estimation drift and for reusing previously built maps.


Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors

no code implementations8 Oct 2020 Cathrin Elich, Martin R. Oswald, Marc Pollefeys, Joerg Stueckler

Our approach learns to decompose images of synthetic scenes with multiple objects on a planar surface into its constituent scene objects and to infer their 3D properties from a single view.

Decision Making Scene Understanding +1

Self-Supervised Learning of Non-Rigid Residual Flow and Ego-Motion

no code implementations22 Sep 2020 Ivan Tishchenko, Sandro Lombardi, Martin R. Oswald, Marc Pollefeys

Most of the current scene flow methods choose to model scene flow as a per point translation vector without differentiating between static and dynamic components of 3D motion.

Self-Supervised Learning Translation

HoloLens 2 Research Mode as a Tool for Computer Vision Research

1 code implementation25 Aug 2020 Dorin Ungureanu, Federica Bogo, Silvano Galliani, Pooja Sama, Xin Duan, Casey Meekhof, Jan Stühmer, Thomas J. Cashman, Bugra Tekin, Johannes L. Schönberger, Pawel Olszta, Marc Pollefeys

Mixed reality headsets, such as the Microsoft HoloLens 2, are powerful sensing devices with integrated compute capabilities, which makes it an ideal platform for computer vision research.

Mixed Reality

LIC-Fusion 2.0: LiDAR-Inertial-Camera Odometry with Sliding-Window Plane-Feature Tracking

no code implementations17 Aug 2020 Xingxing Zuo, Yulin Yang, Patrick Geneva, Jiajun Lv, Yong liu, Guoquan Huang, Marc Pollefeys

Only the tracked planar points belonging to the same plane will be used for plane initialization, which makes the plane extraction efficient and robust.


KAPLAN: A 3D Point Descriptor for Shape Completion

no code implementations31 Jul 2020 Audrey Richard, Ian Cherabier, Martin R. Oswald, Marc Pollefeys, Konrad Schindler

We present a novel 3D shape completion method that operates directly on unstructured point clouds, thus avoiding resource-intensive data structures like voxel grids.

3D Shape Reconstruction

Infrastructure-based Multi-Camera Calibration using Radial Projections

1 code implementation ECCV 2020 Yukai Lin, Viktor Larsson, Marcel Geppert, Zuzana Kukelova, Marc Pollefeys, Torsten Sattler

In particular, our approach is more robust than the naive approach of first estimating intrinsic parameters and pose per camera before refining the extrinsic parameters of the system.

Camera Calibration Self-Driving Cars +1

AdaLAM: Revisiting Handcrafted Outlier Detection

3 code implementations7 Jun 2020 Luca Cavalli, Viktor Larsson, Martin Ralf Oswald, Torsten Sattler, Marc Pollefeys

Local feature matching is a critical component of many computer vision pipelines, including among others Structure-from-Motion, SLAM, and Visual Localization.

Key Point Matching Outlier Detection +1

OmniSLAM: Omnidirectional Localization and Dense Mapping for Wide-baseline Multi-camera Systems

no code implementations18 Mar 2020 Changhee Won, Hochang Seok, Zhaopeng Cui, Marc Pollefeys, Jongwoo Lim

In this paper, we present an omnidirectional localization and dense mapping system for a wide-baseline multiview stereo setup with ultra-wide field-of-view (FOV) fisheye cameras, which has a 360 degrees coverage of stereo observations of the environment.

Depth Estimation Visual Odometry

Multi-View Optimization of Local Feature Geometry

1 code implementation ECCV 2020 Mihai Dusmanu, Johannes L. Schönberger, Marc Pollefeys

In this work, we address the problem of refining the geometry of local image features from multiple views without known scene or camera geometry.

Camera Localization

Convolutional Occupancy Networks

6 code implementations ECCV 2020 Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, Andreas Geiger

Recently, implicit neural representations have gained popularity for learning-based 3D reconstruction.

3D Reconstruction

Self-Supervised Linear Motion Deblurring

1 code implementation10 Feb 2020 Peidong Liu, Joel Janai, Marc Pollefeys, Torsten Sattler, Andreas Geiger

Motion blurry images challenge many computer vision algorithms, e. g, feature detection, motion estimation, or object recognition.

Deblurring Image Deblurring +4

Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation

2 code implementations17 Jan 2020 Lucas Teixeira, Martin R. Oswald, Marc Pollefeys, Margarita Chli

In this paper, we propose a depth completion and uncertainty estimation approach that better handles the challenges of aerial platforms, such as large viewpoint and depth variations, and limited computing resources.

Depth Completion Monocular Depth Estimation +1

Learned Multi-View Texture Super-Resolution

no code implementations14 Jan 2020 Audrey Richard, Ian Cherabier, Martin R. Oswald, Vagia Tsiminaki, Marc Pollefeys, Konrad Schindler

We present a super-resolution method capable of creating a high-resolution texture map for a virtual 3D object from a set of lower-resolution images of that object.

Image Super-Resolution

Reflection Separation using a Pair of Unpolarized and Polarized Images

1 code implementation NeurIPS 2019 Youwei Lyu, Zhaopeng Cui, Si Li, Marc Pollefeys, Boxin Shi

When we take photos through glass windows or doors, the transmitted background scene is often blended with undesirable reflection.

DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing

1 code implementation CVPR 2020 Shaohui Liu, yinda zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, Zhaopeng Cui

We propose a differentiable sphere tracing algorithm to bridge the gap between inverse graphics methods and the recently proposed deep learning based implicit signed distance function.

Slanted Stixels: A way to represent steep streets

no code implementations2 Oct 2019 Daniel Hernandez-Juarez, Lukas Schneider, Pau Cebrian, Antonio Espinosa, David Vazquez, Antonio M. Lopez, Uwe Franke, Marc Pollefeys, Juan C. Moure

This work presents and evaluates a novel compact scene representation based on Stixels that infers geometric and semantic information.

Learned Semantic Multi-Sensor Depth Map Fusion

no code implementations2 Sep 2019 Denys Rozumnyi, Ian Cherabier, Marc Pollefeys, Martin R. Oswald

Our method learns sensor or algorithm properties jointly with semantic depth fusion and scene completion and can also be used as an expert system, e. g. to unify the strengths of various photometric stereo algorithms.

3D Reconstruction Denoising

To Learn or Not to Learn: Visual Localization from Essential Matrices

1 code implementation4 Aug 2019 Qunjie Zhou, Torsten Sattler, Marc Pollefeys, Laura Leal-Taixe

Using a classical feature-based approach within this framework, we show state-of-the-art performance.

Mixed Reality Pose Estimation +2

3D Appearance Super-Resolution with Deep Learning

1 code implementation CVPR 2019 Yawei Li, Vagia Tsiminaki, Radu Timofte, Marc Pollefeys, Luc van Gool

Experimental results demonstrate that our proposed networks successfully incorporate the 3D geometric information and super-resolve the texture maps.


Robust Dense Mapping for Large-Scale Dynamic Environments

no code implementations7 May 2019 Ioan Andrei Bârsan, Peidong Liu, Marc Pollefeys, Andreas Geiger

We use both instance-aware semantic segmentation and sparse scene flow to classify objects as either background, moving, or potentially moving, thereby ensuring that the system is able to model objects with the potential to transition from static to dynamic, such as parked cars.

Semantic Segmentation Visual Odometry

H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions

1 code implementation CVPR 2019 Bugra Tekin, Federica Bogo, Marc Pollefeys

Given a single RGB image, our model jointly estimates the 3D hand and object poses, models their interactions, and recognizes the object and action classes with a single feed-forward pass through a neural network.

Understanding the Limitations of CNN-based Absolute Camera Pose Regression

1 code implementation CVPR 2019 Torsten Sattler, Qunjie Zhou, Marc Pollefeys, Laura Leal-Taixe

We furthermore use our model to show that pose regression is more closely related to pose approximation via image retrieval than to accurate pose estimation via 3D structure.

Image Retrieval Mixed Reality +5

Privacy Preserving Image-Based Localization

no code implementations CVPR 2019 Pablo Speciale, Johannes L. Schönberger, Sing Bing Kang, Sudipta N. Sinha, Marc Pollefeys

Current localization systems rely on the persistent storage of 3D point clouds of the scene to enable camera pose estimation, but such data reveals potentially sensitive scene information.

Image-Based Localization Mixed Reality +2

Incremental Visual-Inertial 3D Mesh Generation with Structural Regularities

4 code implementations4 Mar 2019 Antoni Rosinol, Torsten Sattler, Marc Pollefeys, Luca Carlone

We propose instead to tightly couple mesh regularization and state estimation by detecting and enforcing structural regularities in a novel factor-graph formulation.

3D Reconstruction Simultaneous Localization and Mapping

Episodic Curiosity through Reachability

1 code implementation ICLR 2019 Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, Sylvain Gelly

One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning.

SurfelMeshing: Online Surfel-Based Mesh Reconstruction

1 code implementation1 Oct 2018 Thomas Schöps, Torsten Sattler, Marc Pollefeys

In contrast to most existing approaches, we do not fuse depth measurements in a volume but in a dense surfel cloud.

Night-to-Day Image Translation for Retrieval-based Localization

1 code implementation26 Sep 2018 Asha Anoosheh, Torsten Sattler, Radu Timofte, Marc Pollefeys, Luc van Gool

We then compare the daytime and translated night images to obtain a pose estimate for the night image using the known 6-DOF position of the closest day image.

Image Retrieval Retrieval +3

Efficient 2D-3D Matching for Multi-Camera Visual Localization

no code implementations17 Sep 2018 Marcel Geppert, Peidong Liu, Zhaopeng Cui, Marc Pollefeys, Torsten Sattler

This results in a system that provides reliable and drift-less pose estimations for high speed autonomous driving.


A Dataset of Flash and Ambient Illumination Pairs from the Crowd

no code implementations ECCV 2018 Yagiz Aksoy, Changil Kim, Petr Kellnhofer, Sylvain Paris, Mohamed Elgharib, Marc Pollefeys, Wojciech Matusik

We present a dataset of thousands of ambient and flash illumination pairs to enable studying flash photography and other applications that can benefit from having separate illuminations.

Semantic Match Consistency for Long-Term Visual Localization

no code implementations ECCV 2018 Carl Toft, Erik Stenborg, Lars Hammarstrand, Lucas Brynte, Marc Pollefeys, Torsten Sattler, Fredrik Kahl

Robust and accurate visual localization across large appearance variations due to changes in time of day, seasons, or changes of the environment is a challenging problem which is of importance to application areas such as navigation of autonomous robots.

Visual Localization

VSO: Visual Semantic Odometry

no code implementations ECCV 2018 Konstantinos-Nektarios Lianos, Johannes L. Schonberger, Marc Pollefeys, Torsten Sattler

Robust data association is a core problem of visual odometry, where image-to-image correspondences provide constraints for camera pose and map estimation.

Autonomous Driving Visual Odometry

Learning Priors for Semantic 3D Reconstruction

no code implementations ECCV 2018 Ian Cherabier, Johannes L. Schonberger, Martin R. Oswald, Marc Pollefeys, Andreas Geiger

In contrast to existing variational methods for semantic 3D reconstruction, our model is end-to-end trainable and captures more complex dependencies between the semantic labels and the 3D geometry.

3D Reconstruction

Hybrid Scene Compression for Visual Localization

no code implementations CVPR 2019 Federico Camposeco, Andrea Cohen, Marc Pollefeys, Torsten Sattler

Besides outperforming previous compression techniques in terms of pose accuracy under the same memory constraints, our compression scheme itself is also more efficient.

Quantization Self-Driving Cars +1

Augmenting Crowd-Sourced 3D Reconstructions Using Semantic Detections

no code implementations CVPR 2018 True Price, Johannes L. Schönberger, Zhen Wei, Marc Pollefeys, Jan-Michael Frahm

Image-based 3D reconstruction for Internet photo collections has become a robust technology to produce impressive virtual representations of real-world scenes.

3D Reconstruction

Semantic Visual Localization

no code implementations CVPR 2018 Johannes L. Schönberger, Marc Pollefeys, Andreas Geiger, Torsten Sattler

Robust visual localization under a wide range of viewing conditions is a fundamental problem in computer vision.

Visual Localization

From Point Clouds to Mesh Using Regression

no code implementations ICCV 2017 Lubor Ladicky, Olivier Saurer, SoHyeon Jeong, Fabio Maninchedda, Marc Pollefeys

Surface reconstruction from a point cloud is a standard subproblem in many algorithms for dense 3D reconstruction from RGB images or depth maps.

3D Reconstruction regression +1

Slanted Stixels: Representing San Francisco's Steepest Streets

1 code implementation17 Jul 2017 Daniel Hernandez-Juarez, Lukas Schneider, Antonio Espinosa, David Vázquez, Antonio M. López, Uwe Franke, Marc Pollefeys, Juan C. Moure

In this work we present a novel compact scene representation based on Stixels that infers geometric and semantic information.

Information-Flow Matting

1 code implementation CVPR 2017 Yağız Aksoy, Tunç Ozan Aydın, Marc Pollefeys

Our resulting novel linear system formulation can be solved in closed-form and is robust against several fundamental challenges of natural matting such as holes and remote intricate structures.

Image Matting

Consensus Maximization With Linear Matrix Inequality Constraints

no code implementations CVPR 2017 Pablo Speciale, Danda Pani Paudel, Martin R. Oswald, Till Kroeger, Luc van Gool, Marc Pollefeys

While randomized methods like RANSAC are fast, they do not guarantee global optimality and fail to manage large amounts of outliers.

Comparative Evaluation of Hand-Crafted and Learned Local Features

1 code implementation Conference on Computer Vision and Pattern Recognition 2017 Johannes L. Sch¨onberger, Hans Hardmeier, Torsten Sattler, Marc Pollefeys

In terms of matching performance, we evaluate the different descriptors regarding standard criteria. However, considering matching performance in isolation only provides an incomplete measure of a descriptor’s quality.

Image Retrieval Retrieval

SGM-Nets: Semi-Global Matching With Neural Networks

no code implementations CVPR 2017 Akihito Seki, Marc Pollefeys

Moreover, we propose a novel SGM parameterization, which deploys different penalties depending on either positive or negative disparity changes in order to represent the object structures more discriminatively.

Designing Effective Inter-Pixel Information Flow for Natural Image Matting

no code implementations CVPR 2017 Yagiz Aksoy, Tunc Ozan Aydin, Marc Pollefeys

Our resulting novel linear system formulation can be solved in closed-form and is robust against several fundamental challenges in natural matting such as holes and remote intricate structures.

Image Matting

Matching neural paths: transfer from recognition to correspondence search

no code implementations NeurIPS 2017 Nikolay Savinov, Lubor Ladicky, Marc Pollefeys

We propose to use a hierarchical semantic representation of the objects, coming from a convolutional neural network, to solve this ambiguity. A new Large-scale Point Cloud Classification Benchmark

1 code implementation12 Apr 2017 Timo Hackel, Nikolay Savinov, Lubor Ladicky, Jan D. Wegner, Konrad Schindler, Marc Pollefeys

With the massive data set presented in this paper, we aim at closing this data gap to help unleash the full potential of deep learning methods for 3D labelling tasks.

3D Point Cloud Classification Classification +5

The Stixel world: A medium-level representation of traffic scenes

no code implementations2 Apr 2017 Marius Cordts, Timo Rehfeld, Lukas Schneider, David Pfeiffer, Markus Enzweiler, Stefan Roth, Marc Pollefeys, Uwe Franke

We believe this challenge should be faced by introducing a representation of the sensory data that provides compressed and structured access to all relevant visual content of the scene.

Autonomous Vehicles object-detection +1

Semantically Guided Depth Upsampling

no code implementations2 Aug 2016 Nick Schneider, Lukas Schneider, Peter Pinggera, Uwe Franke, Marc Pollefeys, Christoph Stiller

We present a novel method for accurate and efficient up- sampling of sparse depth data, guided by high-resolution imagery.

Edge Detection Scene Labeling

Do It Yourself Hyperspectral Imaging With Everyday Digital Cameras

no code implementations CVPR 2016 Seoung Wug Oh, Michael S. Brown, Marc Pollefeys, Seon Joo Kim

In particular, due to the differences in spectral sensitivities of the cameras, different cameras yield different RGB measurements for the same spectral signal.

Large-Scale Location Recognition and the Geometric Burstiness Problem

1 code implementation CVPR 2016 Torsten Sattler, Michal Havlena, Konrad Schindler, Marc Pollefeys

Visual location recognition is the task of determining the place depicted in a query image from a given database of geo-tagged images.

Image Retrieval Re-Ranking +1

Sparse to Dense 3D Reconstruction From Rolling Shutter Images

no code implementations CVPR 2016 Olivier Saurer, Marc Pollefeys, Gim Hee Lee

It is well known that the rolling shutter effect in images captured with a moving rolling shutter camera causes inaccuracies to 3D reconstructions.

3D Reconstruction

TI-POOLING: transformation-invariant pooling for feature learning in Convolutional Neural Networks

1 code implementation CVPR 2016 Dmitry Laptev, Nikolay Savinov, Joachim M. Buhmann, Marc Pollefeys

This more efficient use of training data results in better performance on popular benchmark datasets with smaller number of parameters when comparing to standard convolutional neural networks with dataset augmentation and to other baselines.

Automatic 3D Reconstruction of Manifold Meshes via Delaunay Triangulation and Mesh Sweeping

no code implementations21 Apr 2016 Andrea Romanoni, Amaël Delaunoy, Marc Pollefeys, Matteo Matteucci

In this paper we propose a new approach to incrementally initialize a manifold surface for automatic 3D reconstruction from images.

3D Reconstruction

Semantic 3D Reconstruction with Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint

1 code implementation CVPR 2016 Nikolay Savinov, Christian Haene, Lubor Ladicky, Marc Pollefeys

We propose an approach for dense semantic 3D reconstruction which uses a data term that is defined as potentials over viewing rays, combined with continuous surface area penalization.

3D Reconstruction

Merging the Unmatchable: Stitching Visually Disconnected SfM Models

no code implementations ICCV 2015 Andrea Cohen, Torsten Sattler, Marc Pollefeys

An important variant of this problem is the case in which individual sides of a building can be reconstructed but not joined due to the missing visual overlap.

Non-Parametric Structure-Based Calibration of Radially Symmetric Cameras

no code implementations ICCV 2015 Federico Camposeco, Torsten Sattler, Marc Pollefeys

As a second step, we obtain the calibration by finding the translation of the camera center using an ordering constraint.


Entropy Minimization for Convex Relaxation Approaches

no code implementations ICCV 2015 Mohamed Souiai, Martin R. Oswald, Youngwook Kee, Junmo Kim, Marc Pollefeys, Daniel Cremers

Despite their enormous success in solving hard combinatorial problems, convex relaxation approaches often suffer from the fact that the computed solutions are far from binary and that subsequent heuristic binarization may substantially degrade the quality of computed solutions.

Binarization Image Segmentation +1

Capturing Hands in Action using Discriminative Salient Points and Physics Simulation

2 code implementations6 Jun 2015 Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, Juergen Gall

Hand motion capture is a popular research field, recently gaining more attention due to the ubiquity of RGB-D sensors.

Direction Matters: Depth Estimation With a Surface Normal Classifier

no code implementations CVPR 2015 Christian Hane, Lubor Ladicky, Marc Pollefeys

In this work we make use of recent advances in data driven classification to improve standard approaches for binocular stereo matching and single view depth estimation.

Depth Estimation Stereo Matching +1