Search Results for author: Felix Heide

Found 65 papers, 25 papers with code

Neural Spline Fields for Burst Image Fusion and Layer Separation

no code implementations21 Dec 2023 Ilya Chugunov, David Shustin, Ruyu Yan, Chenyang Lei, Felix Heide

Each photo in an image burst can be considered a sample of a complex 3D scene: the product of parallax, diffuse and specular materials, scene motion, and illuminant variation.

Shadow Removal

Dispersed Structured Light for Hyperspectral 3D Imaging

no code implementations30 Nov 2023 Suhyun Shin, Seokjun Choi, Felix Heide, Seung-Hwan Baek

In this work, we present Dispersed Structured Light (DSL), a cost-effective and compact method for accurate hyperspectral 3D imaging.

3D Reconstruction

Spectral and Polarization Vision: Spectro-polarimetric Real-world Dataset

no code implementations29 Nov 2023 Yujin Jeon, Eunsue Choi, Youngchan Kim, Yunseong Moon, Khalid Omer, Felix Heide, Seung-Hwan Baek

Image datasets are essential not only in validating existing methods in computer vision but also in developing new methods.

In the Blink of an Eye: Event-based Emotion Recognition

1 code implementation6 Oct 2023 Haiwei Zhang, Jiqing Zhang, Bo Dong, Pieter Peers, Wenwei Wu, Xiaopeng Wei, Felix Heide, Xin Yang

To the best of our knowledge, our method is the first eye-based emotion recognition method that leverages event-based cameras and spiking neural network.

Emotion Recognition

Spatially Varying Nanophotonic Neural Networks

no code implementations7 Aug 2023 Kaixuan Wei, Xiao Li, Johannes Froech, PRANEETH CHAKRAVARTHULA, James Whitehead, Ethan Tseng, Arka Majumdar, Felix Heide

The explosive growth of computation and energy cost of artificial intelligence has spurred strong interests in new computing modalities as potential alternatives to conventional electronic processors.

Thin On-Sensor Nanophotonic Array Cameras

no code implementations5 Aug 2023 PRANEETH CHAKRAVARTHULA, Jipeng Sun, Xiao Li, Chenyang Lei, Gene Chou, Mario Bijelic, Johannes Froesch, Arka Majumdar, Felix Heide

The optical array is embedded on a metasurface that, at 700~nm height, is flat and sits on the sensor cover glass at 2. 5~mm focal distance from the sensor.

Stochastic Light Field Holography

no code implementations12 Jul 2023 Florian Schiffers, PRANEETH CHAKRAVARTHULA, Nathan Matsuda, Grace Kuo, Ethan Tseng, Douglas Lanman, Felix Heide, Oliver Cossairt

The Visual Turing Test is the ultimate goal to evaluate the realism of holographic displays.

S$^3$Track: Self-supervised Tracking with Soft Assignment Flow

no code implementations17 May 2023 Fatemeh Azimi, Fahim Mannan, Felix Heide

With this training approach in hand, we develop an appearance-based model for learning instance-aware object features used to construct a cost matrix based on the pairwise distances between the object features.

Multiple Object Tracking Object +1

ScatterNeRF: Seeing Through Fog with Physically-Based Inverse Neural Rendering

no code implementations ICCV 2023 Andrea Ramazzina, Mario Bijelic, Stefanie Walz, Alessandro Sanvito, Dominik Scheuble, Felix Heide

With data as bottleneck and most of today's training data relying on good weather conditions with inclement weather as outlier, we rely on an inverse rendering approach to reconstruct the scene content.

Autonomous Vehicles Inverse Rendering +1

Instance Segmentation in the Dark

1 code implementation27 Apr 2023 Linwei Chen, Ying Fu, Kaixuan Wei, Dezhi Zheng, Felix Heide

Existing instance segmentation techniques are primarily tailored for high-visibility inputs, but their performance significantly deteriorates in extremely low-light environments.

Instance Segmentation Object Detection +2

Multi-view Spectral Polarization Propagation for Video Glass Segmentation

no code implementations ICCV 2023 Yu Qiao, Bo Dong, Ao Jin, Yu Fu, Seung-Hwan Baek, Felix Heide, Pieter Peers, Xiaopeng Wei, Xin Yang

In this paper, we present the first polarization-guided video glass segmentation propagation solution (PGVS-Net) that can robustly and coherently propagate glass segmentation in RGB-P video sequences.

Image Segmentation Segmentation +1

LiDAR-in-the-Loop Hyperparameter Optimization

no code implementations CVPR 2023 Félix Goudreault, Dominik Scheuble, Mario Bijelic, Nicolas Robidoux, Felix Heide

The resulting point clouds output by these DSPs are input to downstream 3D vision models -- both, in the form of training datasets or as input at inference time.

3D Object Detection Hyperparameter Optimization +1

Single Depth-image 3D Reflection Symmetry and Shape Prediction

no code implementations ICCV 2023 Zhaoxuan Zhang, Bo Dong, Tong Li, Felix Heide, Pieter Peers, BaoCai Yin, Xin Yang

In this paper, we present Iterative Symmetry Completion Network (ISCNet), a single depth-image shape completion method that exploits reflective symmetry cues to obtain more detailed shapes.

Seeing With Sound: Long-range Acoustic Beamforming for Multimodal Scene Understanding

no code implementations CVPR 2023 PRANEETH CHAKRAVARTHULA, Jim Aldon D’Souza, Ethan Tseng, Joe Bartusek, Felix Heide

We validate the benefit of adding sound detections to existing RGB cameras in challenging automotive scenarios, where camera-only approaches fail or do not deliver the ultra-fast rates of pressure sensors.

Autonomous Vehicles object-detection +2

Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography

no code implementations CVPR 2023 Ilya Chugunov, Yuxuan Zhang, Felix Heide

Modern mobile burst photography pipelines capture and merge a short sequence of frames to recover an enhanced image, but often disregard the 3D nature of the scene they capture, treating pixel motion between images as a 2D aggregation problem.

Depth And Camera Motion Pose Estimation

Neural Volume Super-Resolution

no code implementations9 Dec 2022 Yuval Bahat, Yuxuan Zhang, Hendrik Sommerhoff, Andreas Kolb, Felix Heide

This allows us to super-resolve the 3D scene representation by applying 2D convolutional networks on the 2D feature planes.

Super-Resolution

The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection

1 code implementation CVPR 2023 Geoffroi Côté, Fahim Mannan, Simon Thibault, Jean-François Lalonde, Felix Heide

Recently, joint optimization approaches that design lenses alongside other components of the image acquisition and processing pipeline -- notably, downstream neural networks -- have achieved improved imaging quality or better performance on vision tasks.

object-detection Object Detection

Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions

no code implementations ICCV 2023 Gene Chou, Yuval Bahat, Felix Heide

Probabilistic diffusion models have achieved state-of-the-art results for image synthesis, inpainting, and text-to-image tasks.

Image Generation

GenSDF: Two-Stage Learning of Generalizable Signed Distance Functions

1 code implementation6 Jun 2022 Gene Chou, Ilya Chugunov, Felix Heide

The first stage uses an episodic training scheme to simulate training on unlabeled data and meta-learns initial shape priors.

Meta-Learning Object +1

LiDAR Snowfall Simulation for Robust 3D Object Detection

1 code implementation CVPR 2022 Martin Hahner, Christos Sakaridis, Mario Bijelic, Felix Heide, Fisher Yu, Dengxin Dai, Luc van Gool

Due to the difficulty of collecting and annotating training data in this setting, we propose a physically based method to simulate the effect of snowfall on real clear-weather LiDAR point clouds.

Autonomous Driving Object +3

Glass Segmentation Using Intensity and Spectral Polarization Cues

no code implementations CVPR 2022 Haiyang Mei, Bo Dong, Wen Dong, Jiaxi Yang, Seung-Hwan Baek, Felix Heide, Pieter Peers, Xiaopeng Wei, Xin Yang

Transparent and semi-transparent materials pose significant challenges for existing scene understanding and segmentation algorithms due to their lack of RGB texture which impedes the extraction of meaningful features.

Scene Understanding Segmentation +1

Spiking Transformers for Event-Based Single Object Tracking

no code implementations CVPR 2022 Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, BaoCai Yin, Xin Yang

In particular, the proposed architecture features a transformer module to provide global spatial information and a spiking neural network (SNN) module for extracting temporal cues.

Object Object Tracking

All-photon Polarimetric Time-of-Flight Imaging

no code implementations CVPR 2022 Seung-Hwan Baek, Felix Heide

Conventional ToF imaging methods estimate the depth by sending pulses of light into a scene and measuring the ToF of the first-arriving photons directly reflected from a scene surface without any temporal delay.

Autonomous Driving

All You Need is RAW: Defending Against Adversarial Attacks with Camera Image Pipelines

1 code implementation16 Dec 2021 Yuxuan Zhang, Bo Dong, Felix Heide

Various defense methods have proposed image-to-image mapping methods, either including these perturbations in the training process or removing them in a preprocessing denoising step.

Adversarial Defense Denoising +3

Gated2Gated: Self-Supervised Depth Estimation from Gated Images

1 code implementation CVPR 2022 Amanpreet Walia, Stefanie Walz, Mario Bijelic, Fahim Mannan, Frank Julca-Aguilar, Michael Langer, Werner Ritter, Felix Heide

Gated cameras hold promise as an alternative to scanning LiDAR sensors with high-resolution 3D depth that is robust to back-scatter in fog, snow, and rain.

Depth Estimation

Neural Point Light Fields

no code implementations CVPR 2022 Julian Ost, Issam Laradji, Alejandro Newell, Yuval Bahat, Felix Heide

We introduce Neural Point Light Fields that represent scenes implicitly with a light field living on a sparse point cloud.

Novel View Synthesis

The Implicit Values of A Good Hand Shake: Handheld Multi-Frame Neural Depth Refinement

1 code implementation CVPR 2022 Ilya Chugunov, Yuxuan Zhang, Zhihao Xia, Xuaner, Zhang, Jiawen Chen, Felix Heide

Modern smartphones can continuously stream multi-megapixel RGB images at 60Hz, synchronized with high-quality 3D pose information and low-resolution LiDAR-driven depth estimates.

A survey on computational spectral reconstruction methods from RGB to hyperspectral imaging

no code implementations30 Jun 2021 Jingang Zhang, Runmu Su, Wenqi Ren, Qiang Fu, Felix Heide, Yunfeng Nie

We present a thorough investigation of these state-of-the-art spectral reconstruction methods from the widespread RGB images.

Spectral Reconstruction

Centimeter-Wave Free-Space Time-of-Flight Imaging

no code implementations25 May 2021 Seung-Hwan Baek, Noah Walsh, Ilya Chugunov, Zheng Shi, Felix Heide

In this work, we close this gap and propose a computational imaging method for all-optical free-space correlation before photo-conversion that achieves micron-scale depth resolution with robustness to surface reflectance and ambient light with conventional silicon intensity sensors.

Polarimetric Spatio-Temporal Light Transport Probing

no code implementations25 May 2021 Seung-Hwan Baek, Felix Heide

This allows us, for the first time, to decompose scene light transport into temporal, spatial, and complete polarimetric dimensions that unveil scene properties hidden to conventional methods.

Metamerism Scene Understanding

Neural Ray-Tracing: Learning Surfaces and Reflectance for Relighting and View Synthesis

1 code implementation28 Apr 2021 Julian Knodt, Joe Bartusek, Seung-Hwan Baek, Felix Heide

Recent neural rendering methods have demonstrated accurate view interpolation by predicting volumetric density and color with a neural network.

Neural Rendering

ZeroScatter: Domain Transfer for Long Distance Imaging and Vision through Scattering Media

1 code implementation CVPR 2021 Zheng Shi, Ethan Tseng, Mario Bijelic, Werner Ritter, Felix Heide

Most of today's supervised imaging and vision approaches, however, rely on training data collected in the real world that is biased towards good weather conditions, with dense fog, snow, and heavy rain as outliers in these datasets.

Autonomous Vehicles Decision Making +1

Adversarial Imaging Pipelines

no code implementations CVPR 2021 Buu Phan, Fahim Mannan, Felix Heide

As a result, optimized patterns can become adversarial for the classifier after being transformed by a certain camera ISP and optic but not for others.

Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo

no code implementations CVPR 2021 Seung-Hwan Baek, Felix Heide

Active stereo cameras that recover depth from structured light captures have become a cornerstone sensor modality for 3D scene reconstruction and understanding tasks across application domains.

3D Scene Reconstruction

Neural Scene Graphs for Dynamic Scenes

1 code implementation CVPR 2021 Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, Felix Heide

Recent implicit neural rendering methods have demonstrated that it is possible to learn accurate view synthesis for complex scenes by predicting their volumetric density and color supervised solely by a set of RGB images.

Neural Rendering

Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar

1 code implementation CVPR 2020 Nicolas Scheiner, Florian Kraus, Fangyin Wei, Buu Phan, Fahim Mannan, Nils Appenrodt, Werner Ritter, Jürgen Dickmann, Klaus Dietmayer, Bernhard Sick, Felix Heide

In this work, we depart from visible-wavelength approaches and demonstrate detection, classification, and tracking of hidden objects in large-scale dynamic environments using Doppler radars that can be manufactured at low-cost in series production.

Temporal Sequences

Pixel-Accurate Depth Evaluation in Realistic Driving Scenarios

1 code implementation21 Jun 2019 Tobias Gruber, Mario Bijelic, Felix Heide, Werner Ritter, Klaus Dietmayer

This work introduces an evaluation benchmark for depth estimation and completion using high-resolution depth measurements with angular resolution of up to 25" (arcsecond), akin to a 50 megapixel camera with per-pixel depth available.

Depth Estimation

Defending Against Universal Attacks Through Selective Feature Regeneration

1 code implementation CVPR 2020 Tejas Borkar, Felix Heide, Lina Karam

Deep neural network (DNN) predictions have been shown to be vulnerable to carefully crafted adversarial perturbations.

Adversarial Defense

Hyperparameter Optimization in Black-box Image Processing using Differentiable Proxies

1 code implementation SIGGRAPH 2019 2019 Ethan Tseng, Felix Yu, Yuting Yang, Fahim Mannan, Karl St. Arnaud, Derek Nowrouzezahrai, Jean-François Lalonde, Felix Heide

We present a fully automatic system to optimize the parameters of black-box hardware and software image processing pipelines according to any arbitrary (i. e., application-specific) metric.

Hyperparameter Optimization Image Denoising +3

Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather

1 code implementation CVPR 2020 Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, Felix Heide

The fusion of multimodal sensor streams, such as camera, lidar, and radar measurements, plays a critical role in object detection for autonomous vehicles, which base their decision making on these inputs.

Autonomous Vehicles Decision Making +3

Gated2Depth: Real-time Dense Lidar from Gated Images

2 code implementations ICCV 2019 Tobias Gruber, Frank Julca-Aguilar, Mario Bijelic, Werner Ritter, Klaus Dietmayer, Felix Heide

The proposed replacement for scanning lidar systems is real-time, handles back-scatter and provides dense depth at long ranges.

Scene Understanding

DeepVoxels: Learning Persistent 3D Feature Embeddings

1 code implementation CVPR 2019 Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer

In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis.

3D Reconstruction Novel View Synthesis

Steady-state Non-Line-of-Sight Imaging

no code implementations CVPR 2019 Wenzheng Chen, Simon Daneau, Fahim Mannan, Felix Heide

Relying on consumer color image sensors, with high fill factor, high quantum efficiency and low read-out noise, we demonstrate high-fidelity color NLOS imaging for scene configurations tackled before with picosecond time resolution.

Convolutional Sparse Coding for High Dynamic Range Imaging

no code implementations13 Jun 2018 Ana Serrano, Felix Heide, Diego Gutierrez, Gordon Wetzstein, Belen Masia

Current HDR acquisition techniques are based on either (i) fusing multibracketed, low dynamic range (LDR) images, (ii) modifying existing hardware and capturing different exposures simultaneously with multiple sensors, or (iii) reconstructing a single image with spatially-varying pixel exposures.

Vocal Bursts Intensity Prediction

Reconstructing Transient Images From Single-Photon Sensors

no code implementations CVPR 2017 Matthew O'Toole, Felix Heide, David B. Lindell, Kai Zang, Steven Diamond, Gordon Wetzstein

Computer vision algorithms build on 2D images or 3D videos that capture dynamic events at the millisecond time scale.

Unrolled Optimization with Deep Priors

2 code implementations22 May 2017 Steven Diamond, Vincent Sitzmann, Felix Heide, Gordon Wetzstein

A broad class of problems at the core of computational imaging, sensing, and low-level computer vision reduces to the inverse problem of extracting latent images that follow a prior distribution, from measurements taken under a known physical image formation model.

Deblurring Denoising

Snapshot Difference Imaging using Time-of-Flight Sensors

no code implementations19 May 2017 Clara Callenberg, Felix Heide, Gordon Wetzstein, Matthias Hullin

Computational photography encompasses a diversity of imaging techniques, but one of the core operations performed by many of them is to compute image differences.

Discriminative Transfer Learning for General Image Restoration

no code implementations27 Mar 2017 Lei Xiao, Felix Heide, Wolfgang Heidrich, Bernhard Schölkopf, Michael Hirsch

Recently, several discriminative learning approaches have been proposed for effective image restoration, achieving convincing trade-off between image quality and computational efficiency.

Computational Efficiency Deblurring +3

Dirty Pixels: Towards End-to-End Image Processing and Perception

1 code implementation23 Jan 2017 Steven Diamond, Vincent Sitzmann, Frank Julca-Aguilar, Stephen Boyd, Gordon Wetzstein, Felix Heide

As such, conventional imaging involves processing the RAW sensor measurements in a sequential pipeline of steps, such as demosaicking, denoising, deblurring, tone-mapping and compression.

Autonomous Driving Deblurring +10

Fast and Flexible Convolutional Sparse Coding

no code implementations CVPR 2015 Felix Heide, Wolfgang Heidrich, Gordon Wetzstein

Convolutional sparse coding (CSC) has become an increasingly important tool in machine learning and computer vision.

General Classification

Diffuse Mirrors: 3D Reconstruction from Diffuse Indirect Illumination Using Inexpensive Time-of-Flight Sensors

no code implementations CVPR 2014 Felix Heide, Lei Xiao, Wolfgang Heidrich, Matthias B. Hullin

The functional difference between a diffuse wall and a mirror is well understood: one scatters back into all directions, and the other one preserves the directionality of reflected light.

3D Reconstruction

Stochastic Deconvolution

no code implementations CVPR 2013 James Gregson, Felix Heide, Matthias B. Hullin, Mushfiqur Rouf, Wolfgang Heidrich

We present a novel stochastic framework for non-blind deconvolution based on point samples obtained from random walks.

Cannot find the paper you are looking for? You can Submit a new open access paper.