Search Results for author: Gordon Wetzstein

Found 71 papers, 25 papers with code

Instant Continual Learning of Neural Radiance Fields

no code implementations4 Sep 2023 Ryan Po, Zhengyang Dong, Alexander W. Bergman, Gordon Wetzstein

Neural radiance fields (NeRFs) have emerged as an effective method for novel-view synthesis and 3D scene reconstruction.

3D Scene Reconstruction Continual Learning +1

Efficient 3D Articulated Human Generation with Layered Surface Volumes

no code implementations11 Jul 2023 Yinghao Xu, Wang Yifan, Alexander W. Bergman, Menglei Chai, Bolei Zhou, Gordon Wetzstein

These layers are rendered using alpha compositing with fast differentiable rasterization, and they can be interpreted as a volumetric representation that allocates its capacity to a manifold of finite thickness around the template.

Articulated 3D Head Avatar Generation using Text-to-Image Diffusion Models

no code implementations10 Jul 2023 Alexander W. Bergman, Wang Yifan, Gordon Wetzstein

Recent work on text-guided 3D object generation has shown great promise in addressing these needs.

Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization

no code implementations4 May 2023 Connor Z. Lin, Koki Nagano, Jan Kautz, Eric R. Chan, Umar Iqbal, Leonidas Guibas, Gordon Wetzstein, Sameh Khamis

To tackle this problem, we propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing.

Face Model Face Reconstruction

Learning Controllable Adaptive Simulation for Multi-resolution Physics

1 code implementation1 May 2023 Tailin Wu, Takashi Maruyama, Qingqing Zhao, Gordon Wetzstein, Jure Leskovec

In this work, we introduce Learning controllable Adaptive simulation for Multi-resolution Physics (LAMP) as the first full deep learning-based surrogate model that jointly learns the evolution model and optimizes appropriate spatial resolutions that devote more compute to the highly dynamic regions.

LumiGAN: Unconditional Generation of Relightable 3D Human Faces

no code implementations25 Apr 2023 Boyang Deng, Yifan Wang, Gordon Wetzstein

Unsupervised learning of 3D human faces from unstructured 2D image data is an active research area.

PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized Perception with Neural Sensors

no code implementations11 Apr 2023 Haley M. So, Laurie Bose, Piotr Dudek, Gordon Wetzstein

Conventional image sensors digitize high-resolution images at fast frame rates, producing a large amount of data that needs to be transmitted off the sensor for further processing.

Hand Gesture Recognition Hand-Gesture Recognition +1

CC3D: Layout-Conditioned Generation of Compositional 3D Scenes

no code implementations ICCV 2023 Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Xingguang Yan, Gordon Wetzstein, Leonidas Guibas, Andrea Tagliasacchi

In this work, we introduce CC3D, a conditional generative model that synthesizes complex 3D scenes conditioned on 2D semantic scene layouts, trained using single-view images.

Inductive Bias

Compositional 3D Scene Generation using Locally Conditioned Diffusion

no code implementations21 Mar 2023 Ryan Po, Gordon Wetzstein

Designing complex 3D scenes has been a tedious, manual process requiring domain expertise.

Scene Generation Text to 3D

DehazeNeRF: Multiple Image Haze Removal and 3D Shape Reconstruction using Neural Radiance Fields

no code implementations20 Mar 2023 Wei-Ting Chen, Wang Yifan, Sy-Yen Kuo, Gordon Wetzstein

Neural radiance fields (NeRFs) have demonstrated state-of-the-art performance for 3D computer vision tasks, including novel view synthesis and 3D shape reconstruction.

3D Shape Reconstruction Novel View Synthesis

MELON: NeRF with Unposed Images in SO(3)

no code implementations14 Mar 2023 Axel Levy, Mark Matthews, Matan Sela, Gordon Wetzstein, Dmitry Lagun

Neural radiance fields enable novel-view synthesis and scene reconstruction with photorealistic quality from a few images, but require known and accurate camera poses.

Inverse Rendering Novel View Synthesis +1

PointAvatar: Deformable Point-based Head Avatars from Videos

1 code implementation CVPR 2023 Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, Otmar Hilliges

The ability to create realistic, animatable and relightable head avatars from casual video sequences would open up wide ranging applications in communication and entertainment.

SinGRAF: Learning a 3D Generative Radiance Field for a Single Scene

no code implementations CVPR 2023 Minjung Son, Jeong Joon Park, Leonidas Guibas, Gordon Wetzstein

Generative models have shown great promise in synthesizing photorealistic 3D objects, but they require large amounts of training data.

Amortized Inference for Heterogeneous Reconstruction in Cryo-EM

no code implementations13 Oct 2022 Axel Levy, Gordon Wetzstein, Julien Martel, Frederic Poitevin, Ellen D. Zhong

Cryo-electron microscopy (cryo-EM) is an imaging modality that provides unique insights into the dynamics of proteins and other building blocks of life.

NeuForm: Adaptive Overfitting for Neural Shape Editing

no code implementations18 Jul 2022 Connor Z. Lin, Niloy J. Mitra, Gordon Wetzstein, Leonidas Guibas, Paul Guerrero

Neural representations are popular for representing shapes, as they can be learned form sensor data and used for data cleanup, model completion, shape editing, and shape synthesis.

3D-Aware Video Generation

1 code implementation29 Jun 2022 Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Hao Tang, Gordon Wetzstein, Leonidas Guibas, Luc van Gool, Radu Timofte

Generative models have emerged as an essential building block for many image synthesis and editing tasks.

Image Generation Video Generation

Generative Neural Articulated Radiance Fields

no code implementations28 Jun 2022 Alexander W. Bergman, Petr Kellnhofer, Wang Yifan, Eric R. Chan, David B. Lindell, Gordon Wetzstein

Unsupervised learning of 3D-aware generative adversarial networks (GANs) using only collections of single-view 2D photographs has very recently made much progress.

Learning to Solve PDE-constrained Inverse Problems with Graph Networks

no code implementations1 Jun 2022 Qingqing Zhao, David B. Lindell, Gordon Wetzstein

Given a sparse set of measurements, we are interested in recovering the initial condition or parameters of the PDE.

Time-multiplexed Neural Holography: A flexible framework for holographic near-eye displays with fast heavily-quantized spatial light modulators

no code implementations5 May 2022 Suyeon Choi, Manu Gopakumar, YiFan, Peng, Jonghyun Kim, Matthew O'Toole, Gordon Wetzstein

Holographic near-eye displays offer unprecedented capabilities for virtual and augmented reality systems, including perceptually important focus cues.

Learning Spatially Varying Pixel Exposures for Motion Deblurring

1 code implementation14 Apr 2022 Cindy M. Nguyen, Julien N. P. Martel, Gordon Wetzstein

Computationally removing the motion blur introduced by camera shake or object motion in a captured image remains a challenging task in computational photography.


3D GAN Inversion for Controllable Portrait Image Animation

no code implementations25 Mar 2022 Connor Z. Lin, David B. Lindell, Eric R. Chan, Gordon Wetzstein

Portrait image animation enables the post-capture adjustment of these attributes from a single image while maintaining a photorealistic reconstruction of the subject's likeness or identity.

Image Animation Pose Transfer

CryoAI: Amortized Inference of Poses for Ab Initio Reconstruction of 3D Molecular Volumes from Real Cryo-EM Images

1 code implementation15 Mar 2022 Axel Levy, Frédéric Poitevin, Julien Martel, Youssef Nashed, Ariana Peck, Nina Miolane, Daniel Ratner, Mike Dunne, Gordon Wetzstein

We introduce cryoAI, an ab initio reconstruction algorithm for homogeneous conformations that uses direct gradient-based optimization of particle poses and the electron scattering potential from single-particle cryo-EM data.

Efficient Geometry-aware 3D Generative Adversarial Networks

2 code implementations CVPR 2022 Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, Gordon Wetzstein

Unsupervised generation of high-quality multi-view-consistent images and 3D shapes using only collections of single-view 2D photographs has been a long-standing challenge.

Neural Rendering

BACON: Band-limited Coordinate Networks for Multiscale Scene Representation

1 code implementation CVPR 2022 David B. Lindell, Dave Van Veen, Jeong Joon Park, Gordon Wetzstein

These networks are trained to map continuous input coordinates to the value of a signal at each point.

MantissaCam: Learning Snapshot High-dynamic-range Imaging with Perceptually-based In-pixel Irradiance Encoding

no code implementations9 Dec 2021 Haley M. So, Julien N. P. Martel, Piotr Dudek, Gordon Wetzstein

We demonstrate the efficacy of our method in simulation and show benefits of our algorithm on modulo images captured with a prototype implemented with a programmable sensor.

Fast Training of Neural Lumigraph Representations using Meta Learning

no code implementations NeurIPS 2021 Alexander W. Bergman, Petr Kellnhofer, Gordon Wetzstein

Inspired by neural variants of image-based rendering, we develop a new neural rendering approach with the goal of quickly learning a high-quality representation which can also be rendered in real-time.

Meta-Learning Neural Rendering +1

ACORN: Adaptive Coordinate Networks for Neural Scene Representation

1 code implementation6 May 2021 Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, Gordon Wetzstein

Here, we introduce a new hybrid implicit-explicit network architecture and training strategy that adaptively allocates resources during training and inference based on the local complexity of a signal of interest.

3D Shape Representation Representation Learning

Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems

no code implementations ICCV 2021 Edwin Vargas, Julien N. P. Martel, Gordon Wetzstein, Henry Arguello

Compressive imaging using coded apertures (CA) is a powerful technique that can be used to recover depth, light fields, hyperspectral images and other quantities from a single snapshot.

ScanGAN360: A Generative Model of Realistic Scanpaths for 360$^{\circ}$ Images

no code implementations25 Mar 2021 Daniel Martin, Ana Serrano, Alexander W. Bergman, Gordon Wetzstein, Belen Masia

Generative adversarial approaches could alleviate this challenge by generating a large number of possible scanpaths for unseen images.

Dynamic Time Warping

D-VDAMP: Denoising-based Approximate Message Passing for Compressive MRI

1 code implementation25 Oct 2020 Christopher A. Metzler, Gordon Wetzstein

Plug and play (P&P) algorithms iteratively apply highly optimized image denoisers to impose priors and solve computational image reconstruction problems, to great effect.

Denoising Image Reconstruction

MetaSDF: Meta-learning Signed Distance Functions

2 code implementations NeurIPS 2020 Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, Gordon Wetzstein

Neural implicit shape representations are an emerging paradigm that offers many potential benefits over conventional discrete representations, including memory efficiency at a high spatial resolution.


Implicit Neural Representations with Periodic Activation Functions

23 code implementations NeurIPS 2020 Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, Gordon Wetzstein

However, current network architectures for such implicit neural representations are incapable of modeling signals with fine detail, and fail to represent a signal's spatial and temporal derivatives, despite the fact that these are essential to many physical signals defined implicitly as the solution to partial differential equations.

Image Inpainting

State of the Art on Neural Rendering

no code implementations8 Apr 2020 Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B. Goldman, Michael Zollhöfer

Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e. g., by the integration of differentiable rendering into network training.

BIG-bench Machine Learning Image Generation +2

Event Based, Near Eye Gaze Tracking Beyond 10,000Hz

1 code implementation7 Apr 2020 Anastasios N. Angelopoulos, Julien N. P. Martel, Amit P. S. Kohli, Jorg Conradt, Gordon Wetzstein

The cameras in modern gaze-tracking systems suffer from fundamental bandwidth and power limitations, constraining data acquisition speed to 300 Hz realistically.

Semantic Implicit Neural Scene Representations With Semi-Supervised Training

no code implementations28 Mar 2020 Amit Kohli, Vincent Sitzmann, Gordon Wetzstein

The recent success of implicit neural scene representations has presented a viable new method for how we capture and store 3D scenes.

3D Semantic Segmentation Representation Learning

Deep S$^3$PR: Simultaneous Source Separation and Phase Retrieval Using Deep Generative Models

1 code implementation14 Feb 2020 Christopher A. Metzler, Gordon Wetzstein

This paper introduces and solves the simultaneous source separation and phase retrieval (S$^3$PR) problem.


Keyhole Imaging: Non-Line-of-Sight Imaging and Tracking of Moving Objects Along a Single Optical Path

no code implementations13 Dec 2019 Christopher A. Metzler, David B. Lindell, Gordon Wetzstein

Non-line-of-sight (NLOS) imaging and tracking is an emerging technology that allows the shape or position of objects around corners or behind diffusers to be recovered from transient, time-of-flight measurements.

Autonomous Driving

Deep Optics for Monocular Depth Estimation and 3D Object Detection

no code implementations ICCV 2019 Julie Chang, Gordon Wetzstein

In addition, we train object detection networks on the KITTI dataset and show that the lens optimized for depth estimation also results in improved 3D object detection performance.

3D Object Detection Monocular Depth Estimation +2

LiFF: Light Field Features in Scale and Depth

1 code implementation CVPR 2019 Donald G. Dansereau, Bernd Girod, Gordon Wetzstein

Feature detectors and descriptors are key low-level vision tools that many higher-level tasks build on.

DeepVoxels: Learning Persistent 3D Feature Embeddings

1 code implementation CVPR 2019 Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer

In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis.

3D Reconstruction Novel View Synthesis

Convolutional Sparse Coding for High Dynamic Range Imaging

no code implementations13 Jun 2018 Ana Serrano, Felix Heide, Diego Gutierrez, Gordon Wetzstein, Belen Masia

Current HDR acquisition techniques are based on either (i) fusing multibracketed, low dynamic range (LDR) images, (ii) modifying existing hardware and capturing different exposures simultaneously with multiple sensors, or (iii) reconstructing a single image with spatially-varying pixel exposures.

Vocal Bursts Intensity Prediction

A Wide-Field-Of-View Monocentric Light Field Camera

no code implementations CVPR 2017 Donald G. Dansereau, Glenn Schuster, Joseph Ford, Gordon Wetzstein

Finally, we describe a processing toolchain, including a convenient spherical LF parameterization, and demonstrate depth estimation and post-capture refocus for indoor and outdoor panoramas with 15 x 15 x 1600 x 200 pixels (72 MPix) and a 138-degree FOV.

Depth Estimation

Reconstructing Transient Images From Single-Photon Sensors

no code implementations CVPR 2017 Matthew O'Toole, Felix Heide, David B. Lindell, Kai Zang, Steven Diamond, Gordon Wetzstein

Computer vision algorithms build on 2D images or 3D videos that capture dynamic events at the millisecond time scale.

Unrolled Optimization with Deep Priors

2 code implementations22 May 2017 Steven Diamond, Vincent Sitzmann, Felix Heide, Gordon Wetzstein

A broad class of problems at the core of computational imaging, sensing, and low-level computer vision reduces to the inverse problem of extracting latent images that follow a prior distribution, from measurements taken under a known physical image formation model.

Deblurring Denoising

Snapshot Difference Imaging using Time-of-Flight Sensors

no code implementations19 May 2017 Clara Callenberg, Felix Heide, Gordon Wetzstein, Matthias Hullin

Computational photography encompasses a diversity of imaging techniques, but one of the core operations performed by many of them is to compute image differences.

Dirty Pixels: Towards End-to-End Image Processing and Perception

1 code implementation23 Jan 2017 Steven Diamond, Vincent Sitzmann, Frank Julca-Aguilar, Stephen Boyd, Gordon Wetzstein, Felix Heide

As such, conventional imaging involves processing the RAW sensor measurements in a sequential pipeline of steps, such as demosaicking, denoising, deblurring, tone-mapping and compression.

Autonomous Driving Deblurring +10

How do people explore virtual environments?

no code implementations13 Dec 2016 Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, Gordon Wetzstein

Understanding how people explore immersive virtual environments is crucial for many applications, such as designing virtual reality (VR) content, developing new compression algorithms, or learning computational models of saliency or visual attention.

Fast and Flexible Convolutional Sparse Coding

no code implementations CVPR 2015 Felix Heide, Wolfgang Heidrich, Gordon Wetzstein

Convolutional sparse coding (CSC) has become an increasingly important tool in machine learning and computer vision.

General Classification

Cannot find the paper you are looking for? You can Submit a new open access paper.