Search Results for author:
Found 71 papers, 32 papers with code
Neural Lens Modeling
Recent methods for 3D reconstruction and rendering increasingly benefit from end-to-end optimization of the entire image formation process.
ASIC: Aligning Sparse in-the-wild Image Collections
We present a self-supervised technique that directly optimizes on a sparse collection of images of a particular object/object category to obtain consistent dense correspondences across the collection.
Persistent Nature: A Generative Model of Unbounded 3D Worlds
Despite increasingly realistic image quality, recent 3D image generative models often operate on 3D volumes of fixed extent with limited camera motions.
Accidental Light Probes
Recovering lighting in a scene from a single image is a fundamental problem in computer vision.
Neural Scene Chronology
To tackle these problems, we propose a new scene representation equipped with a novel temporal step function encoding method that can model discrete scene-level content changes as piece-wise constant functions over time.
Omnimatte3D: Associating Objects and Their Effects in Unconstrained Monocular Video
Instead, our method applies recent progress in monocular camera pose and depth estimation to create a full, RGBD video layer for the background, along with a video layer for each foreground object.
Seeing a Rose in Five Thousand Ways
These instances all share the same intrinsics, but appear different due to a combination of variance within these intrinsics and differences in extrinsic factors, such as pose and illumination.
DynIBaR: Neural Dynamic Image-Based Rendering
Our system retains the advantages of prior methods in its ability to model complex scenes and view-dependent effects, but also enables synthesizing photo-realistic novel views from long videos featuring complex scene dynamics with unconstrained camera trajectories.
FactorMatte: Redefining Video Matting for Re-Composition Tasks
Based on this observation, we present a method for solving the factor matting problem that produces useful decompositions even for video with complex cross-layer interactions like splashes, shadows, and reflections.
What's in a Decade? Transforming Faces Through Time
How can one visually characterize people in a decade?
im2nerf: Image to Neural Radiance Field in the Wild
We take a step towards addressing this shortcoming by introducing a model that encodes the input image into a disentangled object representation that contains a code for object shape, a code for object appearance, and an estimated camera pose from which the object image is captured.
InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images
We present a method for learning to generate unbounded flythrough videos of natural scenes starting from a single view, where this capability is learned from a collection of single photographs, without requiring camera poses or even multiple views of each scene.
ARF: Artistic Radiance Fields
We present a method for transferring the artistic features of an arbitrary style image to a 3D scene.
Neural 3D Reconstruction in the Wild
We are witnessing an explosion of neural implicit representations in computer vision and graphics.
3D Moments from Near-Duplicate Photos
As output, we produce a video that smoothly interpolates the scene motion from the first photo to the second, while also producing camera motion with parallax that gives a heightened sense of 3D.
Deformable Sprites for Unsupervised Video Decomposition
We describe a method to extract persistent elements of a dynamic scene from an input video.
IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images
We propose a neural inverse rendering pipeline called IRON that operates on photometric images and outputs high-quality 3D content in the format of triangle meshes and material textures readily deployable in existing graphics pipelines.
Unsupervised Semantic Segmentation by Distilling Feature Correspondences
Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation.
Ranked #1 on
Unsupervised Semantic Segmentation
on COCO-Stuff
Dimensions of Motion: Monocular Prediction through Flow Subspaces
We introduce a way to learn to estimate a scene representation from a single image by predicting a low-dimensional subspace of optical flow for each training example, which encompasses the variety of possible camera and object movement.
Who's Waldo? Linking People Across Text and Images
We present a task and benchmark dataset for person-centric visual grounding, the problem of linking between people named in a caption and people pictured in an image.
Ranked #1 on
Person-centric Visual Grounding
on Who’s Waldo
(using extra training data)
Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Vision
The abundance and richness of Internet photos of landmarks and cities has led to significant progress in 3D vision over the past two decades, including automated 3D reconstructions of the world's landmarks from tourist photos.
Wide-Baseline Relative Camera Pose Estimation with Directional Learning
Modern deep learning techniques that regress the relative camera pose between two images have difficulty dealing with challenging scenarios, such as large camera motions resulting in occlusions and significant changes in perspective that leave little overlap between images.
Extreme Rotation Estimation using Dense Correlation Volumes
We present a technique for estimating the relative 3D rotation of an RGB image pair in an extreme setting, where the images have little or no overlap.
KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control
We cast this as the problem of aligning a source 3D object to a target 3D object from the same object category.
De-rendering the World's Revolutionary Artefacts
Recent works have shown exciting results in unsupervised image de-rendering -- learning to decompose 3D shape, appearance, and lighting from single-image collections without explicit supervision.
PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting
We present PhySG, an end-to-end inverse rendering pipeline that includes a fully differentiable renderer and can reconstruct geometry, materials, and illumination from scratch from a set of RGB input images.
Repopulating Street Scenes
We present a framework for automatically reconfiguring images of street scenes by populating, depopulating, or repopulating them with objects such as pedestrians or vehicles.
IBRNet: Learning Multi-View Image-Based Rendering
Unlike neural scene representation work that optimizes per-scene functions for rendering, we learn a generic view interpolation function that generalizes to novel scenes.
Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image
We introduce the problem of perpetual view generation - long-range generation of novel views corresponding to an arbitrarily long camera trajectory given a single image.
An Ethical Highlighter for People-Centric Dataset Creation
Important ethical concerns arising from computer vision datasets of people have been receiving significant attention, and a number of datasets have been withdrawn as a result.
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
We present a method to perform novel view and time synthesis of dynamic scenes, requiring only a monocular video with known camera poses as input.
Multi-Plane Program Induction with 3D Box Priors
We consider two important aspects in understanding and editing images: modeling regular, program-like texture or patterns in 2D planes, and 3D posing of these planes in the scene.
NeRF++: Analyzing and Improving Neural Radiance Fields
Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings, including 360 capture of bounded scenes and forward-facing capture of bounded and unbounded scenes.
Hidden Footprints: Learning Contextual Walkability from 3D Human Trails
Predicting where people can walk in a scene is important for many tasks, including autonomous driving systems and human behavior analysis.
Learning Gradient Fields for Shape Generation
Point cloud generation thus amounts to moving randomly sampled points to high-density areas.
Learning to Factorize and Relight a City
We propose a learning-based framework for disentangling outdoor scenes into temporally-varying illumination and permanent scene factors.
Crowdsampling the Plenoptic Function
These photos represent a sparse and unstructured sampling of the plenoptic function for a particular scene.
An Analysis of SVD for Deep Rotation Estimation
Symmetric orthogonalization via SVD, and closely related procedures, are well-known techniques for projecting matrices onto $O(n)$ or $SO(n)$.
MetaSDF: Meta-learning Signed Distance Functions
Neural implicit shape representations are an emerging paradigm that offers many potential benefits over conventional discrete representations, including memory efficiency at a high spatial resolution.
Learning Feature Descriptors using Camera Pose Supervision
Recent research on learned visual descriptors has shown promising improvements in correspondence estimation, a key component of many 3D vision tasks.
The Plant Pathology 2020 challenge dataset to classify foliar disease of apples
Appropriate and timely deployment of disease management depends on early disease detection.
Single-View View Synthesis with Multiplane Images
A recent strand of work in view synthesis uses deep learning to generate multiplane images (a camera-centric, layered 3D representation) given two or more input images at known viewpoints.
Depth Sensing Beyond LiDAR Range
Depth sensing is a critical component of autonomous driving technologies, but today's LiDAR- or stereo camera-based solutions have limited range.
DualSDF: Semantic Shape Manipulation using a Two-Level Representation
We are seeing a Cambrian explosion of 3D shape representations for use in machine learning.
Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination
We present a deep learning solution for estimating the incident illumination at any 3D location within a scene from an input narrow-baseline stereo image pair.
Leveraging Vision Reconstruction Pipelines for Satellite Imagery
Reconstructing 3D geometry from satellite imagery is an important topic of research.
GeoStyle: Discovering Fashion Trends and Events
Understanding fashion styles and trends is of great potential interest to retailers and consumers alike.
UprightNet: Geometry-Aware Camera Orientation Estimation from Single Images
We introduce UprightNet, a learning-based approach for estimating 2DoF camera orientation from a single RGB image of an indoor scene.
DeepView: View Synthesis with Learned Gradient Descent
We present a novel approach to view synthesis using multiplane images (MPIs).
Pushing the Boundaries of View Extrapolation with Multiplane Images
We present a theoretical analysis showing how the range of views that can be rendered from an MPI increases linearly with the MPI disparity sampling frequency, as well as a novel MPI prediction procedure that theoretically enables view extrapolations of up to $4\times$ the lateral viewpoint movement allowed by prior work.
Learning the Depths of Moving People by Watching Frozen People
We present a method for predicting dense depth in scenarios where both a monocular camera and people in the scene are freely moving.
Neural Rerendering in the Wild
Starting from internet photos of a tourist landmark, we apply traditional 3D reconstruction to register the photos and approximate the scene as a point cloud.
Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments
We study the problem of jointly reasoning about language and vision through a navigation and spatial reasoning task.
Ranked #10 on
Vision and Language Navigation
on Touchdown Dataset
CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering
Intrinsic image decomposition is a challenging, long-standing computer vision problem for which ground truth data is very difficult to acquire.
Layer-structured 3D Scene Inference via View Synthesis
We present an approach to infer a layer-structured 3D representation of a scene from a single input image.
Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
We demonstrate this framework on 3D pose estimation by proposing a differentiable objective that seeks the optimal set of keypoints for recovering the relative pose between two views of an object.
Stereo Magnification: Learning View Synthesis using Multiplane Images
The view synthesis problem--generating novel views of a scene from known imagery--has garnered recent attention due in part to compelling applications in virtual and augmented reality.
MegaDepth: Learning Single-View Depth Prediction from Internet Photos
We validate the use of large amounts of Internet data by showing that models trained on MegaDepth exhibit strong generalization-not only to novel scenes, but also to other diverse datasets including Make3D, KITTI, and DIW, even when no images from those datasets are seen during training.
Learning Intrinsic Image Decomposition from Watching the World
However, it is difficult to collect ground truth training data at scale for intrinsic images.
StreetStyle: Exploring world-wide clothing styles from millions of photos
Each day billions of photographs are uploaded to photo-sharing services and social media platforms.
Shading Annotations in the Wild
We demonstrate the value of our data and network in an application to intrinsic images, where we can reduce decomposition artifacts produced by existing algorithms.
Unsupervised Learning of Depth and Ego-Motion from Video
We present an unsupervised learning framework for the task of monocular depth and camera motion estimation from unstructured video sequences.
Deep Feature Interpolation for Image Content Changes
We propose Deep Feature Interpolation (DFI), a new data-driven baseline for automatic high-resolution image transformation.
From A to Z: Supervised Transfer of Style and Content Using Deep Neural Network Generators
We propose a new neural network architecture for solving single-image analogies - the generation of an entire set of stylistically similar images from just a single input image.
BubbLeNet: Foveated Imaging for Visual Discovery
We propose a new method for turning an Internet-scale corpus of categorized images into a small set of human-interpretable discriminative visual elements using powerful tools based on deep learning.
DeepStereo: Learning to Predict New Views from the World's Imagery
To our knowledge, our work is the first to apply deep learning to the problem of new view synthesis from sets of real-world, natural imagery.
Material Recognition in the Wild with the Materials in Context Database
In this paper, we introduce a new, large-scale, open dataset of materials in the wild, the Materials in Context Database (MINC), and combine this dataset with deep learning to achieve material recognition and segmentation of images in the wild.
Minimal Scene Descriptions from Structure from Motion Models
How much data do we need to describe a location?
Photometric Ambient Occlusion
We present a method for computing ambient occlusion (AO) for a stack of images of a scene from a fixed viewpoint.
Graph-Based Discriminative Learning for Location Recognition
For a query image, each database image is ranked according to these local distance functions in order to place the image in the right part of the graph.
