Search Results for author:
Found 55 papers, 31 papers with code
LandscapeAR: Large Scale Outdoor Augmented Reality by Matching Photographs with Terrain Models Using Learned Descriptors
We introduce a solution to large scale Augmented Reality for outdoor scenes by registering camera images to textured Digital Elevation Models (DEMs).
Ranked #2 on
Patch Matching
on HPatches
(using extra training data)
Lumiere: A Space-Time Diffusion Model for Video Generation
We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis.
Ranked #6 on
Text-to-Video Generation
on UCF-101
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
VecFusion: Vector Font Generation with Diffusion
We present VecFusion, a new neural architecture that can generate vector fonts with varying topological structures and precise control point positions.
Readout Guidance: Learning Control from Diffusion Features
We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals.
Online Detection of AI-Generated Images
With advancements in AI-generated images coming on a continuous basis, it is increasingly difficult to distinguish traditionally-sourced images (e. g., photos, artwork) from AI-generated ones.
INVE: Interactive Neural Video Editing
We present Interactive Neural Video Editing (INVE), a real-time video editing solution, which can assist the video editing process by consistently propagating sparse frame edits to the entire video clip.
Perspective Fields for Single Image Camera Calibration
We propose perspective fields as a representation that models the local perspective properties of an image.
3inGAN: Learning a 3D Generative Model from Images of a Self-similar Scene
We introduce 3inGAN, an unconditional 3D generative model trained from 2D images of a single self-similar 3D scene.
Videogenic: Video Highlights via Photogenic Moments
This paper investigates the challenge of extracting highlight moments from videos.
Towards Accurate Reconstruction of 3D Scene Shape from A Single Monocular Image
To do so, we propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image, and then exploits 3D point cloud data to predict the depth shift and the camera's focal length that allow us to recover 3D scene shapes.
Towards Domain-agnostic Depth Completion
Our method leverages a data-driven prior in the form of a single image depth prediction network trained on large-scale datasets, the output of which is used as an input to our model.
Neural Volumetric Object Selection
We introduce an approach for selecting objects in neural volumetric 3D representations, such as multi-plane images (MPI) and neural radiance fields (NeRF).
ReLU Fields: The Little Non-linearity That Could
Although the MLPs are able to represent complex scenes with unprecedented quality and memory footprint, this expressive power of the MLPs, however, comes at the cost of long training and inference times.
Layered Neural Atlases for Consistent Video Editing
We present a method that decomposes, or "unwraps", an input video into a set of layered 2D atlases, each providing a unified representation of the appearance of an object (or background) over the video.
GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes
We present an algorithm that learns a coarse 3D representation of objects from unposed multi-view 2D mask supervision, then uses it to generate detailed mask and image texture.
Differentiable Signal Processing With Black-Box Audio Effects
We present a data-driven approach to automate audio signal processing by incorporating stateful third-party, audio effects as layers within a deep neural network.
CharacterGAN: Few-Shot Keypoint Character Animation and Reposing
Our model generates novel poses based on keypoint locations, which can be modified in real time while providing interactive feedback, allowing for intuitive reposing and animation.
Sampling Based Scene-Space Video Processing
We present a novel, sampling-based framework for processing video that enables high-quality scene-space video effects in the presence of inevitable errors in depth and camera pose estimation.
DuctTake: Spatiotemporal Video Compositing
DuctTake is a system designed to enable practical compositing of multiple takes of a scene into a single video.
Learning to Recover 3D Scene Shape from a Single Image
Despite significant progress in monocular depth estimation in the wild, recent state-of-the-art methods cannot be used to recover accurate 3D scene shape due to an unknown depth shift induced by shift-invariant reconstruction losses used in mixed-data depth prediction training, and possible unknown camera focal length.
Ranked #1 on
Monocular Depth Estimation
on IBims-1
(using extra training data)
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
We present a method to perform novel view and time synthesis of dynamic scenes, requiring only a monocular video with known camera poses as input.
Revisiting Adaptive Convolutions for Video Frame Interpolation
Video frame interpolation, the synthesis of novel views in time, is an increasingly popular research direction with many new papers further advancing the state of the art.
Real-time Semantic Segmentation with Fast Attention
The proposed architecture relies on our fast spatial attention, which is a simple yet efficient modification of the popular self-attention mechanism and captures the same rich spatial context at a small fraction of the computational cost, by changing the order of operations.
Ranked #32 on
Semantic Segmentation
on DensePASS
Swapping Autoencoder for Deep Image Manipulation
Deep generative models have become increasingly effective at producing realistic images from randomly sampled seeds, but using such models for controllable manipulation of existing images remains challenging.
Temporally Distributed Networks for Fast Video Semantic Segmentation
We present TDNet, a temporally distributed network designed for fast and accurate video semantic segmentation.
Ranked #2 on
Video Semantic Segmentation
on Cityscapes val
Improved Techniques for Training Single-Image GANs
Recently there has been an interest in the potential of learning generative models from a single image, as opposed to from a large dataset.
Generating Object Stamps
We present an algorithm to generate diverse foreground objects and composite them into background images using a GAN architecture.
CNN-generated images are surprisingly easy to spot... for now
In this work we ask whether it is possible to create a "universal" detector for telling apart real images from these generated by a CNN, regardless of architecture or dataset used.
Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation
We propose an interactive GAN-based sketch-to-image translation method that helps novice users create images of simple objects.
Detecting Photoshopped Faces by Scripting Photoshop
Most malicious photo manipulations are created using standard image editing tools, such as Adobe Photoshop.
Web Stereo Video Supervision for Depth Prediction from Dynamic Scenes
We present a fully data-driven method to compute depth from diverse monocular video sequences that contain large amounts of non-rigid objects, e. g., people.
Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction
In this paper, we address the problem of 3D object mesh reconstruction from RGB videos.
MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks
While Generative Adversarial Networks (GANs) have seen huge successes in image synthesis tasks, they are notoriously difficult to adapt to different datasets, in part due to instability during training and sensitivity to hyperparameters.
Ranked #1 on
Image Generation
on Indian Celebs 256 x 256
B-Script: Transcript-based B-roll Video Editing with Recommendations
In video production, inserting B-roll is a widely used technique to enrich the story and make a video more engaging.
Self-Supervised Generation of Spatial Audio for 360° Video
We introduce an approach to convert mono audio recorded by a 360° video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere.
DeepLens: Shallow Depth Of Field From A Single Image
To achieve this, we propose a novel neural network model comprised of a depth prediction module, a lens blur module, and a guided upsampling module.
Self-Supervised Generation of Spatial Audio for 360 Video
Using our approach, we show that it is possible to infer the spatial location of sound sources based only on 360 video and a mono audio track.
Localizing Moments in Video with Temporal Language
To benchmark whether our model, and other recent video localization models, can effectively reason about temporal language, we collect the novel TEMPOral reasoning in video and language (TEMPO) dataset.
Learning Blind Video Temporal Consistency
Our method takes the original unprocessed and per-frame processed videos as inputs to produce a temporally consistent video.
ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
We address the problem of finding realistic geometric corrections to a foreground object such that it appears natural when composited into a background image.
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics.
Ranked #19 on
Video Quality Assessment
on MSU FR VQA Database
Toward Multimodal Image-to-Image Translation
Our proposed method encourages bijective consistency between the latent encoding and output modes.
GSLAM: Initialization-robust Monocular Visual SLAM via Global Structure-from-Motion
Many monocular visual SLAM algorithms are derived from incremental structure-from-motion (SfM) methods.
Localizing Moments in Video with Natural Language
A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment.
Deep Video Deblurring for Hand-Held Cameras
We show that the features learned from this dataset extend to deblurring motion blur that arises due to camera shake in a wide range of videos, and compare the quality of results to a number of other baselines.
Efficient Large-scale Approximate Nearest Neighbor Search on the GPU
We present a new approach for efficient approximate nearest neighbor (ANN) search in high dimensional spaces, extending the idea of Product Quantization.
High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis
Recent advances in deep learning have shown exciting promise in filling large holes in natural images with semantically plausible and context aware details, impacting fundamental image manipulation tasks such as object removal.
Deep Video Deblurring
We show that the features learned from this dataset extend to deblurring motion blur that arises due to camera shake in a wide range of videos, and compare the quality of results to a number of other baselines.
Bilateral Space Video Segmentation
In this work, we propose a novel approach to video segmentation that operates in bilateral space.
Ranked #76 on
Semi-Supervised Video Object Segmentation
on DAVIS 2016
Fully Connected Object Proposals for Video Segmentation
We present a novel approach to video segmentation using multiple object proposals.
Ranked #77 on
Semi-Supervised Video Object Segmentation
on DAVIS 2016
FaceDirector: Continuous Control of Facial Performance in Video
We present a method to continuously blend between multiple facial performances of an actor, which can contain different facial expressions or emotional states.
Scalable Structure From Motion for Densely Sampled Videos
Videos consisting of thousands of high resolution frames are challenging for existing structure from motion (SfM) and simultaneous-localization and mapping (SLAM) techniques.
Phase-Based Frame Interpolation for Video
Standard approaches to computing interpolated (in-between) frames in a video sequence require accurate pixel correspondences between images e. g. using optical flow.
