Search Results for author:
Found 95 papers, 62 papers with code
Disentangled 3D Scene Generation with Layout Learning
We introduce a method to generate 3D scenes that are disentangled into their component objects.
Rethinking Patch Dependence for Masked Autoencoders
In this work, we re-examine inter-patch dependencies in the decoding mechanism of masked autoencoders (MAE).
Synthesizing Moving People with 3D Control
This disentangled approach allows our method to generate a sequence of images that are faithful to the target motion in the 3D pose and, to the input image in terms of visual similarity.
COLMAP-Free 3D Gaussian Splatting
While neural rendering has led to impressive advances in scene reconstruction and novel view synthesis, it relies heavily on accurately pre-computed camera poses.
Idempotent Generative Network
We define the target manifold as the set of all instances that $f$ maps to themselves.
Interpreting CLIP's Image Representation via Text-Based Decomposition
We decompose the image representation as a sum across individual image patches, model layers, and attention heads, and use CLIP's text representation to interpret the summands.
Test-Time Training on Video Streams
Before making a prediction on each test instance, the model is trained on the same instance using a self-supervised task, such as image reconstruction with masked autoencoders.
Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives
We compare our approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness on real-life captures from BlendedMVS and Nerfstudio.
Rosetta Neurons: Mining the Common Units in a Model Zoo
In this paper, we demonstrate the existence of common features we call "Rosetta Neurons" across a range of models with different architectures, different tasks (generative and discriminative), and different types of supervision (class-supervised, text-supervised, self-supervised).
Evaluating Data Attribution for Text-to-Image Models
The problem of data attribution in such models -- which of the images in the training set are most responsible for the appearance of a given generated image -- is a difficult yet important one.
Diffusion Self-Guidance for Controllable Image Generation
However, many aspects of an image are difficult or impossible to convey through text.
Generalizing Dataset Distillation via Deep Generative Prior
Dataset Distillation aims to distill an entire dataset's knowledge into a few synthetic images.
Putting People in Their Place: Affordance-Aware Human Insertion into Scenes
Given a scene image with a marked region and an image of a person, we insert the person into the scene while respecting the scene affordances.
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions
We propose a method for editing NeRF scenes with text-instructions.
Internet Explorer: Targeted Representation Learning on the Open Web
Modern vision models typically rely on fine-tuning general-purpose models pre-trained on large, static datasets.
InstructPix2Pix: Learning to Follow Image Editing Instructions
We propose a method for editing images from human instructions: given an input image and a written instruction that tells the model what to do, our model follows these instructions to edit the image.
Understanding Collapse in Non-Contrastive Siamese Representation Learning
We empirically analyze these non-contrastive methods and find that SimSiam is extraordinarily sensitive to dataset and model size.
Learning to Learn with Generative Models of Neural Network Checkpoints
We explore a data-driven approach for learning to optimize neural networks.
Test-Time Training with Masked Autoencoders
Test-time training adapts to a new test distribution on the fly by optimizing a model for each test input using self-supervision.
Studying Bias in GANs through the Lens of Race
In this work, we study how the performance and evaluation of generative image models are impacted by the racial composition of their training datasets.
Visual Prompting via Image Inpainting
How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification?
Ranked #5 on
Personalized Segmentation
on PerSeg
Generating Long Videos of Dynamic Scenes
Existing video generation methods often fail to produce new content as a function of time while maintaining consistencies expected in real environments, such as plausible dynamics and object persistence.
BlobGAN: Spatially Disentangled Scene Representations
Blobs are differentiably placed onto a feature grid that is decoded into an image by a generative adversarial network.
Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency
Approaches for single-view reconstruction typically rely on viewpoint annotations, silhouettes, the absence of background, multiple views of the same instance, a template shape, or symmetry.
3D Object Reconstruction From A Single Image
3D Reconstruction
+2
Dataset Distillation by Matching Training Trajectories
To efficiently obtain the initial and target network parameters for large-scale datasets, we pre-compute and store training trajectories of expert networks trained on the real dataset.
Learning Pixel Trajectories with Multiscale Contrastive Random Walks
A range of video modeling tasks, from optical flow to multiple object tracking, share the same fundamental challenge: establishing space-time correspondence.
Hallucinating Pose-Compatible Scenes
We double the capacity of our model with respect to StyleGAN2 to handle such complex data, and design a pose conditioning mechanism that drives our model to learn the nuanced relationship between pose and scene.
GAN-Supervised Dense Visual Alignment
We propose GAN-Supervised Learning, a framework for learning discriminative models and their GAN-generated training data jointly end-to-end.
Learning Co-segmentation by Segment Swapping for Retrieval and Discovery
The goal of this work is to efficiently identify visually similar patterns in images, e. g. identifying an artwork detail copied between an engraving and an oil painting, or recognizing parts of a night-time photograph visible in its daytime counterpart.
Video Autoencoder: self-supervised disentanglement of static 3D structure and motion
Relying on temporal continuity in videos, our work assumes that the 3D scene structure in nearby video frames remains static.
MarioNette: Self-Supervised Sprite Learning
Artists and video game designers often construct 2D animations using libraries of sprites -- textured patches of objects and characters.
Few-shot Image Generation via Cross-domain Correspondence
Training generative models, such as GANs, on a target domain containing limited examples (e. g., 10) can easily result in overfitting.
Ranked #3 on
10-shot image generation
on Babies
Strumming to the Beat: Audio-Conditioned Contrastive Video Textures
We learn representations for video frames and frame-to-frame transition probabilities by fitting a video-specific model trained using contrastive learning.
Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency
We propose \textit{dynamics cycles} that align dynamic robot behavior across two domains using a cycle-consistency constraint.
What Should Not Be Contrastive in Contrastive Learning
Recent self-supervised contrastive methods have been able to produce impressive transferable visual representations by learning to be invariant to different data augmentations.
Learning to Factorize and Relight a City
We propose a learning-based framework for disentangling outdoor scenes into temporally-varying illumination and permanent scene factors.
Contrastive Learning for Unpaired Image-to-Image Translation
Furthermore, we draw negatives from within the input image itself, rather than from the rest of the dataset.
Unselfie: Translating Selfies to Neutral-pose Portraits in the Wild
Due to the ubiquity of smartphones, it is popular to take photos of one's self, or "selfies."
Self-Supervised Policy Adaptation during Deployment
A natural solution would be to keep training after deployment in the new environment, but this cannot be done if the new environment offers no reward signal.
Swapping Autoencoder for Deep Image Manipulation
Deep generative models have become increasingly effective at producing realistic images from randomly sampled seeds, but using such models for controllable manipulation of existing images remains challenging.
Space-Time Correspondence as a Contrastive Random Walk
We cast correspondence as prediction of links in a space-time graph constructed from video.
RANSAC-Flow: generic two-stage image alignment
Coarse alignment is performed using RANSAC on off-the-shelf deep features.
CNN-generated images are surprisingly easy to spot... for now
In this work we ask whether it is possible to create a "universal" detector for telling apart real images from these generated by a CNN, regardless of architecture or dataset used.
Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
In this paper, we propose Test-Time Training, a general approach for improving the performance of predictive models when training and test data come from different distributions.
Ranked #34 on
Language Modelling
on LAMBADA
Building change detection for remote sensing images
CARLA MAP Leaderboard
+6
Unsupervised Domain Adaptation through Self-Supervision
This paper addresses unsupervised domain adaptation, the setting where labeled training data is available on a source domain, but the goal is to have good performance on a target domain with only unlabeled data.
Test-Time Training for Out-of-Distribution Generalization
We introduce a general approach, called test-time training, for improving the performance of predictive models when test and training data come from different distributions.
Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation
We propose an interactive GAN-based sketch-to-image translation method that helps novice users create images of simple objects.
Detecting Photoshopped Faces by Scripting Photoshop
Most malicious photo manipulations are created using standard image editing tools, such as Adobe Photoshop.
Meta-Learning to Guide Segmentation
To explore generalization, we analyze guidance as a bridge between different levels of supervision to segment classes as the union of instances.
Learning Correspondence from the Cycle-Consistency of Time
We introduce a self-supervised method for learning visual correspondence from unlabeled video.
Discovering Visual Patterns in Art Collections with Spatially-consistent Feature Learning
Our goal in this paper is to discover near duplicate patterns in large collections of artworks.
Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity
We evaluate the performance of these dynamic and modular agents in simulated environments.
Dataset Distillation
Model distillation aims to distill the knowledge of a complex model into a simpler one.
Time-Agnostic Prediction: Predicting Predictable Video Frames
Prediction is arguably one of the most basic functions of an intelligent system.
Everybody Dance Now
This paper presents a simple method for "do as I do" motion transfer: given a source video of a person dancing, we can transfer that performance to a novel (amateur) target after only a few minutes of the target subject performing standard moves.
Improving Generalization via Scalable Neighborhood Component Analysis
Current major approaches to visual recognition follow an end-to-end formulation that classifies an input image into one of the pre-determined set of semantic categories.
Large-Scale Study of Curiosity-Driven Learning
However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent.
Ranked #14 on
Atari Games
on Atari 2600 Montezuma's Revenge
Few-Shot Segmentation Propagation with Guided Networks
Learning-based methods for visual segmentation have made progress on particular types of segmentation tasks, but are limited by the necessary supervision, the narrow definitions of fixed tasks, and the lack of control during inference for correcting errors.
Fighting Fake News: Image Splice Detection via Learned Self-Consistency
In this paper, we propose a learning algorithm for detecting visual image manipulations that is trained only using a large dataset of real photographs.
Zero-Shot Visual Imitation
In our framework, the role of the expert is only to communicate the goals (i. e., what to imitate) during inference.
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
The thud of a bouncing ball, the onset of speech as lips open -- when visual and audio events occur together, it suggests that there might be a common, underlying event that produced both signals.
Learning Beyond Human Expertise with Generative Models for Dental Restorations
In the dental industry, it takes a technician years of training to design synthetic crowns that restore the function and integrity of missing teeth.
Learning Category-Specific Mesh Reconstruction from Image Collections
The shape is represented as a deformable 3D mesh model of an object category where a shape is parameterized by a learned mean shape and per-instance predicted deformation.
Investigating Human Priors for Playing Video Games
What makes humans so good at solving seemingly complex video games?
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics.
Ranked #19 on
Video Quality Assessment
on MSU FR VQA Database
Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction
We present a framework for learning single-view shape and pose prediction without using direct supervision for either.
From Lifestyle Vlogs to Everyday Interactions
A major stumbling block to progress in understanding basic human interactions, such as getting out of bed or opening a refrigerator, is lack of good training data.
Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene
The goal of this paper is to take a single 2D image of a scene and recover the 3D structure in terms of a small set of factors: a layout representing the enclosing surfaces as well as a set of objects represented in terms of shape and pose.
Toward Multimodal Image-to-Image Translation
Our proposed method encourages bijective consistency between the latent encoding and output modes.
CyCADA: Cycle-Consistent Adversarial Domain Adaptation
Domain adaptation is critical for success in new, unseen environments.
3D Sketching using Multi-View Deep Volumetric Prediction
The main strengths of our approach are its robustness to freehand bitmap drawings, its ability to adapt to different object categories, and the continuum it offers between single-view and multi-view sketch-based modeling.
Curiosity-driven Exploration by Self-supervised Prediction
In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether.
Real-Time User-Guided Image Colorization with Learned Deep Priors
The system directly maps a grayscale image, along with sparse, local user "hints" to an output colorization with a Convolutional Neural Network (CNN).
Light Field Video Capture Using a Learning-Based Hybrid Imaging System
Given a 3 fps light field sequence and a standard 30 fps 2D video, our system can then generate a full light field video at 30 fps.
Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency
We study the notion of consistency between a 3D shape and a 2D observation and propose a differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view.
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs.
Ranked #1 on
Image-to-Image Translation
on zebra2horse
(Frechet Inception Distance metric)
Multimodal Unsupervised Image-To-Image Translation
Style Transfer
+2
Learning Shape Abstractions by Assembling Volumetric Primitives
We present a learning framework for abstracting complex shapes by learning to assemble objects using 3D volumetric primitives.
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
We propose split-brain autoencoders, a straightforward modification of the traditional autoencoder architecture, for unsupervised representation learning.
Ranked #127 on
Self-Supervised Image Classification
on ImageNet
Representation Learning
Self-Supervised Image Classification
+1
Image-to-Image Translation with Conditional Adversarial Networks
We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems.
Generative Visual Manipulation on the Natural Image Manifold
Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result.
What makes ImageNet good for transfer learning?
Which is better: more classes or more examples per class?
A 4D Light-Field Dataset and CNN Architectures for Material Recognition
We introduce a new light-field dataset of materials, and take advantage of the recent success of deep learning to perform material recognition on the 4D light-field.
SVBRDF-Invariant Shape and Reflectance Estimation From Light-Field Cameras
Light-field cameras have recently emerged as a powerful tool for one-shot passive 3D shape capture.
View Synthesis by Appearance Flow
We address the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints.
Context Encoders: Feature Learning by Inpainting
In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s).
Learning Dense Correspondence via 3D-guided Cycle Consistency
We use ground-truth synthetic-to-synthetic correspondences, provided by the rendering engine, to train a ConvNet to predict synthetic-to-real, real-to-real and real-to-synthetic correspondences that are cycle-consistent with the ground-truth.
Colorful Image Colorization
We embrace the underlying uncertainty of the problem by posing it as a classification task and use class-rebalancing at training time to increase the diversity of colors in the result.
Ranked #128 on
Self-Supervised Image Classification
on ImageNet
Occlusion-Aware Depth Estimation Using Light-Field Cameras
In this paper, we develop a depth estimation algorithm that treats occlusion explicitly; the method also enables identification of occlusion edges, which may be useful in other applications.
A Century of Portraits: A Visual Historical Record of American High School Yearbooks
4) A new method for discovering and displaying the visual elements used by the CNN-based date-prediction model to date portraits, finding that they correspond to the tell-tale fashions of each era.
Learning Data-driven Reflectance Priors for Intrinsic Image Decomposition
We propose a data-driven approach for intrinsic image decomposition, which is the process of inferring the confounding factors of reflectance and shading in an image.
Learning a Discriminative Model for the Perception of Realism in Composite Images
What makes an image appear realistic?
Unsupervised Visual Representation Learning by Context Prediction
This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation.
Seeing 3D Chairs: Exemplar Part-based 2D-3D Alignment using a Large Dataset of CAD Models
This paper poses object category detection in images as a type of 2D-to-3D alignment problem, utilizing the large quantities of 3D CAD models that have been made publicly available online.
Mid-level Visual Element Discovery as Discriminative Mode Seeking
We also propose the Purity-Coverage plot as a principled way of experimentally analyzing and evaluating different visual discovery approaches, and compare our method against prior work on the Paris Street View dataset.
Dating historical color images
We introduce the task of automatically estimating the age of historical color photographs.
