Search Results for author: Alexei A. Efros

Found 100 papers, 64 papers with code

Interpreting the Weight Space of Customized Diffusion Models

1 code implementation13 Jun 2024 Amil Dravid, Yossi Gandelsman, Kuan-Chieh Wang, Rameen Abdal, Gordon Wetzstein, Alexei A. Efros, Kfir Aberman

First, as each point in the space corresponds to an identity, sampling a set of weights from it results in a model encoding a novel identity.

Data Attribution for Text-to-Image Models by Unlearning Synthesized Images

1 code implementation13 Jun 2024 Sheng-Yu Wang, Aaron Hertzmann, Alexei A. Efros, Jun-Yan Zhu, Richard Zhang

The goal of data attribution for text-to-image models is to identify the training images that most influence the generation of a new image.

Rethinking Score Distillation as a Bridge Between Image Distributions

no code implementations13 Jun 2024 David McAllister, Songwei Ge, Jia-Bin Huang, David W. Jacobs, Alexei A. Efros, Aleksander Holynski, Angjoo Kanazawa

We compare our method to existing approaches for score distillation sampling and show that it can produce high-frequency details with realistic colors.

Interpreting the Second-Order Effects of Neurons in CLIP

no code implementations6 Jun 2024 Yossi Gandelsman, Alexei A. Efros, Jacob Steinhardt

We interpret the function of individual neurons in CLIP by automatically describing them using text.

Attribute Zero Shot Segmentation

Synthesizing Moving People with 3D Control

no code implementations19 Jan 2024 Boyi Li, Jathushan Rajasegaran, Yossi Gandelsman, Alexei A. Efros, Jitendra Malik

This disentangled approach allows our method to generate a sequence of images that are faithful to the target motion in the 3D pose and, to the input image in terms of visual similarity.

COLMAP-Free 3D Gaussian Splatting

no code implementations CVPR 2024 Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, Xiaolong Wang

While neural rendering has led to impressive advances in scene reconstruction and novel view synthesis, it relies heavily on accurately pre-computed camera poses.

Neural Rendering Novel View Synthesis +1

Idempotent Generative Network

1 code implementation2 Nov 2023 Assaf Shocher, Amil Dravid, Yossi Gandelsman, Inbar Mosseri, Michael Rubinstein, Alexei A. Efros

We define the target manifold as the set of all instances that $f$ maps to themselves.

Interpreting CLIP's Image Representation via Text-Based Decomposition

1 code implementation9 Oct 2023 Yossi Gandelsman, Alexei A. Efros, Jacob Steinhardt

We decompose the image representation as a sum across individual image patches, model layers, and attention heads, and use CLIP's text representation to interpret the summands.

Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives

no code implementations NeurIPS 2023 Tom Monnier, Jake Austin, Angjoo Kanazawa, Alexei A. Efros, Mathieu Aubry

We compare our approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness on real-life captures from BlendedMVS and Nerfstudio.

Physical Simulations

Test-Time Training on Video Streams

no code implementations11 Jul 2023 Renhao Wang, Yu Sun, Yossi Gandelsman, Xinlei Chen, Alexei A. Efros, Xiaolong Wang

Before making a prediction on each test instance, the model is trained on the same instance using a self-supervised task, such as image reconstruction with masked autoencoders.

Image Reconstruction Panoptic Segmentation

Evaluating Data Attribution for Text-to-Image Models

1 code implementation ICCV 2023 Sheng-Yu Wang, Alexei A. Efros, Jun-Yan Zhu, Richard Zhang

The problem of data attribution in such models -- which of the images in the training set are most responsible for the appearance of a given generated image -- is a difficult yet important one.

Rosetta Neurons: Mining the Common Units in a Model Zoo

no code implementations ICCV 2023 Amil Dravid, Yossi Gandelsman, Alexei A. Efros, Assaf Shocher

In this paper, we demonstrate the existence of common features we call "Rosetta Neurons" across a range of models with different architectures, different tasks (generative and discriminative), and different types of supervision (class-supervised, text-supervised, self-supervised).

Putting People in Their Place: Affordance-Aware Human Insertion into Scenes

1 code implementation CVPR 2023 Sumith Kulal, Tim Brooks, Alex Aiken, Jiajun Wu, Jimei Yang, Jingwan Lu, Alexei A. Efros, Krishna Kumar Singh

Given a scene image with a marked region and an image of a person, we insert the person into the scene while respecting the scene affordances.

Internet Explorer: Targeted Representation Learning on the Open Web

1 code implementation27 Feb 2023 Alexander C. Li, Ellis Brown, Alexei A. Efros, Deepak Pathak

Modern vision models typically rely on fine-tuning general-purpose models pre-trained on large, static datasets.

Classification Representation Learning +1

InstructPix2Pix: Learning to Follow Image Editing Instructions

7 code implementations CVPR 2023 Tim Brooks, Aleksander Holynski, Alexei A. Efros

We propose a method for editing images from human instructions: given an input image and a written instruction that tells the model what to do, our model follows these instructions to edit the image.

Language Modelling Text-based Image Editing

Understanding Collapse in Non-Contrastive Siamese Representation Learning

1 code implementation29 Sep 2022 Alexander C. Li, Alexei A. Efros, Deepak Pathak

We empirically analyze these non-contrastive methods and find that SimSiam is extraordinarily sensitive to dataset and model size.

Continual Learning Contrastive Learning +1

Test-Time Training with Masked Autoencoders

1 code implementation15 Sep 2022 Yossi Gandelsman, Yu Sun, Xinlei Chen, Alexei A. Efros

Test-time training adapts to a new test distribution on the fly by optimizing a model for each test input using self-supervision.

Studying Bias in GANs through the Lens of Race

no code implementations6 Sep 2022 Vongani H. Maluleke, Neerja Thakkar, Tim Brooks, Ethan Weber, Trevor Darrell, Alexei A. Efros, Angjoo Kanazawa, Devin Guillory

In this work, we study how the performance and evaluation of generative image models are impacted by the racial composition of their training datasets.

Visual Prompting via Image Inpainting

1 code implementation1 Sep 2022 Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, Alexei A. Efros

How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification?

Colorization Edge Detection +6

Generating Long Videos of Dynamic Scenes

1 code implementation7 Jun 2022 Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei A. Efros, Tero Karras

Existing video generation methods often fail to produce new content as a function of time while maintaining consistencies expected in real environments, such as plausible dynamics and object persistence.

MORPH Video Generation

BlobGAN: Spatially Disentangled Scene Representations

no code implementations5 May 2022 Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, Alexei A. Efros

Blobs are differentiably placed onto a feature grid that is decoded into an image by a generative adversarial network.

Generative Adversarial Network

Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency

1 code implementation21 Apr 2022 Tom Monnier, Matthew Fisher, Alexei A. Efros, Mathieu Aubry

Approaches for single-view reconstruction typically rely on viewpoint annotations, silhouettes, the absence of background, multiple views of the same instance, a template shape, or symmetry.

3D Object Reconstruction From A Single Image 3D Reconstruction +2

Dataset Distillation by Matching Training Trajectories

5 code implementations CVPR 2022 George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, Jun-Yan Zhu

To efficiently obtain the initial and target network parameters for large-scale datasets, we pre-compute and store training trajectories of expert networks trained on the real dataset.

Dataset Distillation - 1IPC

Learning Pixel Trajectories with Multiscale Contrastive Random Walks

no code implementations CVPR 2022 Zhangxing Bian, Allan Jabri, Alexei A. Efros, Andrew Owens

A range of video modeling tasks, from optical flow to multiple object tracking, share the same fundamental challenge: establishing space-time correspondence.

Multiple Object Tracking Object +5

Hallucinating Pose-Compatible Scenes

no code implementations13 Dec 2021 Tim Brooks, Alexei A. Efros

We double the capacity of our model with respect to StyleGAN2 to handle such complex data, and design a pose conditioning mechanism that drives our model to learn the nuanced relationship between pose and scene.

Generative Adversarial Network Scene Generation

GAN-Supervised Dense Visual Alignment

1 code implementation CVPR 2022 William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei A. Efros, Eli Shechtman

We propose GAN-Supervised Learning, a framework for learning discriminative models and their GAN-generated training data jointly end-to-end.

Data Augmentation Dense Pixel Correspondence Estimation

Learning Co-segmentation by Segment Swapping for Retrieval and Discovery

1 code implementation29 Oct 2021 Xi Shen, Alexei A. Efros, Armand Joulin, Mathieu Aubry

The goal of this work is to efficiently identify visually similar patterns in images, e. g. identifying an artwork detail copied between an engraving and an oil painting, or recognizing parts of a night-time photograph visible in its daytime counterpart.

Graph Clustering Object Discovery +3

MarioNette: Self-Supervised Sprite Learning

1 code implementation NeurIPS 2021 Dmitriy Smirnov, Michael Gharbi, Matthew Fisher, Vitor Guizilini, Alexei A. Efros, Justin Solomon

Artists and video game designers often construct 2D animations using libraries of sprites -- textured patches of objects and characters.

Strumming to the Beat: Audio-Conditioned Contrastive Video Textures

no code implementations6 Apr 2021 Medhini Narasimhan, Shiry Ginosar, Andrew Owens, Alexei A. Efros, Trevor Darrell

We learn representations for video frames and frame-to-frame transition probabilities by fitting a video-specific model trained using contrastive learning.

Contrastive Learning Self-Supervised Learning +1

What Should Not Be Contrastive in Contrastive Learning

no code implementations ICLR 2021 Tete Xiao, Xiaolong Wang, Alexei A. Efros, Trevor Darrell

Recent self-supervised contrastive methods have been able to produce impressive transferable visual representations by learning to be invariant to different data augmentations.

Contrastive Learning

Learning to Factorize and Relight a City

no code implementations ECCV 2020 Andrew Liu, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros, Noah Snavely

We propose a learning-based framework for disentangling outdoor scenes into temporally-varying illumination and permanent scene factors.

Intrinsic Image Decomposition

Contrastive Learning for Unpaired Image-to-Image Translation

10 code implementations30 Jul 2020 Taesung Park, Alexei A. Efros, Richard Zhang, Jun-Yan Zhu

Furthermore, we draw negatives from within the input image itself, rather than from the rest of the dataset.

Contrastive Learning Image-to-Image Translation +1

Self-Supervised Policy Adaptation during Deployment

2 code implementations ICLR 2021 Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, Xiaolong Wang

A natural solution would be to keep training after deployment in the new environment, but this cannot be done if the new environment offers no reward signal.

Swapping Autoencoder for Deep Image Manipulation

4 code implementations NeurIPS 2020 Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, Richard Zhang

Deep generative models have become increasingly effective at producing realistic images from randomly sampled seeds, but using such models for controllable manipulation of existing images remains challenging.

Image Manipulation

CNN-generated images are surprisingly easy to spot... for now

4 code implementations CVPR 2020 Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, Alexei A. Efros

In this work we ask whether it is possible to create a "universal" detector for telling apart real images from these generated by a CNN, regardless of architecture or dataset used.

Data Augmentation Image Generation +1

Test-Time Training with Self-Supervision for Generalization under Distribution Shifts

3 code implementations29 Sep 2019 Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, Moritz Hardt

In this paper, we propose Test-Time Training, a general approach for improving the performance of predictive models when training and test data come from different distributions.

Building change detection for remote sensing images CARLA MAP Leaderboard +6

Unsupervised Domain Adaptation through Self-Supervision

3 code implementations26 Sep 2019 Yu Sun, Eric Tzeng, Trevor Darrell, Alexei A. Efros

This paper addresses unsupervised domain adaptation, the setting where labeled training data is available on a source domain, but the goal is to have good performance on a target domain with only unlabeled data.

Unsupervised Domain Adaptation

Test-Time Training for Out-of-Distribution Generalization

no code implementations25 Sep 2019 Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, Moritz Hardt

We introduce a general approach, called test-time training, for improving the performance of predictive models when test and training data come from different distributions.

Image Classification Out-of-Distribution Generalization +1

Detecting Photoshopped Faces by Scripting Photoshop

2 code implementations ICCV 2019 Sheng-Yu Wang, Oliver Wang, Andrew Owens, Richard Zhang, Alexei A. Efros

Most malicious photo manipulations are created using standard image editing tools, such as Adobe Photoshop.

Image Manipulation Detection

Meta-Learning to Guide Segmentation

no code implementations ICLR 2019 Kate Rakelly*, Evan Shelhamer*, Trevor Darrell, Alexei A. Efros, Sergey Levine

To explore generalization, we analyze guidance as a bridge between different levels of supervision to segment classes as the union of instances.

Meta-Learning Segmentation

Dataset Distillation

5 code implementations27 Nov 2018 Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, Alexei A. Efros

Model distillation aims to distill the knowledge of a complex model into a simpler one.

Everybody Dance Now

13 code implementations ICCV 2019 Caroline Chan, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros

This paper presents a simple method for "do as I do" motion transfer: given a source video of a person dancing, we can transfer that performance to a novel (amateur) target after only a few minutes of the target subject performing standard moves.

Face Generation Image-to-Image Translation +1

Improving Generalization via Scalable Neighborhood Component Analysis

2 code implementations ECCV 2018 Zhirong Wu, Alexei A. Efros, Stella X. Yu

Current major approaches to visual recognition follow an end-to-end formulation that classifies an input image into one of the pre-determined set of semantic categories.

Large-Scale Study of Curiosity-Driven Learning

4 code implementations ICLR 2019 Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, Alexei A. Efros

However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent.

Atari Games SNES Games

Few-Shot Segmentation Propagation with Guided Networks

1 code implementation25 May 2018 Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alexei A. Efros, Sergey Levine

Learning-based methods for visual segmentation have made progress on particular types of segmentation tasks, but are limited by the necessary supervision, the narrow definitions of fixed tasks, and the lack of control during inference for correcting errors.

Interactive Segmentation Segmentation +3

Fighting Fake News: Image Splice Detection via Learned Self-Consistency

3 code implementations ECCV 2018 Minyoung Huh, Andrew Liu, Andrew Owens, Alexei A. Efros

In this paper, we propose a learning algorithm for detecting visual image manipulations that is trained only using a large dataset of real photographs.

Image Forensics

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

1 code implementation ECCV 2018 Andrew Owens, Alexei A. Efros

The thud of a bouncing ball, the onset of speech as lips open -- when visual and audio events occur together, it suggests that there might be a common, underlying event that produced both signals.

Action Recognition Audio Source Separation +1

Learning Beyond Human Expertise with Generative Models for Dental Restorations

no code implementations30 Mar 2018 Jyh-Jing Hwang, Sergei Azernikov, Alexei A. Efros, Stella X. Yu

In the dental industry, it takes a technician years of training to design synthetic crowns that restore the function and integrity of missing teeth.

Object Recognition

Learning Category-Specific Mesh Reconstruction from Image Collections

no code implementations ECCV 2018 Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, Jitendra Malik

The shape is represented as a deformable 3D mesh model of an object category where a shape is parameterized by a learned mean shape and per-instance predicted deformation.

Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction

no code implementations CVPR 2018 Shubham Tulsiani, Alexei A. Efros, Jitendra Malik

We present a framework for learning single-view shape and pose prediction without using direct supervision for either.

Pose Prediction

From Lifestyle Vlogs to Everyday Interactions

no code implementations CVPR 2018 David F. Fouhey, Wei-cheng Kuo, Alexei A. Efros, Jitendra Malik

A major stumbling block to progress in understanding basic human interactions, such as getting out of bed or opening a refrigerator, is lack of good training data.

Future prediction

Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene

no code implementations CVPR 2018 Shubham Tulsiani, Saurabh Gupta, David Fouhey, Alexei A. Efros, Jitendra Malik

The goal of this paper is to take a single 2D image of a scene and recover the 3D structure in terms of a small set of factors: a layout representing the enclosing surfaces as well as a set of objects represented in terms of shape and pose.

3D Sketching using Multi-View Deep Volumetric Prediction

no code implementations26 Jul 2017 Johanna Delanoy, Mathieu Aubry, Phillip Isola, Alexei A. Efros, Adrien Bousseau

The main strengths of our approach are its robustness to freehand bitmap drawings, its ability to adapt to different object categories, and the continuum it offers between single-view and multi-view sketch-based modeling.

3D Reconstruction

Light Field Video Capture Using a Learning-Based Hybrid Imaging System

1 code implementation8 May 2017 Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, Ravi Ramamoorthi

Given a 3 fps light field sequence and a standard 30 fps 2D video, our system can then generate a full light field video at 30 fps.

Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency

no code implementations CVPR 2017 Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, Jitendra Malik

We study the notion of consistency between a 3D shape and a 2D observation and propose a differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view.

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

187 code implementations ICCV 2017 Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs.

 Ranked #1 on Image-to-Image Translation on zebra2horse (Frechet Inception Distance metric)

Multimodal Unsupervised Image-To-Image Translation Style Transfer +2

Learning Shape Abstractions by Assembling Volumetric Primitives

4 code implementations CVPR 2017 Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, Jitendra Malik

We present a learning framework for abstracting complex shapes by learning to assemble objects using 3D volumetric primitives.

Generative Visual Manipulation on the Natural Image Manifold

1 code implementation12 Sep 2016 Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, Alexei A. Efros

Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result.

Image Manipulation

A 4D Light-Field Dataset and CNN Architectures for Material Recognition

no code implementations24 Aug 2016 Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, Ravi Ramamoorthi

We introduce a new light-field dataset of materials, and take advantage of the recent success of deep learning to perform material recognition on the 4D light-field.

Image Classification Image Segmentation +4

View Synthesis by Appearance Flow

4 code implementations11 May 2016 Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, Alexei A. Efros

We address the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints.

Novel View Synthesis

Context Encoders: Feature Learning by Inpainting

11 code implementations CVPR 2016 Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A. Efros

In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s).

Learning Dense Correspondence via 3D-guided Cycle Consistency

no code implementations CVPR 2016 Tinghui Zhou, Philipp Krähenbühl, Mathieu Aubry, Qi-Xing Huang, Alexei A. Efros

We use ground-truth synthetic-to-synthetic correspondences, provided by the rendering engine, to train a ConvNet to predict synthetic-to-real, real-to-real and real-to-synthetic correspondences that are cycle-consistent with the ground-truth.

Colorful Image Colorization

39 code implementations28 Mar 2016 Richard Zhang, Phillip Isola, Alexei A. Efros

We embrace the underlying uncertainty of the problem by posing it as a classification task and use class-rebalancing at training time to increase the diversity of colors in the result.

Colorization Image Colorization +1

Occlusion-Aware Depth Estimation Using Light-Field Cameras

no code implementations ICCV 2015 Ting-Chun Wang, Alexei A. Efros, Ravi Ramamoorthi

In this paper, we develop a depth estimation algorithm that treats occlusion explicitly; the method also enables identification of occlusion edges, which may be useful in other applications.

Depth Estimation

A Century of Portraits: A Visual Historical Record of American High School Yearbooks

2 code implementations9 Nov 2015 Shiry Ginosar, Kate Rakelly, Sarah Sachs, Brian Yin, Crystal Lee, Philipp Krahenbuhl, Alexei A. Efros

4) A new method for discovering and displaying the visual elements used by the CNN-based date-prediction model to date portraits, finding that they correspond to the tell-tale fashions of each era.

Cultural Vocal Bursts Intensity Prediction

Learning Data-driven Reflectance Priors for Intrinsic Image Decomposition

no code implementations ICCV 2015 Tinghui Zhou, Philipp Krähenbühl, Alexei A. Efros

We propose a data-driven approach for intrinsic image decomposition, which is the process of inferring the confounding factors of reflectance and shading in an image.

Image Relighting Intrinsic Image Decomposition

Unsupervised Visual Representation Learning by Context Prediction

3 code implementations ICCV 2015 Carl Doersch, Abhinav Gupta, Alexei A. Efros

This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation.

Representation Learning

Seeing 3D Chairs: Exemplar Part-based 2D-3D Alignment using a Large Dataset of CAD Models

no code implementations CVPR 2014 Mathieu Aubry, Daniel Maturana, Alexei A. Efros, Bryan C. Russell, Josef Sivic

This paper poses object category detection in images as a type of 2D-to-3D alignment problem, utilizing the large quantities of 3D CAD models that have been made publicly available online.

Mid-level Visual Element Discovery as Discriminative Mode Seeking

no code implementations NeurIPS 2013 Carl Doersch, Abhinav Gupta, Alexei A. Efros

We also propose the Purity-Coverage plot as a principled way of experimentally analyzing and evaluating different visual discovery approaches, and compare our method against prior work on the Paris Street View dataset.

Scene Classification

Dating historical color images

no code implementations ECCV 2012 Frank Palermo, James Hays, Alexei A. Efros

We introduce the task of automatically estimating the age of historical color photographs.

Age Estimation

Cannot find the paper you are looking for? You can Submit a new open access paper.