Search Results for author:
Found 22 papers, 13 papers with code
Learning Video Representations without Natural Videos
A VideoMAE model pre-trained on our synthetic videos closes 97. 2\% of the performance gap on UCF101 action classification between training from scratch and self-supervised pre-training from natural videos, and outperforms the pre-trained model on HMDB51.
Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations
We investigate the internal representations of vision-language models (VLMs) to address hallucinations, a persistent challenge despite advances in model size and training.
Quantifying and Enabling the Interpretability of CLIP-like Models
However, little is known about the inner workings of CLIP.
Interpreting the Weight Space of Customized Diffusion Models
First, sampling a set of weights from this space results in a new model encoding a novel identity.
Interpreting the Second-Order Effects of Neurons in CLIP
We interpret the function of individual neurons in CLIP by automatically describing them using text.
The More You See in 2D, the More You Perceive in 3D
Inspired by this behavior, we introduce SAP3D, a system for 3D reconstruction and novel view synthesis from an arbitrary number of unposed images.
Synthesizing Moving People with 3D Control
This disentangled approach allows our method to generate a sequence of images that are faithful to the target motion in the 3D pose and, to the input image in terms of visual similarity.
The More You See in 2D the More You Perceive in 3D
Inspired by this behavior we introduce SAP3D a system for 3D reconstruction and novel view synthesis from an arbitrary number of unposed images.
IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks
Given a textual description of a visual task (e. g. "Left: input image, Right: foreground segmentation"), a few input-output visual examples, or both, the model in-context learns to solve it for a new test input.
Idempotent Generative Network
We define the target manifold as the set of all instances that $f$ maps to themselves.
Interpreting CLIP's Image Representation via Text-Based Decomposition
We decompose the image representation as a sum across individual image patches, model layers, and attention heads, and use CLIP's text representation to interpret the summands.
Reconstruction of Patient-Specific Confounders in AI-based Radiologic Image Interpretation using Generative Pretraining
Current techniques for evaluating deep learning models cannot visualize confounding factors at a diagnostic level.
Test-Time Training on Video Streams
Before making a prediction on each test instance, the model is trained on the same instance using a self-supervised task, such as image reconstruction with masked autoencoders.
Rosetta Neurons: Mining the Common Units in a Model Zoo
In this paper, we demonstrate the existence of common features we call "Rosetta Neurons" across a range of models with different architectures, different tasks (generative and discriminative), and different types of supervision (class-supervised, text-supervised, self-supervised).
Test-Time Training with Masked Autoencoders
Test-time training adapts to a new test distribution on the fly by optimizing a model for each test input using self-supervision.
Visual Prompting via Image Inpainting
How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification?
Ranked #5 on
Personalized Segmentation
on PerSeg
MyStyle: A Personalized Generative Prior
Given a small reference set of portrait images of a person (~100), we tune the weights of a pretrained StyleGAN face generator to form a local, low-dimensional, personalized manifold in the latent space.
Deep ViT Features as Dense Visual Descriptors
To distill the power of ViT features from convoluted design choices, we restrict ourselves to lightweight zero-shot methodologies (e. g., binning and clustering) applied directly to the features.
Ranked #6 on
Feature Upsampling
on ImageNet
Deep Saliency Prior for Reducing Visual Distraction
Using only a model that was trained to predict where people look at images, and no additional training data, we can produce a range of powerful editing effects for reducing distraction in images.
Explaining in Style: Training a GAN to explain a classifier in StyleSpace
A natural source for such attributes is the StyleSpace of StyleGAN, which is known to generate semantically meaningful dimensions in the image.
Semantic Pyramid for Image Generation
We demonstrate that our model results in a versatile and flexible framework that can be used in various classic and novel image generation tasks.
"Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors
It was shown [Ulyanov et al] that the structure of a single DIP generator network is sufficient to capture the low-level statistics of a single image.
