Search Results for author:
Found 53 papers, 30 papers with code
Koala: Key frame-conditioned long video-LLM
Long video question answering is a challenging task that involves recognizing short-term activities and reasoning about their fine-grained relationships.
Machine-generated Text Localization
Machine-Generated Text (MGT) detection aims to identify a piece of text as machine or human written.
Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks
Furthermore, prior work's Typographic attacks against CLIP randomly sample a misleading class from a predefined set of categories.
UniHuman: A Unified Model for Editing Human Images in the Wild
In this paper, we propose UniHuman, a unified model that addresses multiple facets of human image editing in real-world settings.
CLAMP: Contrastive LAnguage Model Prompt-tuning
Our results show that LLMs can, indeed, achieve good image classification performance when adapted this way.
Learning to Compose SuperWeights for Neural Parameter Allocation Search
To address this, we generate layer weights by learning to compose sets of SuperWeights, which represent a group of trainable parameters.
A Unified Framework for Connecting Noise Modeling to Boost Noise Detection
In this work, we explore the integration of these two approaches, proposing an interconnected structure with three crucial blocks: noise modeling, source knowledge identification, and enhanced noise detection using noise source-knowledge-integration methods.
MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters
Newly grown layer weights are generated by using a new linear combination of existing templates for a layer.
CHAMMI: A benchmark for channel-adaptive models in microscopy imaging
In this paper, we present a benchmark for investigating channel-adaptive models in microscopy imaging, which consists of 1) a dataset of varied-channel single-cell images, and 2) a biologically relevant evaluation framework.
Let Models Speak Ciphers: Multiagent Debate through Embeddings
Although natural language is an obvious choice for communication due to LLM's language understanding capability, the token sampling step needed when generating natural language poses a potential risk of information loss, as it uses only one token to represent the model's belief across the entire vocabulary.
Socratis: Are large multimodal models emotionally aware?
We further see that current captioning metrics based on large vision-language models also fail to correlate with human preferences.
From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Bias
By training on real and synthetic data separately, FFR avoids the issue of bias toward signals from the pair $(B, G)$.
Multiscale Video Pretraining for Long-Term Activity Forecasting
To alleviate this issue, we propose Multiscale Video Pretraining (MVP), a novel self-supervised pretraining approach that learns robust representations for forecasting by learning to predict contextualized representations of future video clips over multiple timescales.
LNL+K: Learning with Noisy Labels and Noise Source Distribution Knowledge
Learning with noisy labels (LNL) is challenging as the model tends to memorize noisy labels, which can lead to overfitting.
Text-to-image Editing by Image Information Removal
Using the input image as a control could mitigate these issues, but since these models are trained via reconstruction, a model can simply hide information about the original image when encoding it to perfectly reconstruct the image without learning the editing task.
WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset
Webpages have been a rich resource for language and vision-language tasks.
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding
Webpages have been a rich, scalable resource for vision-language and language only tasks.
ERM++: An Improved Baseline for Domain Generalization
We also explore the relationship between DG performance and similarity to pre-training data, and find that similarity to pre-training data distributions is an important driver of performance, but that ERM++ with stronger initializations can deliver strong performance even on dissimilar datasets. Code is released at https://github. com/piotr-teterwak/erm_plusplus.
Language-Guided Audio-Visual Source Separation via Trimodal Consistency
We propose a self-supervised approach for learning to perform audio source separation in videos based on natural language queries, using only unlabeled video and audio pairs as training data.
Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark
For example, asking a model to generate a varying number of the same object to measure its ability to count or providing a text prompt with several objects that each have a different attribute to identify its ability to match objects and attributes correctly.
Collecting The Puzzle Pieces: Disentangled Self-Driven Human Pose Transfer by Permuting Textures
Then we reconstruct the input image by sampling from the permuted textures for patch-level disentanglement.
Bias Mimicking: A Simple Sampling Approach for Bias Mitigation
Using this notion, BM, through a novel training procedure, ensures that the model is exposed to the entire distribution per epoch without repeating samples.
NewsStories: Illustrating articles with visual summaries
Thus, we explore a novel setting where the goal is to learn a self-supervised visual-language representation that is robust to varying text length and the number of images.
Supervised Attribute Information Removal and Reconstruction for Image Manipulation
Thus, the source attribute information can often be hidden in the disentangled features, leading to unwanted image editing effects.
Complex Scene Image Editing by Scene Graph Comprehension
Unlike object detection methods based solely on object category, our method can accurately recognize the target object by comprehending the objects and their semantic relationships within a complex scene.
Movie Genre Classification by Language Augmentation and Shot Sampling
We evaluate our method on MovieNet and Condensed Movies datasets, achieving approximate 6-9% improvement in mean Average Precision (mAP) over the baselines.
A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility
To study VLN with unknown command feasibility, we introduce a new dataset Mobile app Tasks with Iterative Feedback (MoTIF), where the goal is to complete a natural language command in a mobile app.
Show, Write, and Retrieve: Entity-aware Article Generation and Retrieval
Article comprehension is an important challenge in natural language processing with many applications such as article generation or image-to-article retrieval.
Anchoring to Exemplars for Training Mixture-of-Expert Cell Embeddings
Analyzing the morphology of cells in microscopy images can provide insights into the mechanism of compounds or the function of genes.
From Coarse to Fine-grained Concept based Discrimination for Phrase Detection
Phrase detection requires methods to identify if a phrase is relevant to an image and localize it, if applicable.
Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations.
MixtureEnsembles: Leveraging Parameter Sharing for Efficient Ensembles
We improve on these methods with MixtureEnsembles, which learns to factorize ensemble members with shared parameters by constructing each layer with a linear combination of templates.
Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments
In recent years, vision-language research has shifted to study tasks which require more complex reasoning, such as interactive question answering, visual common sense reasoning, and question-answer plausibility prediction.
CDS: Cross-Domain Self-Supervised Pre-Training
We present a two-stage pre-training approach that improves the generalization ability of standard single-domain pre-training.
Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News
In addition to the valuable insights gleaned from our user study experiments, we provide a relatively effective approach based on detecting visual-semantic inconsistencies, which will serve as an effective first line of defense and a useful reference for future work in defending against machine-generated disinformation.
Neural Parameter Allocation Search
We introduce Neural Parameter Allocation Search (NPAS), a novel task where the goal is to train a neural network given an arbitrary, fixed parameter budget.
Learning to Scale Multilingual Representations for Vision-Language Tasks
Current multilingual vision-language models either require a large number of additional parameters for each supported language, or suffer performance degradation as languages are added.
Cross-domain Self-supervised Learning for Domain Adaptation with Few Source Labels
We show that when labeled source examples are limited, existing methods often fail to learn discriminative features applicable for both source and target domains.
MILA: Multi-Task Learning from Videos via Efficient Inter-Frame Attention
We embed the attention module in a ``slow-fast'' architecture, where the slower network runs on sparsely sampled keyframes and the light-weight shallow network runs on non-keyframes at a high frame rate.
LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
However, while such approaches tend to focus on identifying relationships between elements of the video and language modalities, there is less emphasis on modeling relational context between video frames given the semantic context of the query.
wMAN: WEAKLY-SUPERVISED MOMENT ALIGNMENT NETWORK FOR TEXT-BASED VIDEO SEGMENT RETRIEVAL
Given a video and a sentence, the goal of weakly-supervised video moment retrieval is to locate the video segment which is described by the sentence without having access to temporal annotations during training.
MULE: Multimodal Universal Language Embedding
In this paper, we present a modular approach which can easily be incorporated into existing vision-language methods in order to support many languages.
Learning Similarity Conditions Without Explicit Supervision
Many real-world tasks require models to compare images along multiple similarity conditions (e. g. similarity in color, category or shape).
Language Features Matter: Effective Language Representations for Vision-Language Tasks
Shouldn't language and vision features be treated equally in vision-language (VL) tasks?
Why do These Match? Explaining the Behavior of Image Similarity Models
Explaining a deep learning model can help users understand its behavior and allow researchers to discern its shortcomings.
Revisiting Image-Language Networks for Open-ended Phrase Detection
Most existing work that grounds natural language phrases in images starts with the assumption that the phrase in question is relevant to the image.
Give me a hint! Navigating Image Databases using Human-in-the-loop Feedback
In this paper, we introduce an attribute-based interactive image search which can leverage human-in-the-loop feedback to iteratively refine image search results.
Multilevel Language and Vision Integration for Text-to-Clip Retrieval
To capture the inherent structures present in both text and video, we introduce a multilevel model that integrates vision and language features earlier and more tightly than prior work.
Learning Type-Aware Embeddings for Fashion Compatibility
Outfits in online fashion data are composed of items of many different types (e. g. top, bottom, shoes) that share some stylistic relationship with one another.
Conditional Image-Text Embedding Networks
This paper presents an approach for grounding phrases in images which jointly learns multiple text-conditioned embeddings in a single end-to-end model.
Enhancing Video Summarization via Vision-Language Embedding
This paper addresses video summarization, or the problem of distilling a raw video into a shorter form while still capturing the original story.
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
The Flickr30k dataset has become a standard benchmark for sentence-based image description.
Ranked #17 on
Image Retrieval
on Flickr30K 1K test
