Search Results for author:
Found 50 papers, 29 papers with code
TAP: Targeted Prompting for Task Adaptive Generation of Textual Training Instances for Visual Classification
Vision and Language Models (VLMs), such as CLIP, have enabled visual recognition of a potentially unlimited set of categories described by text prompts.
Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models
Vision and Language (VL) models offer an effective method for aligning representation spaces of images and text, leading to numerous applications such as cross-modal retrieval, visual question answering, captioning, and more.
LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections
Recently, large-scale pre-trained Vision and Language (VL) models have set a new state-of-the-art (SOTA) in zero-shot visual classification enabling open-vocabulary recognition of potentially unlimited set of categories defined as simple language prompts.
Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages
Recent models such as XLS-R and Whisper have made multilingual speech technologies more accessible by pre-training on audio from around 100 spoken languages each.
Listen, Think, and Understand
In this paper, we propose a novel audio foundation model, called LTU (Listen, Think, and Understand).
Ranked #3 on
Music Question Answering
on MusicQA Dataset
(using extra training data)
Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs
Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks.
Constructive Assimilation: Boosting Contrastive Learning Performance through View Generation Strategies
Recently, several attempts have been made to replace such domain-specific, human-designed transformations with generated views that are learned.
Going Beyond Nouns With Vision & Language Models Using Synthetic Data
We contribute Synthetic Visual Concepts (SyViC) - a million-scale synthetic dataset and data generation codebase allowing to generate additional suitable data to improve VLC understanding and compositional reasoning of VL models.
MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge
We adapt a VL model for zero-shot and few-shot action recognition using a collection of unlabeled videos and an unpaired action dictionary.
Ranked #2 on
Zero-Shot Action Recognition
on Kinetics
Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning
Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks.
Learning to Grow Pretrained Models for Efficient Transformer Training
Scaling transformers has led to significant breakthroughs in many domains, leading to a paradigm in which larger versions of existing models are trained and released on a periodic basis.
Teaching Structured Vision & Language Concepts to Vision & Language Models
Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks.
PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data
We present a multi-task prompt learning approach for video transformers, where a shared video transformer backbone is enhanced by a small set of specialized parameters for each task.
MAEDAY: MAE for few and zero shot AnomalY-Detection
We show that MAEDAY performs surprisingly well at this task.
CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning
Our experiments show that we outperform the current SOTA method DualPrompt on established benchmarks by as much as 4. 5% in average final accuracy.
On the Transferability of Visual Features in Generalized Zero-Shot Learning
Generalized Zero-Shot Learning (GZSL) aims to train a classifier that can generalize to unseen classes, using a set of attributes as auxiliary information, and the visual features extracted from a pre-trained convolutional neural network.
Teaching Structured Vision&Language Concepts to Vision&Language Models
Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks.
ConStruct-VL: Data-Free Continual Structured VL Concepts Learning
This leads to reasoning mistakes, which need to be corrected as they occur by teaching VL models the missing SVLC skills; often this must be done using private data where the issue was found, which naturally leads to a data-free continual (no task-id) VL learning setting.
On the Importance of Calibration in Semi-supervised Learning
State-of-the-art (SOTA) semi-supervised learning (SSL) methods have been highly successful in leveraging a mix of labeled and unlabeled data by combining techniques of consistency regularization and pseudo-labeling.
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval
Inspired by the fact that English text-video retrieval outperforms other languages, we train a student model using input text in different languages to match the cross-modal predictions from teacher models using input text in English.
Contrastive Audio-Visual Masked Autoencoder
In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities.
Ranked #1 on
Audio Tagging
on AudioSet
(using extra training data)
VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
We follow up with the analysis of the attribute-based zero-shot learning capabilities of these models, evaluating how well this classical zero-shot notion emerges from large-scale webly supervision.
FETA: Towards Specializing Foundation Models for Expert Task Applications
However, as we show in this paper, FMs still have poor out-of-the-box performance on expert tasks (e. g. retrieval of car manuals technical illustrations from language queries), data for which is either unseen or belonging to a long-tail part of the data distribution of the huge datasets used for FM pre-training.
Ranked #1 on
Image-to-Text Retrieval
on FETA Car-Manuals
Structured Video Tokens @ Ego4D PNR Temporal Localization Challenge 2022
First, as both images and videos contain structured information, we enrich a transformer model with a set of \emph{object tokens} that can be used across images and videos.
Point- of-no-return (PNR) temporal localization
Temporal Localization
Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokens
We explore a particular instantiation of scene structure, namely a \emph{Hand-Object Graph}, consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contact as edges.
Task2Sim: Towards Effective Pre-Training and Transfer From Synthetic Data
It is thus better to tailor synthetic pre-training data to a specific downstream task, for best performance.
Unsupervised Domain Generalization by Learning a Bridge Across Domains
The ability to generalize learned representations across significantly different visual domains, such as between real photos, clipart, paintings, and sketches, is a fundamental capacity of the human visual system.
Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data
It is thus better to tailor synthetic pre-training data to a specific downstream task, for best performance.
CHARTER: heatmap-based multi-type chart data extraction
The digital conversion of information stored in documents is a great source of knowledge.
Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
As the base dataset and unlabeled dataset are from different domains, projecting the target images in the class-domain of the base dataset with a fixed pretrained model might be sub-optimal.
Detector-Free Weakly Supervised Grounding by Separation
In this work, we focus on the task of Detector-Free WSG (DF-WSG) to solve WSG without relying on a pre-trained detector.
Ranked #1 on
Phrase Grounding
on Visual Genome
A Broad Study on the Transferability of Visual Representations with Contrastive Learning
Tremendous progress has been made in visual representation learning, notably with the recent success of self-supervised contrastive learning methods.
Self-Supervised Classification Network
To guarantee non-degenerate solutions (i. e., solutions where all labels are assigned to the same class) we propose a mathematically motivated variant of the cross-entropy loss that has a uniform prior asserted on the predicted labels.
Ranked #3 on
Unsupervised Image Classification
on ImageNet
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
Temporal modelling is the key for efficient video action recognition.
A Maximal Correlation Approach to Imposing Fairness in Machine Learning
As machine learning algorithms grow in popularity and diversify to many industries, ethical and legal concerns regarding their fairness have become increasingly relevant.
Fine-grained Angular Contrastive Learning with Coarse Labels
A very practical example of C2FS is when the target classes are sub-classes of the training classes.
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition
Specifically, given a video frame, a policy network is used to decide what input resolution should be used for processing by the action recognition model, with the goal of improving both accuracy and efficiency.
OnlineAugment: Online Data Augmentation with Less Domain Knowledge
First is that most if not all modern augmentation search methods are offline and learning policies are isolated from their usage.
StarNet: towards Weakly Supervised Few-Shot Object Detection
Few-shot detection and classification have advanced significantly in recent years.
TAFSSL: Task-Adaptive Feature Sub-Space Learning for few-shot classification
The field of Few-Shot Learning (FSL), or learning from very few (typically $1$ or $5$) examples per novel class (unseen during training), has received a lot of attention and significant performance advances in the recent literature.
A Broader Study of Cross-Domain Few-Shot Learning
Extensive experiments on the proposed benchmark are performed to evaluate state-of-art meta-learning approaches, transfer learning approaches, and newer methods for cross-domain few-shot learning.
cross-domain few-shot learning
Few-Shot Image Classification
+1
MetAdapt: Meta-Learned Task-Adaptive Architecture for Few-Shot Classification
In this work, we propose to employ tools inspired by the Differentiable Neural Architecture Search (D-NAS) literature in order to optimize the architecture for FSL without over-fitting.
Baby steps towards few-shot learning with multiple semantics
Learning from one or few visual examples is one of the key capabilities of humans since early infancy, but is still a significant challenge for modern AI systems.
Ranked #9 on
Few-Shot Image Classification
on Mini-ImageNet - 1-Shot Learning
(using extra training data)
RepMet: Representative-Based Metric Learning for Classification and Few-Shot Object Detection
Distance metric learning (DML) has been successfully applied to object classification, both in the standard regime of rich training data and in the few-shot scenario, where each category is represented by only a few examples.
LaSO: Label-Set Operations networks for multi-label few-shot learning
We conduct numerous experiments showing promising results for the label-set manipulation capabilities of the proposed approach, both directly (using the classification and retrieval metrics), and in the context of performing data augmentation for multi-label few-shot learning.
Co-regularized Alignment for Unsupervised Domain Adaptation
Deep neural networks, trained with large amount of labeled data, can fail to generalize well when tested with examples from a \emph{target domain} whose distribution differs from the training data distribution, referred as the \emph{source domain}.
Delta-encoder: an effective sample synthesis method for few-shot object recognition
Our approach is based on a modified auto-encoder, denoted Delta-encoder, that learns to synthesize new samples for an unseen category just by seeing few examples from it.
RepMet: Representative-based metric learning for classification and one-shot object detection
Distance metric learning (DML) has been successfully applied to object classification, both in the standard regime of rich training data and in the few-shot scenario, where each category is represented by only a few examples.
Fine-Grained Recognition of Thousands of Object Categories With Single-Example Training
We approach the problem of fast detection and recognition of a large number (thousands) of object categories while training on a very limited amount of examples, usually one per category.
Using body-anchored priors for identifying actions in single images
The task is easy for humans but difficult for current approaches to object recognition, because action instances may be similar in terms of body pose, and often require detailed examination of relations between participating objects and body parts in order to be recognized.
