Search Results for author:
Found 74 papers, 46 papers with code
MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge
We adapt a VL model for zero-shot and few-shot action recognition using a collection of unlabeled videos and an unpaired action dictionary.
Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning
Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks.
Learning to Grow Pretrained Models for Efficient Transformer Training
Scaling transformers has led to significant breakthroughs in many domains, leading to a paradigm in which larger versions of existing models are trained and released on a periodic basis.
Synthetic Pre-Training Tasks for Neural Machine Translation
Our second approach explores the effect of pre-training on procedurally generated synthetic parallel data that does not depend on any real human language corpus.
Procedural Image Programs for Representation Learning
Learning image representations using synthetic data allows training neural networks without some of the concerns associated with real images, such as privacy and bias.
Exploring Consistency in Cross-Domain Transformer for Domain Adaptive Semantic Segmentation
We propose to perform adaptation on attention maps with cross-domain attention layers that share features between the source and the target domains.
CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning
Our experiments show that we outperform the current SOTA method DualPrompt on established benchmarks by as much as 5. 4% in average accuracy.
Teaching Structured Vision&Language Concepts to Vision&Language Models
Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks.
ConStruct-VL: Data-Free Continual Structured VL Concepts Learning
We, therefore, propose a data-free method comprised of a new approach of Adversarial Pseudo-Replay (APR) which generates adversarial reminders of past tasks from past task models.
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval
Inspired by the fact that English text-video retrieval outperforms other languages, we train a student model using input text in different languages to match the cross-modal predictions from teacher models using input text in English.
FETA: Towards Specializing Foundation Models for Expert Task Applications
However, as we show in this paper, FMs still have poor out-of-the-box performance on expert tasks (e. g. retrieval of car manuals technical illustrations from language queries), data for which is either unseen or belonging to a long-tail part of the data distribution of the huge datasets used for FM pre-training.
Ranked #1 on
Image-to-Text Retrieval
on FETA Car-Manuals
VALHALLA: Visual Hallucination for Machine Translation
In particular, given a source sentence an autoregressive hallucination transformer is used to predict a discrete visual representation from the input text, and the combined text and hallucinated representations are utilized to obtain the target translation.
SimVQA: Exploring Simulated Environments for Visual Question Answering
By exploiting 3D and physics simulation platforms, we provide a pipeline to generate synthetic data to expand and replace type-specific questions and answers without risking the exposure of sensitive or personal data that might be present in real images.
Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval
Multi-modal learning from video data has seen increased attention recently as it allows to train semantically meaningful embeddings without human annotation enabling tasks like zero-shot retrieval and classification.
Unsupervised Domain Generalization by Learning a Bridge Across Domains
The ability to generalize learned representations across significantly different visual domains, such as between real photos, clipart, paintings, and sketches, is a fundamental capacity of the human visual system.
Targeted Supervised Contrastive Learning for Long-Tailed Recognition
This forces all classes, including minority classes, to maintain a uniform distribution in the feature space, improves class boundaries, and provides better generalization even in the presence of long-tail data.
Ranked #15 on
Long-tail Learning
on CIFAR-10-LT (ρ=100)
Cascaded Multilingual Audio-Visual Learning from Videos
In this paper, we explore self-supervised audio-visual models that learn from instructional videos.
Dynamic Network Quantization for Efficient Video Inference
Deep convolutional networks have recently achieved great success in video recognition, yet their practical realization remains a challenge due to the large amount of computational resources required to achieve robust recognition.
Separating Skills and Concepts for Novel Visual Question Answering
Generalization to out-of-distribution data has been a problem for Visual Question Answering (VQA) models.
IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
The self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision.
Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
As the base dataset and unlabeled dataset are from different domains, projecting the target images in the class-domain of the base dataset with a fixed pretrained model might be sub-optimal.
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition
Specifically, given a video segment, a multi-modal policy network is used to decide what modalities should be used for processing by the recognition model, with the goal of improving both accuracy and efficiency.
Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions
With this in mind, the descriptions people generate for videos of different dynamic events can greatly improve our understanding of the key information of interest in each video.
Pseudo-IoU: Improving Label Assignment in Anchor-Free Object Detection
Current anchor-free object detectors are quite simple and effective yet lack accurate label assignment methods, which limits their potential in competing with classic anchor-based models that are supported by well-designed assignment methods based on the Intersection-over-Union~(IoU) metric.
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos
Multimodal self-supervised learning is getting more and more attention as it allows not only to train large networks without human supervision but also to search and retrieve data across various modalities.
Detector-Free Weakly Supervised Grounding by Separation
In this work, we focus on the task of Detector-Free WSG (DF-WSG) to solve WSG without relying on a pre-trained detector.
A Broad Study on the Transferability of Visual Representations with Contrastive Learning
Tremendous progress has been made in visual representation learning, notably with the recent success of self-supervised contrastive learning methods.
Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
Second, to effectively transfer knowledge, we develop a dynamic block swapping method by randomly replacing the blocks in the lower-precision student network with the corresponding blocks in the higher-precision teacher network.
VA-RED$^2$: Video Adaptive Redundancy Reduction
An inherent property of real-world videos is the high correlation of information across frames which can translate into redundancy in either temporal or spatial feature maps of the models, or both.
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
Temporal modelling is the key for efficient video action recognition.
Semi-Supervised Action Recognition with Temporal Contrastive Learning
We approach this problem by learning a two-pathway temporal contrastive model using unlabeled videos at two different speeds leveraging the fact that changing video speed does not change an action.
A Maximal Correlation Approach to Imposing Fairness in Machine Learning
As machine learning algorithms grow in popularity and diversify to many industries, ethical and legal concerns regarding their fairness have become increasingly relevant.
Addressing Feature Suppression in Unsupervised Visual Representations
However, contrastive learning is susceptible to feature suppression, i. e., it may discard important information relevant to the task of interest, and learn irrelevant features.
Fine-grained Angular Contrastive Learning with Coarse Labels
A very practical example of C2FS is when the target classes are sub-classes of the training classes.
Select, Label, and Mix: Learning Discriminative Invariant Feature Representations for Partial Domain Adaptation
Partial domain adaptation which assumes that the unknown target label space is a subset of the source label space has attracted much attention in computer vision.
Ranked #1 on
Partial Domain Adaptation
on Office-31
Learning from Lexical Perturbations for Consistent Visual Question Answering
Existing Visual Question Answering (VQA) models are often fragile and sensitive to input variations.
Large Scale Neural Architecture Search with Polyharmonic Splines
Neural Architecture Search (NAS) is a powerful tool to automatically design deep neural networks for many tasks, including image classification.
Deep Analysis of CNN-based Spatio-temporal Representations for Action Recognition
In recent years, a number of approaches based on 2D or 3D convolutional neural networks (CNN) have emerged for video action recognition, achieving state-of-the-art results on several large-scale benchmark datasets.
Mitigating Dataset Imbalance via Joint Generation and Classification
In this work we address these questions from the perspective of dataset imbalance resulting out of severe under-representation of annotated training data for certain classes and its effect on both deep classification and generation methods.
We Have So Much In Common: Modeling Semantic Relational Set Abstractions in Videos
This allows our model to perform cognitive tasks such as set abstraction (which general concept is in common among a set of videos?
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition
Specifically, given a video frame, a policy network is used to decide what input resolution should be used for processing by the action recognition model, with the goal of improving both accuracy and efficiency.
OnlineAugment: Online Data Augmentation with Less Domain Knowledge
First is that most if not all modern augmentation search methods are offline and learning policies are isolated from their usage.
NASTransfer: Analyzing Architecture Transferability in Large Scale Neural Architecture Search
The typical way of conducting large scale NAS is to search for an architectural building block on a small dataset (either using a proxy set from the large dataset or a completely different small scale dataset) and then transfer the block to a larger dataset.
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos
Further, we propose a tri-modal model that jointly processes raw audio, video, and text captions from videos to learn a multi-modal semantic embedding space useful for text-video retrieval.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Differential Treatment for Stuff and Things: A Simple Unsupervised Domain Adaptation Method for Semantic Segmentation
We consider the problem of unsupervised domain adaptation for semantic segmentation by easing the domain shift between the source domain (synthetic data) and the target domain (real data) in this work.
Ranked #8 on
Semantic Segmentation
on DensePASS
StarNet: towards Weakly Supervised Few-Shot Object Detection
Few-shot detection and classification have advanced significantly in recent years.
TAFSSL: Task-Adaptive Feature Sub-Space Learning for few-shot classification
The field of Few-Shot Learning (FSL), or learning from very few (typically $1$ or $5$) examples per novel class (unseen during training), has received a lot of attention and significant performance advances in the recent literature.
A Broader Study of Cross-Domain Few-Shot Learning
Extensive experiments on the proposed benchmark are performed to evaluate state-of-art meta-learning approaches, transfer learning approaches, and newer methods for cross-domain few-shot learning.
cross-domain few-shot learning
Few-Shot Image Classification
+1
MetAdapt: Meta-Learned Task-Adaptive Architecture for Few-Shot Classification
In this work, we propose to employ tools inspired by the Differentiable Neural Architecture Search (D-NAS) literature in order to optimize the architecture for FSL without over-fitting.
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
Multi-task learning is an open and challenging problem in computer vision.
Ranked #66 on
Semantic Segmentation
on NYU Depth v2
Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understanding
Videos capture events that typically contain multiple sequential, and simultaneous, actions even in the span of only a few seconds.
Baby steps towards few-shot learning with multiple semantics
Learning from one or few visual examples is one of the key capabilities of humans since early infancy, but is still a significant challenge for modern AI systems.
Ranked #9 on
Few-Shot Image Classification
on Mini-ImageNet - 1-Shot Learning
(using extra training data)
Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
We provide a detailed analysis of the characteristics of the Fashion IQ data, and present a transformer-based user simulator and interactive image retriever that can seamlessly integrate visual attributes with image features, user feedback, and dialog history, leading to improved performance over the state of the art in dialog-based image retrieval.
Examining Interpretable Feature Relationships in Deep Networks for Action recognition
A number of recent methods to understand neural networks have focused on quantifying the role of individual features.
LaSO: Label-Set Operations networks for multi-label few-shot learning
We conduct numerous experiments showing promising results for the label-set manipulation capabilities of the proposed approach, both directly (using the classification and retrieval metrics), and in the context of performing data augmentation for multi-label few-shot learning.
Depthwise Convolution is All You Need for Learning Multiple Visual Domains
A model aware of the relationships between different domains can also be trained to work on new domains with less resources.
SpotTune: Transfer Learning through Adaptive Fine-tuning
Transfer learning, which allows a source task to affect the inductive bias of the target task, is widely used in computer vision.
Co-regularized Alignment for Unsupervised Domain Adaptation
Deep neural networks, trained with large amount of labeled data, can fail to generalize well when tested with examples from a \emph{target domain} whose distribution differs from the training data distribution, referred as the \emph{source domain}.
Decoupled Classification Refinement: Hard False Positive Suppression for Object Detection
In particular, DCR places a separate classification network in parallel with the localization network (base detector).
Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition
The proposed approach demonstrates improvement of model efficiency and performance on both object recognition and speech recognition tasks, using popular architectures including ResNet and ResNeXt.
Delta-encoder: an effective sample synthesis method for few-shot object recognition
Our approach is based on a modified auto-encoder, denoted Delta-encoder, that learns to synthesize new samples for an unseen category just by seeing few examples from it.
RepMet: Representative-based metric learning for classification and one-shot object detection
Distance metric learning (DML) has been successfully applied to object classification, both in the standard regime of rich training data and in the few-shot scenario, where each category is represented by only a few examples.
Collaborative Human-AI (CHAI): Evidence-Based Interpretable Melanoma Classification in Dermoscopic Images
Quantitative relevance of results, according to non-expert similarity, as well as localized image regions, are also significantly improved.
Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation
Random data augmentation is a critical technique to avoid overfitting in training deep neural network models.
Ranked #3 on
Pose Estimation
on Leeds Sports Poses
Segmentation of both Diseased and Healthy Skin from Clinical Photographs in a Primary Care Setting
The stark performance improvement with fine-tuning compared to direct transfer and direct training emphasizes both the need for adequate representative data of diseased skin, and the utility of other publicly available data sources for this task.
Learning to Separate Object Sounds by Watching Unlabeled Video
Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video.
Revisiting RCNN: On Awakening the Classification Power of Faster RCNN
Recent region-based object detectors are usually built with separate classification and localization branches on top of shared feature extraction networks.
Improving Object Detection from Scratch via Gated Feature Reuse
In this paper, we present a simple and parameter-efficient drop-in module for one-stage object detectors like SSD when learning from scratch (i. e., without pre-trained models).
BlockDrop: Dynamic Inference Paths in Residual Networks
Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world applications.
Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification
Multi-task learning aims to improve generalization performance of multiple prediction tasks by appropriately sharing relevant information across them.
S3Pool: Pooling with Stochastic Spatial Sampling
We view the pooling operation in CNNs as a two-step procedure: first, a pooling window (e. g., $2\times 2$) slides over the feature map with stride one which leaves the spatial resolution intact, and second, downsampling is performed by selecting one pixel from each non-overlapping pooling window in an often uniform and deterministic (e. g., top-left) manner.
Generative Adversarial Networks as Variational Training of Energy Based Models
We propose VGAN, which works by minimizing a variational lower bound of the negative log likelihood (NLL) of an energy based model (EBM), where the model density $p(\mathbf{x})$ is approximated by a variational distribution $q(\mathbf{x})$ that is easy to sample from.
Deep Domain Adaptation for Describing People Based on Fine-Grained Clothing Attributes
We address the problem of describing people based on fine-grained clothing attributes.
Efficient Maximum Appearance Search for Large-Scale Object Detection
In recent years, efficiency of large-scale object detection has arisen as an important topic due to the exponential growth in the size of benchmark object detection datasets.
