Search Results for author:
Found 61 papers, 29 papers with code
Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models
Vision and Language (VL) models offer an effective method for aligning representation spaces of images and text, leading to numerous applications such as cross-modal retrieval, visual question answering, captioning, and more.
Going Beyond Nouns With Vision & Language Models Using Synthetic Data
We contribute Synthetic Visual Concepts (SyViC) - a million-scale synthetic dataset and data generation codebase allowing to generate additional suitable data to improve VLC understanding and compositional reasoning of VL models.
Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Models
This paper proposes KD-NAS, the use of Neural Architecture Search (NAS) guided by the Knowledge Distillation process to find the optimal student model for distillation from a teacher, for a given natural language task.
MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge
We adapt a VL model for zero-shot and few-shot action recognition using a collection of unlabeled videos and an unpaired action dictionary.
Ranked #2 on
Zero-Shot Action Recognition
on Kinetics
Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning
Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks.
Learning to Grow Pretrained Models for Efficient Transformer Training
Scaling transformers has led to significant breakthroughs in many domains, leading to a paradigm in which larger versions of existing models are trained and released on a periodic basis.
Energy Transformer
Transformers have become the de facto models of choice in machine learning, typically leading to impressive performance on many applications.
Teaching Structured Vision & Language Concepts to Vision & Language Models
Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks.
Synthetic Pre-Training Tasks for Neural Machine Translation
Pre-training models with large crawled corpora can lead to issues such as toxicity and bias, as well as copyright and privacy concerns.
CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning
Our experiments show that we outperform the current SOTA method DualPrompt on established benchmarks by as much as 4. 5% in average final accuracy.
Teaching Structured Vision&Language Concepts to Vision&Language Models
Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks.
ConStruct-VL: Data-Free Continual Structured VL Concepts Learning
This leads to reasoning mistakes, which need to be corrected as they occur by teaching VL models the missing SVLC skills; often this must be done using private data where the issue was found, which naturally leads to a data-free continual (no task-id) VL learning setting.
Semi-Supervised Domain Adaptation with Auto-Encoder via Simultaneous Learning
We present a new semi-supervised domain adaptation framework that combines a novel auto-encoder-based domain adaptation model with a simultaneous learning scheme providing stable improvements over state-of-the-art domain adaptation models.
FETA: Towards Specializing Foundation Models for Expert Task Applications
However, as we show in this paper, FMs still have poor out-of-the-box performance on expert tasks (e. g. retrieval of car manuals technical illustrations from language queries), data for which is either unseen or belonging to a long-tail part of the data distribution of the huge datasets used for FM pre-training.
Ranked #1 on
Image-to-Text Retrieval
on FETA Car-Manuals
VALHALLA: Visual Hallucination for Machine Translation
In particular, given a source sentence an autoregressive hallucination transformer is used to predict a discrete visual representation from the input text, and the combined text and hallucinated representations are utilized to obtain the target translation.
A Maximal Correlation Framework for Fair Machine Learning
As machine learning algorithms grow in popularity and diversify to many industries, ethical and legal concerns regarding their fairness have become increasingly relevant.
Task2Sim: Towards Effective Pre-Training and Transfer From Synthetic Data
It is thus better to tailor synthetic pre-training data to a specific downstream task, for best performance.
Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data
It is thus better to tailor synthetic pre-training data to a specific downstream task, for best performance.
Cascaded Multilingual Audio-Visual Learning from Videos
In this paper, we explore self-supervised audio-visual models that learn from instructional videos.
Selective Regression Under Fairness Criteria
Selective regression allows abstention from prediction if the confidence to make an accurate prediction is not sufficient.
Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing
Unsupervised domain adaptation which aims to adapt models trained on a labeled source domain to a completely unlabeled target domain has attracted much attention in recent years.
Dynamic Network Quantization for Efficient Video Inference
Deep convolutional networks have recently achieved great success in video recognition, yet their practical realization remains a challenge due to the large amount of computational resources required to achieve robust recognition.
Can An Image Classifier Suffice For Action Recognition?
We explore a new perspective on video understanding by casting the video recognition problem as an image recognition task.
IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
The self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision.
Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
As the base dataset and unlabeled dataset are from different domains, projecting the target images in the class-domain of the base dataset with a fixed pretrained model might be sub-optimal.
RegionViT: Regional-to-Local Attention for Vision Transformers
The regional-to-local attention includes two steps: first, the regional self-attention extract global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the associated local tokens via self-attention.
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition
Specifically, given a video segment, a multi-modal policy network is used to decide what modalities should be used for processing by the recognition model, with the goal of improving both accuracy and efficiency.
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos
Multimodal self-supervised learning is getting more and more attention as it allows not only to train large networks without human supervision but also to search and retrieve data across various modalities.
Detector-Free Weakly Supervised Grounding by Separation
In this work, we focus on the task of Detector-Free WSG (DF-WSG) to solve WSG without relying on a pre-trained detector.
Ranked #1 on
Phrase Grounding
on Visual Genome
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
To this end, we propose a dual-branch transformer to combine image patches (i. e., tokens in a transformer) of different sizes to produce stronger image features.
Ranked #419 on
Image Classification
on ImageNet
A Broad Study on the Transferability of Visual Representations with Contrastive Learning
Tremendous progress has been made in visual representation learning, notably with the recent success of self-supervised contrastive learning methods.
Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
Second, to effectively transfer knowledge, we develop a dynamic block swapping method by randomly replacing the blocks in the lower-precision student network with the corresponding blocks in the higher-precision teacher network.
VA-RED$^2$: Video Adaptive Redundancy Reduction
An inherent property of real-world videos is the high correlation of information across frames which can translate into redundancy in either temporal or spatial feature maps of the models, or both.
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
Temporal modelling is the key for efficient video action recognition.
Semi-Supervised Action Recognition with Temporal Contrastive Learning
We approach this problem by learning a two-pathway temporal contrastive model using unlabeled videos at two different speeds leveraging the fact that changing video speed does not change an action.
A Maximal Correlation Approach to Imposing Fairness in Machine Learning
As machine learning algorithms grow in popularity and diversify to many industries, ethical and legal concerns regarding their fairness have become increasingly relevant.
Select, Label, and Mix: Learning Discriminative Invariant Feature Representations for Partial Domain Adaptation
Partial domain adaptation which assumes that the unknown target label space is a subset of the source label space has attracted much attention in computer vision.
Ranked #1 on
Partial Domain Adaptation
on Office-31
Large Scale Neural Architecture Search with Polyharmonic Splines
Neural Architecture Search (NAS) is a powerful tool to automatically design deep neural networks for many tasks, including image classification.
Deep Analysis of CNN-based Spatio-temporal Representations for Action Recognition
In recent years, a number of approaches based on 2D or 3D convolutional neural networks (CNN) have emerged for video action recognition, achieving state-of-the-art results on several large-scale benchmark datasets.
Measurement-driven Security Analysis of Imperceptible Impersonation Attacks
The emergence of Internet of Things (IoT) brings about new security challenges at the intersection of cyber and physical spaces.
Adversarial Knowledge Transfer from Unlabeled Data
While machine learning approaches to visual recognition offer great promise, most of the existing methods rely heavily on the availability of large quantities of labeled training data.
Mitigating Dataset Imbalance via Joint Generation and Classification
In this work we address these questions from the perspective of dataset imbalance resulting out of severe under-representation of annotated training data for certain classes and its effect on both deep classification and generation methods.
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition
Specifically, given a video frame, a policy network is used to decide what input resolution should be used for processing by the action recognition model, with the goal of improving both accuracy and efficiency.
Camera On-boarding for Person Re-identification using Hypothesis Transfer Learning
Most of the existing approaches for person re-identification consider a static setting where the number of cameras in the network is fixed.
NASTransfer: Analyzing Architecture Transferability in Large Scale Neural Architecture Search
The typical way of conducting large scale NAS is to search for an architectural building block on a small dataset (either using a proxy set from the large dataset or a completely different small scale dataset) and then transfer the block to a larger dataset.
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos
Further, we propose a tri-modal model that jointly processes raw audio, video, and text captions from videos to learn a multi-modal semantic embedding space useful for text-video retrieval.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Non-Adversarial Video Synthesis with Learned Priors
Different from these methods, we focus on the problem of generating videos from latent noise vectors, without any reference input frames.
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
Multi-task learning is an open and challenging problem in computer vision.
Ranked #95 on
Semantic Segmentation
on NYU Depth v2
Estimating Skin Tone and Effects on Classification Performance in Dermatology Datasets
We find that the majority of the data in the the two datasets have ITA values between 34. 5{\deg} and 48{\deg}, which are associated with lighter skin, and is consistent with under-representation of darker skinned populations in these datasets.
Exploiting Global Camera Network Constraints for Unsupervised Video Person Re-identification
Additionally, a cross-view matching strategy followed by global camera network constraints is proposed to explore the matching relationships across the entire camera network.
Webly Supervised Joint Embedding for Cross-Modal lmage-Text Retrieval
Inspired by the recent success of web-supervised learning in deep neural networks, we capitalize on readily-available web images with noisy annotations to learn robust image-text joint representation.
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval
Inspired by the recent success of webly supervised learning in deep neural networks, we capitalize on readily-available web images with noisy annotations to learn robust image-text joint representation.
Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias
While machine learning approaches to visual emotion recognition offer great promise, current methods consider training and testing models on small scale datasets covering limited visual emotion concepts.
FFNet: Video Fast-Forwarding via Reinforcement Learning
The first group is supported by video summarization techniques, which require processing of the entire video to select an important subset for showing to users.
Weakly Supervised Summarization of Web Videos
Casting the problem as a weakly supervised learning problem, we propose a flexible deep 3D CNN architecture to learn the notion of importance using only video-level annotation, and without any human-crafted training data.
Collaborative Summarization of Topic-Related Videos
Large collections of videos are grouped into clusters by a topic keyword, such as Eiffel Tower or Surfing, with many important visual concepts repeating across them.
Multi-View Surveillance Video Summarization via Joint Embedding and Sparse Optimization
In this paper, with the aim of summarizing multi-view videos, we introduce a novel unsupervised framework via joint embedding and sparse representative selection.
Unsupervised Adaptive Re-identification in Open World Dynamic Camera Networks
Most approaches have neglected the dynamic and open world nature of the re-identification problem, where a new camera may be temporarily inserted into an existing system to get additional information.
Diversity-aware Multi-Video Summarization
Most video summarization approaches have focused on extracting a summary from a single video; we propose an unsupervised framework for summarizing a collection of videos.
Video Summarization in a Multi-View Camera Network
While most existing video summarization approaches aim to extract an informative summary of a single video, we propose a novel framework for summarizing multi-view videos by exploiting both intra- and inter-view content correlations in a joint embedding space.
Continuous Adaptation of Multi-Camera Person Identification Models through Sparse Non-redundant Representative Selection
We demonstrate the effectiveness of our approach on multi-camera person re-identification datasets, to demonstrate the feasibility of learning online classification models in multi-camera big data applications.
