Search Results for author:
Found 22 papers, 11 papers with code
Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks
Experiments demonstrate that MAD leads to consistent gains in the low-shot, domain-shifted, and fully-supervised conditions on VCR, SNLI-VE, and VQA, achieving SOTA performance on VCR compared to other single models pretrained with image-text data.
CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
Experiments demonstrate that our proposed CLIP-TD leads to exceptional gains in the low-shot (up to 51. 9%) and domain-shifted (up to 71. 3%) conditions of VCR, while simultaneously improving performance under standard fully-supervised conditions (up to 2%), achieving state-of-art performance on VCR compared to other single models that are pretrained with image-text data only.
CLIP-Event: Connecting Text and Images with Event Structures
Vision-language (V+L) pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text.
RegionCLIP: Region-based Language-Image Pretraining
However, we show that directly applying such models to recognize image regions for object detection leads to poor performance due to a domain shift: CLIP was trained to match an image as a whole to a text description, without capturing the fine-grained alignment between image regions and text spans.
BEVT: BERT Pretraining of Video Transformers
This design is motivated by two observations: 1) transformers learned on image datasets provide decent spatial priors that can ease the learning of video transformers, which are often times computationally-intensive if trained from scratch; 2) discriminative clues, i. e., spatial and temporal information, needed to make correct predictions vary among different videos due to large intra-class and inter-class variations.
Ranked #3 on
Action Recognition
on Diving-48
Florence: A New Foundation Model for Computer Vision
Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
Ranked #1 on
Action Recognition In Videos
on Kinetics-600
MA-CLIP: Towards Modality-Agnostic Contrastive Language-Image Pre-training
Large-scale multimodal contrastive pretraining has demonstrated great utility to support high performance in a range of downstream tasks by mapping multiple modalities into a shared embedding space.
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task.
CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
This work concerns video-language pre-training and representation learning.
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
Vision-and-language pre-training has achieved impressive success in learning multimodal representations between vision and language.
Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle.
Ranked #4 on
Visual Question Answering
on MSRVTT-QA
(using extra training data)
Temporally Guided Articulated Hand Pose Tracking in Surgical Videos
Additionally, we collect the first dataset, Surgical Hands, that provides multi-instance articulated hand pose annotations for in-vivo videos.
Cluster-Former: Clustering-based Sparse Transformer for Question Answering
Transformer has become ubiquitous in the deep learning field.
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
Transformer has become ubiquitous in the deep learning field.
Ranked #1 on
Question Answering
on Quasart-T
Unified Vision-Language Pre-Training for Image Captioning and VQA
The model is unified in that (1) it can be fine-tuned for either vision-language generation (e. g., image captioning) or understanding (e. g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models.
Ranked #1 on
Image Captioning
on Flickr30k Captions test
Grounded Video Description
Our dataset, ActivityNet-Entities, augments the challenging ActivityNet Captions dataset with 158k bounding box annotations, each grounding a noun phrase.
Dynamic Graph Modules for Modeling Object-Object Interactions in Activity Recognition
Video action recognition, a critical problem in video understanding, has been gaining increasing attention.
Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
A naive extension of this approach to the video domain is to treat the entire segment as a bag of spatial object proposals.
End-to-End Dense Video Captioning with Masked Transformer
To address this problem, we propose an end-to-end transformer model for dense video captioning.
Ranked #4 on
Video Captioning
on YouCook2
Towards Automatic Learning of Procedures from Web Instructional Videos
To answer this question, we introduce the problem of procedure segmentation--to segment a video procedure into category-independent procedure segments.
Watch What You Just Said: Image Captioning with Text-Conditional Attention
Attention mechanisms have attracted considerable interest in image captioning due to its powerful performance.
Multi-agent Reinforcement Learning with Sparse Interactions by Negotiation and Knowledge Transfer
In this paper, a novel algorithm named negotiation-based MARL with sparse interactions (NegoSI) is presented.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
