Search Results for author:
Found 38 papers, 23 papers with code
Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception
IMP achieves competitive performance on a wide range of downstream tasks including image classification, video classification, image-text, and video-text retrieval.
Ranked #1 on
Zero-Shot Action Recognition
on UCF101
(using extra training data)
Towards Understanding the Effect of Pretraining Label Granularity
In this paper, we study how pretraining label granularity affects the generalization of deep neural networks in image classification tasks.
Unified Visual Relationship Detection with Vision and Language Models
To address this challenge, we propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models (VLMs).
Human-Object Interaction Detection
Relationship Detection
+2
A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models
In particular, we ask "Given a large pool of prompts, can we automatically score the prompts and ensemble those that are most suitable for a particular downstream dataset, without needing access to labeled validation data?".
Train-Once-for-All Personalization
We study the problem of how to train a "personalization-friendly" model such that given only the task descriptions, the model can be adapted to different end-users' needs, e. g., for accurately classifying different subsets of objects.
MovieCLIP: Visual Scene Recognition in Movies
Longform media such as movies have complex narrative structures, with events spanning a rich variety of ambient visual scenes.
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
We present F-VLM, a simple open-vocabulary object detection method built upon Frozen Vision and Language Models.
Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models
Utilizing vision and language models (VLMs) pre-trained on large-scale image-text pairs is becoming a promising paradigm for open-vocabulary visual recognition.
Ranked #1 on
Zero-Shot Action Recognition
on HMDB51
Surrogate Gap Minimization Improves Sharpness-Aware Training
Instead, we define a \textit{surrogate gap}, a measure equivalent to the dominant eigenvalue of Hessian at a local minimum when the radius of the neighborhood (to derive the perturbed loss) is small.
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
We propose OpenSeg to address the above issue while still making use of scalable image-level supervision of captions.
Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text
The experiments show that the resultant unified foundation transformer works surprisingly well on both the vision-only and text-only tasks, and the proposed knowledge distillation and gradient masking strategy can effectively lift the performance to approach the level of separately-trained models.
Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision
Modern self-supervised learning algorithms typically enforce persistency of instance representations across views.
Exploring Temporal Granularity in Self-Supervised Video Representation Learning
The training objective consists of two parts: a fine-grained temporal learning objective to maximize the similarity between corresponding temporal embeddings in the short clip and the long clip, and a persistent temporal learning objective to pull together global embeddings of the two clips.
Revisiting 3D ResNets for Video Recognition
A recent work from Bello shows that training and scaling strategies may be more significant than model architectures for visual recognition.
Ranked #37 on
Action Classification
on Kinetics-600
Federated Multi-Target Domain Adaptation
We further take the server-client and inter-client domain shifts into account and pose a domain adaptation problem with one source (centralized server data) and multiple targets (distributed client data).
SITTA: Single Image Texture Translation for Data Augmentation
In this paper, we propose and explore the problem of image translation for data augmentation.
Bridging the Gap Between Object Detection and User Intent via Query-Modulation
When interacting with objects through cameras, or pictures, users often have a specific intent.
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
On COCO, ViLD outperforms the previous state-of-the-art by 4. 8 on novel AP and 11. 4 on overall AP.
Ranked #1 on
Open Vocabulary Object Detection
on LVIS v1.0
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval.
Ranked #8 on
Action Classification
on Moments in Time
(using extra training data)
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation
Our baseline model outperforms the LVIS 2020 Challenge winning entry by +3. 6 mask AP on rare categories.
Ranked #2 on
Semantic Segmentation
on PASCAL VOC 2012 val
Efficient Scale-Permuted Backbone with Learned Resource Distribution
Furthermore, SpineNet is built with a uniform resource distribution over operations.
Small-floating Target Detection in Sea Clutter via Visual Feature Classifying in the Time-Doppler Spectra
It is challenging to detect small-floating object in the sea clutter for a surface radar.
Spatiotemporal Contrastive Video Representation Learning
Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away.
Ranked #1 on
Self-Supervised Action Recognition
on Kinetics-600
Rethinking Pre-training and Self-training
For example, on the COCO object detection dataset, pre-training benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data.
Ranked #1 on
Semantic Segmentation
on PASCAL VOC 2012 val
Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset
In this work we explore the task of instance segmentation with attribute localization, which unifies instance segmentation (detect and segment each object instance) and fine-grained visual attribute categorization (recognize one or multiple attributes).
Fine-Grained Visual Categorization
Fine-Grained Visual Recognition
+3
Measuring Dataset Granularity
We also investigate the interplay between dataset granularity with a variety of factors and find that fine-grained datasets are more difficult to learn from, more difficult to transfer to, more difficult to perform few-shot learning with, and more vulnerable to adversarial attacks.
SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
We propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search.
Ranked #8 on
Image Classification
on iNaturalist
The iMaterialist Fashion Attribute Dataset
The dataset is constructed from over one million fashion images with a label space that includes 8 groups of 228 fine-grained attributes in total.
Class-Balanced Loss Based on Effective Number of Samples
We design a re-weighting scheme that uses the effective number of samples for each class to re-balance the loss, thereby yielding a class-balanced loss.
Ranked #2 on
Long-tail Learning
on EGTEA
Learning Single-View 3D Reconstruction with Limited Pose Supervision
It is expensive to label images with 3D structure or precise camera pose.
Learning to Evaluate Image Captioning
To address these two challenges, we propose a novel learning based discriminative evaluation metric that is directly trained to distinguish between human and machine-generated captions.
Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning
We propose a measure to estimate domain similarity via Earth Mover's Distance and demonstrate that transfer learning benefits from pre-training on a source domain that is similar to the target domain by this measure.
Ranked #27 on
Fine-Grained Image Classification
on CUB-200-2011
Fine-Grained Image Classification
Fine-Grained Visual Categorization
+1
The iNaturalist Species Classification and Detection Dataset
Existing image classification datasets used in computer vision tend to have a uniform distribution of images across object categories.
Ranked #7 on
Image Classification
on iNaturalist
Kernel Pooling for Convolutional Neural Networks
We demonstrate how to approximate kernels such as Gaussian RBF up to a given order using compact explicit feature maps in a parameter-free manner.
Collaborative Metric Learning
Metric learning algorithms produce distance metrics that capture the important relationships among data.
Ranked #1 on
Recommendation Systems
on MovieLens 20M
(Recall@100 metric)
Fine-grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loop
To demonstrate the effectiveness of the proposed framework, we bootstrap a fine-grained flower dataset with 620 categories from Instagram images.
Learning Deep Representations for Ground-to-Aerial Geolocalization
Most approaches predict the location of a query image by matching to ground-level images with known locations (e. g., street-view data).
Building A Large Concept Bank for Representing Events in Video
In this paper, we propose to build Concept Bank, the largest concept library consisting of 4, 876 concepts specifically designed to cover 631 real-world events.
