Search Results for author:
Found 39 papers, 25 papers with code
Spatiotemporal Contrastive Video Representation Learning
Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away.
Ranked #1 on
Self-Supervised Action Recognition
on Kinetics-600
The iNaturalist Species Classification and Detection Dataset
Existing image classification datasets used in computer vision tend to have a uniform distribution of images across object categories.
Ranked #8 on
Image Classification
on iNaturalist
Revisiting 3D ResNets for Video Recognition
A recent work from Bello shows that training and scaling strategies may be more significant than model architectures for visual recognition.
Ranked #40 on
Action Classification
on Kinetics-600
Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision
Modern self-supervised learning algorithms typically enforce persistency of instance representations across views.
VideoGLUE: Video General Understanding Evaluation of Foundation Models
We evaluate existing foundation models video understanding capabilities using a carefully designed experiment protocol consisting of three hallmark tasks (action recognition, temporal localization, and spatiotemporal localization), eight datasets well received by the community, and four adaptation methods tailoring a foundation model (FM) for a downstream task.
SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
We propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search.
Ranked #9 on
Image Classification
on iNaturalist
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation
Our baseline model outperforms the LVIS 2020 Challenge winning entry by +3. 6 mask AP on rare categories.
Ranked #1 on
Object Detection
on PASCAL VOC 2007
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval.
Ranked #3 on
Zero-Shot Video Retrieval
on YouCook2
(text-to-video Mean Rank metric)
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
We present F-VLM, a simple open-vocabulary object detection method built upon Frozen Vision and Language Models.
Surrogate Gap Minimization Improves Sharpness-Aware Training
Instead, we define a \textit{surrogate gap}, a measure equivalent to the dominant eigenvalue of Hessian at a local minimum when the radius of the neighborhood (to derive the perturbed loss) is small.
Rethinking Pre-training and Self-training
For example, on the COCO object detection dataset, pre-training benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data.
Ranked #1 on
Semantic Segmentation
on PASCAL VOC 2012 val
Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset
In this work we explore the task of instance segmentation with attribute localization, which unifies instance segmentation (detect and segment each object instance) and fine-grained visual attribute categorization (recognize one or multiple attributes).
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
On COCO, ViLD outperforms the previous state-of-the-art by 4. 8 on novel AP and 11. 4 on overall AP.
Ranked #2 on
Open Vocabulary Object Detection
on Objects365
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
We propose OpenSeg to address the above issue while still making use of scalable image-level supervision of captions.
Unified Visual Relationship Detection with Vision and Language Models
To address this challenge, we propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models (VLMs).
Human-Object Interaction Detection
Relationship Detection
+2
Class-Balanced Loss Based on Effective Number of Samples
We design a re-weighting scheme that uses the effective number of samples for each class to re-balance the loss, thereby yielding a class-balanced loss.
Ranked #2 on
Long-tail Learning
on EGTEA
Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning
We propose a measure to estimate domain similarity via Earth Mover's Distance and demonstrate that transfer learning benefits from pre-training on a source domain that is similar to the target domain by this measure.
Ranked #28 on
Fine-Grained Image Classification
on CUB-200-2011
Fine-Grained Image Classification
Fine-Grained Visual Categorization
+1
Collaborative Metric Learning
Metric learning algorithms produce distance metrics that capture the important relationships among data.
Ranked #1 on
Recommendation Systems
on MovieLens 20M
(Recall@100 metric)
Learning to Evaluate Image Captioning
To address these two challenges, we propose a novel learning based discriminative evaluation metric that is directly trained to distinguish between human and machine-generated captions.
The iMaterialist Fashion Attribute Dataset
The dataset is constructed from over one million fashion images with a label space that includes 8 groups of 228 fine-grained attributes in total.
Learning Single-View 3D Reconstruction with Limited Pose Supervision
It is expensive to label images with 3D structure or precise camera pose.
SITTA: Single Image Texture Translation for Data Augmentation
In this paper, we propose and explore the problem of image translation for data augmentation.
MovieCLIP: Visual Scene Recognition in Movies
Longform media such as movies have complex narrative structures, with events spanning a rich variety of ambient visual scenes.
Small-floating Target Detection in Sea Clutter via Visual Feature Classifying in the Time-Doppler Spectra
It is challenging to detect small-floating object in the sea clutter for a surface radar.
Measuring Dataset Granularity
We also investigate the interplay between dataset granularity with a variety of factors and find that fine-grained datasets are more difficult to learn from, more difficult to transfer to, more difficult to perform few-shot learning with, and more vulnerable to adversarial attacks.
Fine-grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loop
To demonstrate the effectiveness of the proposed framework, we bootstrap a fine-grained flower dataset with 620 categories from Instagram images.
Building A Large Concept Bank for Representing Events in Video
In this paper, we propose to build Concept Bank, the largest concept library consisting of 4, 876 concepts specifically designed to cover 631 real-world events.
Learning Deep Representations for Ground-to-Aerial Geolocalization
Most approaches predict the location of a query image by matching to ground-level images with known locations (e. g., street-view data).
Kernel Pooling for Convolutional Neural Networks
We demonstrate how to approximate kernels such as Gaussian RBF up to a given order using compact explicit feature maps in a parameter-free manner.
Efficient Scale-Permuted Backbone with Learned Resource Distribution
Furthermore, SpineNet is built with a uniform resource distribution over operations.
Bridging the Gap Between Object Detection and User Intent via Query-Modulation
When interacting with objects through cameras, or pictures, users often have a specific intent.
Federated Multi-Target Domain Adaptation
We further take the server-client and inter-client domain shifts into account and pose a domain adaptation problem with one source (centralized server data) and multiple targets (distributed client data).
Exploring Temporal Granularity in Self-Supervised Video Representation Learning
The training objective consists of two parts: a fine-grained temporal learning objective to maximize the similarity between corresponding temporal embeddings in the short clip and the long clip, and a persistent temporal learning objective to pull together global embeddings of the two clips.
Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text
The experiments show that the resultant unified foundation transformer works surprisingly well on both the vision-only and text-only tasks, and the proposed knowledge distillation and gradient masking strategy can effectively lift the performance to approach the level of separately-trained models.
Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models
Utilizing vision and language models (VLMs) pre-trained on large-scale image-text pairs is becoming a promising paradigm for open-vocabulary visual recognition.
Ranked #1 on
Zero-Shot Action Recognition
on HMDB51
A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models
In particular, we ask "Given a large pool of prompts, can we automatically score the prompts and ensemble those that are most suitable for a particular downstream dataset, without needing access to labeled validation data?".
Towards Understanding the Effect of Pretraining Label Granularity
Furthermore, we perform comprehensive experiments using the label hierarchies of iNaturalist 2021 and observe that the following conditions, in addition to proper choice of label granularity, enable the transfer to work well in practice: 1) the pretraining dataset needs to have a meaningful label hierarchy, and 2) the pretraining and target label functions need to align well.
Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception
We conduct extensive empirical studies and reveal the following key insights: 1) Performing gradient descent updates by alternating on diverse modalities, loss functions, and tasks, with varying input resolutions, efficiently improves the model.
Ranked #1 on
Zero-Shot Action Recognition
on Kinetics
(using extra training data)
Train-Once-for-All Personalization
We study the problem of how to train a "personalization-friendly" model such that given only the task descriptions, the model can be adapted to different end-users' needs, e. g., for accurately classifying different subsets of objects.
