Search Results for author:
Found 92 papers, 42 papers with code
Spatiotemporal Contrastive Video Representation Learning
Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away.
Ranked #1 on
Self-Supervised Action Recognition
on Kinetics-600
MoViNets: Mobile Video Networks for Efficient Video Recognition
We present Mobile Video Networks (MoViNets), a family of computation and memory efficient video networks that can operate on streaming video for online inference.
Ranked #3 on
Action Classification
on Charades
Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision
Modern self-supervised learning algorithms typically enforce persistency of instance representations across views.
VideoGLUE: Video General Understanding Evaluation of Foundation Models
We evaluate existing foundation models video understanding capabilities using a carefully designed experiment protocol consisting of three hallmark tasks (action recognition, temporal localization, and spatiotemporal localization), eight datasets well received by the community, and four adaptation methods tailoring a foundation model (FM) for a downstream task.
Ranking Neural Checkpoints
This paper is concerned with ranking many pre-trained deep neural networks (DNNs), called checkpoints, for the transfer learning to a downstream task.
Ranked #6 on
Transferability
on classification benchmark
Class-Balanced Distillation for Long-Tailed Visual Recognition
An effective and simple approach to long-tailed visual recognition is to learn feature representations and a classifier separately, with instance and class-balanced sampling, respectively.
Ranked #11 on
Long-tail Learning
on iNaturalist 2018
Video Timeline Modeling For News Story Understanding
In this paper, we present a novel problem, namely video timeline modeling.
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval.
Ranked #3 on
Zero-Shot Video Retrieval
on YouCook2
(text-to-video Mean Rank metric)
Adversarial Examples Improve Image Recognition
We show that AdvProp improves a wide range of models on various image recognition tasks and performs better when the models are bigger.
Ranked #4 on
Domain Generalization
on VizWiz-Classification
When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
Vision Transformers (ViTs) and MLPs signal further efforts on replacing hand-wired features or inductive biases with general-purpose neural architectures.
Surrogate Gap Minimization Improves Sharpness-Aware Training
Instead, we define a \textit{surrogate gap}, a measure equivalent to the dominant eigenvalue of Hessian at a local minimum when the radius of the neighborhood (to derive the perturbed loss) is small.
Robust and Accurate Object Detection via Adversarial Learning
Data augmentation has become a de facto component for training high-performance deep image classifiers, but its potential is under-explored for object detection.
Ranked #17 on
Object Detection
on COCO-O
Unified Visual Relationship Detection with Vision and Language Models
To address this challenge, we propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models (VLMs).
Human-Object Interaction Detection
Relationship Detection
+2
Synthesized Classifiers for Zero-Shot Learning
Given semantic descriptions of object classes, zero-shot learning aims to accurately recognize objects of the unseen classes, from which no examples are available at the training stage, by associating them to the seen classes, from which labeled examples are provided.
Ranked #1 on
Few-Shot Image Classification
on AWA - 0-Shot
Large-Scale Long-Tailed Recognition in an Open World
We define Open Long-Tailed Recognition (OLTR) as learning from such naturally distributed data and optimizing the classification accuracy over a balanced test set which include head, tail, and open classes.
PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation
The need for fine-grained perception in autonomous driving systems has resulted in recently increased research on online semantic segmentation of single-scan LiDAR.
Ranked #11 on
Robust 3D Semantic Segmentation
on nuScenes-C
End-to-End Learning of Motion Representation for Video Understanding
Despite the recent success of end-to-end learned representations, hand-crafted optical flow features are still widely used in video analysis tasks.
Ranked #42 on
Action Recognition
on UCF101
A Fast and Accurate One-Stage Approach to Visual Grounding
We propose a simple, fast, and accurate one-stage approach to visual grounding, inspired by the following insight.
Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
Hence, we propose a curriculum-style learning approach to minimize the domain gap in urban scenery semantic segmentation.
Ranked #27 on
Image-to-Image Translation
on SYNTHIA-to-Cityscapes
A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes
Hence, we propose a curriculum-style learning approach to minimizing the domain gap in urban scene semantic segmentation.
Ranked #26 on
Image-to-Image Translation
on SYNTHIA-to-Cityscapes
Constructing Self-motivated Pyramid Curriculums for Cross-Domain Semantic Segmentation: A Non-Adversarial Approach
We propose a new approach, called self-motivated pyramid curriculum domain adaptation (PyCDA), to facilitate the adaptation of semantic segmentation neural networks from synthetic source domains to real target domains.
Ranked #14 on
Image-to-Image Translation
on SYNTHIA-to-Cityscapes
Smooth Adversarial Training
SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82. 2% accuracy and 58. 6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9. 5% for accuracy and 11. 6% for robustness.
DHER: Hindsight Experience Replay for Dynamic Goals
Dealing with sparse rewards is one of the most important challenges in reinforcement learning (RL), especially when a goal is dynamic (e. g., to grasp a moving object).
Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
Despite being impactful on a variety of problems and applications, the generative adversarial nets (GANs) are remarkably difficult to train.
NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks
Powerful adversarial attack methods are vital for understanding how to construct robust deep neural networks (DNNs) and for thoroughly testing defense techniques.
An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild
Zero-shot learning (ZSL) methods have been studied in the unrealistic setting where test data are assumed to come from unseen classes only.
Classifier and Exemplar Synthesis for Zero-Shot Learning
Zero-shot learning (ZSL) enables solving a task without the need to see its examples.
On Model Calibration for Long-Tailed Object Detection and Instance Segmentation
We propose NorCal, Normalized Calibration for long-tailed object detection and instance segmentation, a simple and straightforward recipe that reweighs the predicted scores of each class by its training sample size.
End-to-End Video Captioning with Multitask Reinforcement Learning
Although end-to-end (E2E) learning has led to impressive progress on a variety of visual understanding tasks, it is often impeded by hardware constraints (e. g., GPU memory) and is prone to overfitting.
MosaicOS: A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection
Many objects do not appear frequently enough in complex scenes (e. g., certain handbags in living rooms) for training an accurate object detector, but are often found frequently by themselves (e. g., in product images).
Domain Generalization with Adversarial Intensity Attack for Medical Image Segmentation
In real-world scenarios, however, it is common for models to encounter data from new and different domains to which they were not exposed to during training.
MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly.
Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective
Object frequency in the real world often follows a power law, leading to a mismatch between datasets with long-tailed class distributions seen by a machine learning model and our expectation of the model to perform well on all classes.
Ranked #27 on
Long-tail Learning
on Places-LT
VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
Many seemingly distant annotations (e. g., semantic segmentation and visual question answering (VQA)) are inherently connected in that they reveal different levels and perspectives of human understandings about the same visual scenes --- and even the same set of images (e. g., of COCO).
Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model
The other is that the number of images used for the knowledge distillation should be small; otherwise, it violates our expectation of reducing the dependence on large-scale datasets.
Synthesized Policies for Transfer and Adaptation across Tasks and Environments
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
2.5D Visual Relationship Detection
To enable progress on this task, we create a new dataset consisting of 220k human-annotated 2. 5D relationships among 512K objects from 11K images.
On Calibrating Semantic Segmentation Models: Analyses and An Algorithm
Then, we study popular existing calibration methods and compare them with selective scaling on semantic segmentation calibration.
CAMOU: Learning Physical Vehicle Camouflages to Adversarially Attack Detectors in the Wild
In particular, we learn a camouflage pattern to hide vehicles from being detected by state-of-the-art convolutional neural network based detectors.
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
In this paper, we attempt to approach the problem of Audio-Visual Embodied Navigation, the task of planning the shortest path from a random starting location in a scene to the sound source in an indoor environment, given only raw egocentric visual and audio sensory data.
Anti-Neuron Watermarking: Protecting Personal Data Against Unauthorized Neural Networks
We study protecting a user's data (images in this work) against a learner's unauthorized use in training neural networks.
medXGAN: Visual Explanations for Medical Classifiers through a Generative Latent Space
Despite the surge of deep learning in the past decade, some users are skeptical to deploy these models in practice due to their black-box nature.
A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels
In the first stage, we identify a small portion of images from the noisy training set of which the labels are correct with a high probability.
Blind Pre-Processing: A Robust Defense Method Against Adversarial Examples
Blind pre-processing improves the white box attack accuracy of MNIST from 94. 3\% to 98. 7\%.
Infinite-Label Learning with Semantic Output Codes
We develop a new statistical machine learning paradigm, named infinite-label learning, to annotate a data point with more than one relevant labels from a candidate set, which pools both the finite labels observed at training and a potentially infinite number of previously unseen labels.
Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach
However, one of the main obstacles to the research on video summarization is the user subjectivity - users have various preferences over the summaries.
Improving Facial Attribute Prediction using Semantic Segmentation
We build our facial attribute prediction model jointly with a deep semantic segmentation network.
Ranked #2 on
Facial Attribute Classification
on LFWA
Improved Dropout for Shallow and Deep Learning
To exhibit the optimal dropout probabilities, we analyze the shallow learning with multinomial dropout and establish the risk bound for stochastic optimization.
Query-Focused Extractive Video Summarization
The decision to include a shot in the summary depends on the shot's relevance to the user query and importance in the context of the video, jointly.
Fast Zero-Shot Image Tagging
The well-known word analogy experiments show that the recent word vectors capture fine-grained linguistic regularities in words by linear vector offsets, but it is unclear how well the simple vector offsets can encode visual regularities over words.
Ranked #5 on
Multi-label zero-shot learning
on Open Images V4
Learning Attributes Equals Multi-Source Domain Generalization
Attributes possess appealing properties and benefit many computer vision problems, such as object recognition, learning with humans in the loop, and image retrieval.
Large-Margin Determinantal Point Processes
Extensive empirical studies validate our contributions, including applications on challenging document and video summarization, where flexibility in modeling the kernel matrix and balancing different errors is indispensable.
How Local is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization
The large volume of video content and high viewing frequency demand automatic video summarization algorithms, of which a key property is the capability of modeling diversity.
Defend Deep Neural Networks Against Adversarial Examples via Fixed and Dynamic Quantized Activation Functions
Recent studies have shown that deep neural networks (DNNs) are vulnerable to adversarial attacks.
Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy
Deep convolution neural network has achieved great success in many artificial intelligence applications.
Improving Sequential Determinantal Point Processes for Supervised Video Summarization
In terms of modeling, we design a new probabilistic distribution such that, when it is integrated into SeqDPP, the resulting model accepts user input about the expected length of the summary.
Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation
The recent advances in deep learning have made it possible to generate photo-realistic images by using neural networks and even to extrapolate video frames from an input video clip.
Synthesize Policies for Transfer and Adaptation across Tasks and Environments
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
Diverse Sequential Subset Selection for Supervised Video Summarization
Video summarization is a challenging problem with great application potential.
Reshaping Visual Datasets for Domain Adaptation
By maximum distinctiveness, we require the underlying distributions of the identified domains to be different from each other; by maximum learnability, we ensure that a strong discriminative model can be learned from the domain.
Geometry Guided Convolutional Neural Networks for Self-Supervised Video Representation Learning
In addition, we also find that a progressive training strategy can foster a better neural network for the video recognition task than blindly pooling the distinct sources of geometry cues together.
Deep Face Detector Adaptation Without Negative Transfer or Catastrophic Forgetting
Arguably, no single face detector fits all real-life scenarios.
NATTACK: A STRONG AND UNIVERSAL GAUSSIAN BLACK-BOX ADVERSARIAL ATTACK
In other words, there is a population of adversarial examples, instead of only one, for any input to a DNN.
Joint Modeling of Dense and Incomplete Trajectories for Citywide Traffic Volume Inference
In this paper, we propose a novel framework for the citywide traffic volume inference using both dense GPS trajectories and incomplete trajectories captured by camera surveillance systems.
Defending Against Adversarial Attacks Using Random Forests
As deep neural networks (DNNs) have become increasingly important and popular, the robustness of DNNs is the key to the safety of both the Internet and the physical world.
Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data
To this end, we propose a new approach of domain randomization and pyramid consistency to learn a model with high generalizability.
Open Compound Domain Adaptation
A typical domain adaptation approach is to adapt models trained on the annotated data in a source domain (e. g., sunny weather) for achieving high performance on the test data in a target domain (e. g., rainy weather).
When Ensembling Smaller Models is More Efficient than Single Large Models
Ensembling is a simple and popular technique for boosting evaluation performance by training multiple models (e. g., with different initializations) and aggregating their predictions.
Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Clouds
We study an unsupervised domain adaptation problem for the semantic labeling of 3D point clouds, with a particular focus on domain discrepancies induced by different LiDAR sensors.
Improving Object Detection with Selective Self-supervised Self-training
They fail to improve object detectors in their vanilla forms due to the domain gap between the Web images and curated datasets.
A Lazy Approach to Long-Horizon Gradient-Based Meta-Learning
Gradient-based meta-learning relates task-specific models to a meta-model by gradients.
Contrastive Learning for Label-Efficient Semantic Segmentation
While recent Convolutional Neural Network (CNN) based semantic segmentation approaches have achieved impressive results by using large amounts of labeled training data, their performance drops significantly as the amount of labeled data decreases.
Large-Scale Meta-Learning with Continual Trajectory Shifting
Meta-learning of shared initialization parameters has shown to be highly effective in solving few-shot learning tasks.
Adversarially Adaptive Normalization for Single Domain Generalization
As a generic tool, the improvement introduced by ASR-Norm is agnostic to the choice of ADA methods.
Bridging the Gap Between Object Detection and User Intent via Query-Modulation
When interacting with objects through cameras, or pictures, users often have a specific intent.
Federated Multi-Target Domain Adaptation
We further take the server-client and inter-client domain shifts into account and pose a domain adaptation problem with one source (centralized server data) and multiple targets (distributed client data).
Contrastive Learning for Label Efficient Semantic Segmentation
While recent Convolutional Neural Network (CNN) based semantic segmentation approaches have achieved impressive results by using large amounts of labeled training data, their performance drops significantly as the amount of labeled data decreases.
CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization
One challenge in evaluating visual question answering (VQA) models in the cross-dataset adaptation setting is that the distribution shifts are multi-modal, making it difficult to identify if it is the shifts in visual or language features that play a key role.
Exploring Temporal Granularity in Self-Supervised Video Representation Learning
The training objective consists of two parts: a fine-grained temporal learning objective to maximize the similarity between corresponding temporal embeddings in the short clip and the long clip, and a persistent temporal learning objective to pull together global embeddings of the two clips.
Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text
The experiments show that the resultant unified foundation transformer works surprisingly well on both the vision-only and text-only tasks, and the proposed knowledge distillation and gradient masking strategy can effectively lift the performance to approach the level of separately-trained models.
Open Long-Tailed Recognition in a Dynamic World
A practical recognition system must balance between majority (head) and minority (tail) classes, generalize across the distribution, and acknowledge novelty upon the instances of unseen classes (open classes).
LESS: Label-Efficient Semantic Segmentation for LiDAR Point Clouds
Our method co-designs an efficient labeling process with semi/weakly supervised learning and is applicable to nearly any 3D semantic segmentation backbones.
Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding
Existing video-language pre-training methods primarily focus on instance-level alignment between video clips and captions via global contrastive learning but neglect rich fine-grained local information in both videos and text, which is of importance to downstream tasks requiring temporal localization and semantic reasoning.
Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models
This paper proposes a method for generating images of customized objects specified by users.
Federated Learning of Shareable Bases for Personalization-Friendly Image Classification
FedBasis learns a set of few shareable ``basis'' models, which can be linearly combined to form personalized models for clients.
Identity Encoder for Personalized Diffusion
Our approach greatly reduces the overhead for personalized image generation and is more applicable in many potential applications.
Multi-modal Domain Adaptation for REG via Relation Transfer
Domain adaptation, which aims to transfer knowledge between domains, has been well studied in many areas such as image classification and object detection.
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation.
Ranked #2 on
Video Prediction
on Kinetics-600 12 frames, 64x64
Towards A Unified Neural Architecture for Visual Recognition and Reasoning
Motivated by the recent success of multi-task transformers for visual recognition and language understanding, we propose a unified neural architecture for visual recognition and reasoning with a generic interface (e. g., tokens) for both.
Instruct-Imagen: Image Generation with Multi-modal Instruction
We introduce *multi-modal instruction* for image generation, a task representation articulating a range of generation intents with precision.
Distilling Vision-Language Models on Millions of Videos
Our best model outperforms state-of-the-art methods on MSR-VTT zero-shot text-to-video retrieval by 6%.
VideoPrism: A Foundational Visual Encoder for Video Understanding
We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model.
