Search Results for author:
Found 97 papers, 46 papers with code
CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization
One challenge in evaluating visual question answering (VQA) models in the cross-dataset adaptation setting is that the distribution shifts are multi-modal, making it difficult to identify if it is the shifts in visual or language features that play a key role.
Synthesize Policies for Transfer and Adaptation across Tasks and Environments
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
On Discrete Prompt Optimization for Diffusion Models
Two major challenges arise in efficiently finding a solution to this problem: (1) Enormous Domain Space: Setting the domain to the entire language space poses significant difficulty to the optimization process.
Understanding the Impact of Negative Prompts: When and How Do They Take Effect?
The concept of negative prompts, emerging from conditional generation models like Stable Diffusion, allows users to specify what to exclude from the generated images.%, demonstrating significant practical efficacy.
The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noise
We identify these patches by analyzing the dispersion of object bounding boxes across generated images, leading to the development of a posterior analysis technique.
Automatic Jailbreaking of the Text-to-Image Generative AI Systems
Recent AI systems have shown extremely powerful performance, even surpassing human performance, on various tasks such as information retrieval, language generation, and image generation based on large language models (LLMs).
Large-Scale Multi-Center CT and MRI Segmentation of Pancreas with Deep Learning
We also collected CT scans of 1, 350 patients from publicly available sources for benchmarking purposes.
VideoPrism: A Foundational Visual Encoder for Video Understanding
We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model.
Distilling Vision-Language Models on Millions of Videos
Our best model outperforms state-of-the-art methods on MSR-VTT zero-shot text-to-video retrieval by 6%.
Instruct-Imagen: Image Generation with Multi-modal Instruction
We introduce *multi-modal instruction* for image generation, a task representation articulating a range of generation intents with precision.
Towards A Unified Neural Architecture for Visual Recognition and Reasoning
Motivated by the recent success of multi-task transformers for visual recognition and language understanding, we propose a unified neural architecture for visual recognition and reasoning with a generic interface (e. g., tokens) for both.
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation.
Ranked #2 on
Video Prediction
on Kinetics-600 12 frames, 64x64
Video Timeline Modeling For News Story Understanding
In this paper, we present a novel problem, namely video timeline modeling.
Multi-modal Domain Adaptation for REG via Relation Transfer
Domain adaptation, which aims to transfer knowledge between domains, has been well studied in many areas such as image classification and object detection.
VideoGLUE: Video General Understanding Evaluation of Foundation Models
We evaluate existing foundation models video understanding capabilities using a carefully designed experiment protocol consisting of three hallmark tasks (action recognition, temporal localization, and spatiotemporal localization), eight datasets well received by the community, and four adaptation methods tailoring a foundation model (FM) for a downstream task.
Federated Learning of Shareable Bases for Personalization-Friendly Image Classification
FedBasis learns a set of few shareable ``basis'' models, which can be linearly combined to form personalized models for clients.
Identity Encoder for Personalized Diffusion
Our approach greatly reduces the overhead for personalized image generation and is more applicable in many potential applications.
Domain Generalization with Adversarial Intensity Attack for Medical Image Segmentation
In real-world scenarios, however, it is common for models to encounter data from new and different domains to which they were not exposed to during training.
Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models
This paper proposes a method for generating images of customized objects specified by users.
Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding
Existing video-language pre-training methods primarily focus on instance-level alignment between video clips and captions via global contrastive learning but neglect rich fine-grained local information in both videos and text, which is of importance to downstream tasks requiring temporal localization and semantic reasoning.
Unified Visual Relationship Detection with Vision and Language Models
To address this challenge, we propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models (VLMs).
Human-Object Interaction Detection
Relationship Detection
+2
On Calibrating Semantic Segmentation Models: Analyses and An Algorithm
Then, we study popular existing calibration methods and compare them with selective scaling on semantic segmentation calibration.
LESS: Label-Efficient Semantic Segmentation for LiDAR Point Clouds
Our method co-designs an efficient labeling process with semi/weakly supervised learning and is applicable to nearly any 3D semantic segmentation backbones.
Open Long-Tailed Recognition in a Dynamic World
A practical recognition system must balance between majority (head) and minority (tail) classes, generalize across the distribution, and acknowledge novelty upon the instances of unseen classes (open classes).
medXGAN: Visual Explanations for Medical Classifiers through a Generative Latent Space
Despite the surge of deep learning in the past decade, some users are skeptical to deploy these models in practice due to their black-box nature.
Surrogate Gap Minimization Improves Sharpness-Aware Training
Instead, we define a \textit{surrogate gap}, a measure equivalent to the dominant eigenvalue of Hessian at a local minimum when the radius of the neighborhood (to derive the perturbed loss) is small.
Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text
The experiments show that the resultant unified foundation transformer works surprisingly well on both the vision-only and text-only tasks, and the proposed knowledge distillation and gradient masking strategy can effectively lift the performance to approach the level of separately-trained models.
Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision
Modern self-supervised learning algorithms typically enforce persistency of instance representations across views.
Exploring Temporal Granularity in Self-Supervised Video Representation Learning
The training objective consists of two parts: a fine-grained temporal learning objective to maximize the similarity between corresponding temporal embeddings in the short clip and the long clip, and a persistent temporal learning objective to pull together global embeddings of the two clips.
Anti-Neuron Watermarking: Protecting Personal Data Against Unauthorized Neural Networks
We study protecting a user's data (images in this work) against a learner's unauthorized use in training neural networks.
Federated Multi-Target Domain Adaptation
We further take the server-client and inter-client domain shifts into account and pose a domain adaptation problem with one source (centralized server data) and multiple targets (distributed client data).
On Model Calibration for Long-Tailed Object Detection and Instance Segmentation
We propose NorCal, Normalized Calibration for long-tailed object detection and instance segmentation, a simple and straightforward recipe that reweighs the predicted scores of each class by its training sample size.
Bridging the Gap Between Object Detection and User Intent via Query-Modulation
When interacting with objects through cameras, or pictures, users often have a specific intent.
When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
Vision Transformers (ViTs) and MLPs signal further efforts on replacing hand-wired features or inductive biases with general-purpose neural architectures.
Adversarially Adaptive Normalization for Single Domain Generalization
As a generic tool, the improvement introduced by ASR-Norm is agnostic to the choice of ADA methods.
2.5D Visual Relationship Detection
To enable progress on this task, we create a new dataset consisting of 220k human-annotated 2. 5D relationships among 512K objects from 11K images.
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval.
Ranked #3 on
Zero-Shot Video Retrieval
on YouCook2
(text-to-video Mean Rank metric)
Class-Balanced Distillation for Long-Tailed Visual Recognition
An effective and simple approach to long-tailed visual recognition is to learn feature representations and a classifier separately, with instance and class-balanced sampling, respectively.
Ranked #13 on
Long-tail Learning
on iNaturalist 2018
Robust and Accurate Object Detection via Adversarial Learning
Data augmentation has become a de facto component for training high-performance deep image classifiers, but its potential is under-explored for object detection.
Ranked #17 on
Object Detection
on COCO-O
MoViNets: Mobile Video Networks for Efficient Video Recognition
We present Mobile Video Networks (MoViNets), a family of computation and memory efficient video networks that can operate on streaming video for online inference.
Ranked #3 on
Action Classification
on Charades
MosaicOS: A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection
Many objects do not appear frequently enough in complex scenes (e. g., certain handbags in living rooms) for training an accurate object detector, but are often found frequently by themselves (e. g., in product images).
Large-Scale Meta-Learning with Continual Trajectory Shifting
Meta-learning of shared initialization parameters has shown to be highly effective in solving few-shot learning tasks.
Contrastive Learning for Label Efficient Semantic Segmentation
While recent Convolutional Neural Network (CNN) based semantic segmentation approaches have achieved impressive results by using large amounts of labeled training data, their performance drops significantly as the amount of labeled data decreases.
A Lazy Approach to Long-Horizon Gradient-Based Meta-Learning
Gradient-based meta-learning relates task-specific models to a meta-model by gradients.
Contrastive Learning for Label-Efficient Semantic Segmentation
While recent Convolutional Neural Network (CNN) based semantic segmentation approaches have achieved impressive results by using large amounts of labeled training data, their performance drops significantly as the amount of labeled data decreases.
Ranking Neural Checkpoints
This paper is concerned with ranking many pre-trained deep neural networks (DNNs), called checkpoints, for the transfer learning to a downstream task.
Ranked #6 on
Transferability
on classification benchmark
Spatiotemporal Contrastive Video Representation Learning
Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away.
Ranked #1 on
Self-Supervised Action Recognition
on Kinetics-600
Improving Object Detection with Selective Self-supervised Self-training
They fail to improve object detectors in their vanilla forms due to the domain gap between the Web images and curated datasets.
Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Clouds
We study an unsupervised domain adaptation problem for the semantic labeling of 3D point clouds, with a particular focus on domain discrepancies induced by different LiDAR sensors.
Smooth Adversarial Training
SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82. 2% accuracy and 58. 6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9. 5% for accuracy and 11. 6% for robustness.
When Ensembling Smaller Models is More Efficient than Single Large Models
Ensembling is a simple and popular technique for boosting evaluation performance by training multiple models (e. g., with different initializations) and aggregating their predictions.
PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation
The need for fine-grained perception in autonomous driving systems has resulted in recently increased research on online semantic segmentation of single-scan LiDAR.
Ranked #11 on
Robust 3D Semantic Segmentation
on nuScenes-C
Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model
The other is that the number of images used for the knowledge distillation should be small; otherwise, it violates our expectation of reducing the dependence on large-scale datasets.
Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective
Object frequency in the real world often follows a power law, leading to a mismatch between datasets with long-tailed class distributions seen by a machine learning model and our expectation of the model to perform well on all classes.
Ranked #29 on
Long-tail Learning
on Places-LT
MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly.
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
In this paper, we attempt to approach the problem of Audio-Visual Embodied Navigation, the task of planning the shortest path from a random starting location in a scene to the sound source in an indoor environment, given only raw egocentric visual and audio sensory data.
Adversarial Examples Improve Image Recognition
We show that AdvProp improves a wide range of models on various image recognition tasks and performs better when the models are bigger.
Ranked #4 on
Domain Generalization
on VizWiz-Classification
Open Compound Domain Adaptation
A typical domain adaptation approach is to adapt models trained on the annotated data in a source domain (e. g., sunny weather) for achieving high performance on the test data in a target domain (e. g., rainy weather).
Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data
To this end, we propose a new approach of domain randomization and pyramid consistency to learn a model with high generalizability.
Constructing Self-motivated Pyramid Curriculums for Cross-Domain Semantic Segmentation: A Non-Adversarial Approach
We propose a new approach, called self-motivated pyramid curriculum domain adaptation (PyCDA), to facilitate the adaptation of semantic segmentation neural networks from synthetic source domains to real target domains.
Ranked #14 on
Image-to-Image Translation
on SYNTHIA-to-Cityscapes
A Fast and Accurate One-Stage Approach to Visual Grounding
We propose a simple, fast, and accurate one-stage approach to visual grounding, inspired by the following insight.
Defending Against Adversarial Attacks Using Random Forests
As deep neural networks (DNNs) have become increasingly important and popular, the robustness of DNNs is the key to the safety of both the Internet and the physical world.
NATTACK: A STRONG AND UNIVERSAL GAUSSIAN BLACK-BOX ADVERSARIAL ATTACK
In other words, there is a population of adversarial examples, instead of only one, for any input to a DNN.
DHER: Hindsight Experience Replay for Dynamic Goals
Dealing with sparse rewards is one of the most important challenges in reinforcement learning (RL), especially when a goal is dynamic (e. g., to grasp a moving object).
NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks
Powerful adversarial attack methods are vital for understanding how to construct robust deep neural networks (DNNs) and for thoroughly testing defense techniques.
CAMOU: Learning Physical Vehicle Camouflages to Adversarially Attack Detectors in the Wild
In particular, we learn a camouflage pattern to hide vehicles from being detected by state-of-the-art convolutional neural network based detectors.
Large-Scale Long-Tailed Recognition in an Open World
We define Open Long-Tailed Recognition (OLTR) as learning from such naturally distributed data and optimizing the classification accuracy over a balanced test set which include head, tail, and open classes.
Synthesized Policies for Transfer and Adaptation across Tasks and Environments
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
Joint Modeling of Dense and Incomplete Trajectories for Citywide Traffic Volume Inference
In this paper, we propose a novel framework for the citywide traffic volume inference using both dense GPS trajectories and incomplete trajectories captured by camera surveillance systems.
A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes
Hence, we propose a curriculum-style learning approach to minimizing the domain gap in urban scene semantic segmentation.
Ranked #26 on
Image-to-Image Translation
on SYNTHIA-to-Cityscapes
Classifier and Exemplar Synthesis for Zero-Shot Learning
Zero-shot learning (ZSL) enables solving a task without the need to see its examples.
Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation
The recent advances in deep learning have made it possible to generate photo-realistic images by using neural networks and even to extrapolate video frames from an input video clip.
Improving Sequential Determinantal Point Processes for Supervised Video Summarization
In terms of modeling, we design a new probabilistic distribution such that, when it is integrated into SeqDPP, the resulting model accepts user input about the expected length of the summary.
Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy
Deep convolution neural network has achieved great success in many artificial intelligence applications.
Defend Deep Neural Networks Against Adversarial Examples via Fixed and Dynamic Quantized Activation Functions
Recent studies have shown that deep neural networks (DNNs) are vulnerable to adversarial attacks.
How Local is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization
The large volume of video content and high viewing frequency demand automatic video summarization algorithms, of which a key property is the capability of modeling diversity.
Geometry Guided Convolutional Neural Networks for Self-Supervised Video Representation Learning
In addition, we also find that a progressive training strategy can foster a better neural network for the video recognition task than blindly pooling the distinct sources of geometry cues together.
Deep Face Detector Adaptation Without Negative Transfer or Catastrophic Forgetting
Arguably, no single face detector fits all real-life scenarios.
End-to-End Learning of Motion Representation for Video Understanding
Despite the recent success of end-to-end learned representations, hand-crafted optical flow features are still widely used in video analysis tasks.
Ranked #42 on
Action Recognition
on UCF101
End-to-End Video Captioning with Multitask Reinforcement Learning
Although end-to-end (E2E) learning has led to impressive progress on a variety of visual understanding tasks, it is often impeded by hardware constraints (e. g., GPU memory) and is prone to overfitting.
Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
Despite being impactful on a variety of problems and applications, the generative adversarial nets (GANs) are remarkably difficult to train.
A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels
In the first stage, we identify a small portion of images from the noisy training set of which the labels are correct with a high probability.
Blind Pre-Processing: A Robust Defense Method Against Adversarial Examples
Blind pre-processing improves the white box attack accuracy of MNIST from 94. 3\% to 98. 7\%.
VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
Many seemingly distant annotations (e. g., semantic segmentation and visual question answering (VQA)) are inherently connected in that they reveal different levels and perspectives of human understandings about the same visual scenes --- and even the same set of images (e. g., of COCO).
Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
Hence, we propose a curriculum-style learning approach to minimize the domain gap in urban scenery semantic segmentation.
Ranked #27 on
Image-to-Image Translation
on SYNTHIA-to-Cityscapes
Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach
However, one of the main obstacles to the research on video summarization is the user subjectivity - users have various preferences over the summaries.
Improving Facial Attribute Prediction using Semantic Segmentation
We build our facial attribute prediction model jointly with a deep semantic segmentation network.
Ranked #2 on
Facial Attribute Classification
on LFWA
Infinite-Label Learning with Semantic Output Codes
We develop a new statistical machine learning paradigm, named infinite-label learning, to annotate a data point with more than one relevant labels from a candidate set, which pools both the finite labels observed at training and a potentially infinite number of previously unseen labels.
Query-Focused Extractive Video Summarization
The decision to include a shot in the summary depends on the shot's relevance to the user query and importance in the context of the video, jointly.
Fast Zero-Shot Image Tagging
The well-known word analogy experiments show that the recent word vectors capture fine-grained linguistic regularities in words by linear vector offsets, but it is unclear how well the simple vector offsets can encode visual regularities over words.
Ranked #5 on
Multi-label zero-shot learning
on Open Images V4
An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild
Zero-shot learning (ZSL) methods have been studied in the unrealistic setting where test data are assumed to come from unseen classes only.
Learning Attributes Equals Multi-Source Domain Generalization
Attributes possess appealing properties and benefit many computer vision problems, such as object recognition, learning with humans in the loop, and image retrieval.
Synthesized Classifiers for Zero-Shot Learning
Given semantic descriptions of object classes, zero-shot learning aims to accurately recognize objects of the unseen classes, from which no examples are available at the training stage, by associating them to the seen classes, from which labeled examples are provided.
Ranked #1 on
Few-Shot Image Classification
on AWA - 0-Shot
Improved Dropout for Shallow and Deep Learning
To exhibit the optimal dropout probabilities, we analyze the shallow learning with multinomial dropout and establish the risk bound for stochastic optimization.
Diverse Sequential Subset Selection for Supervised Video Summarization
Video summarization is a challenging problem with great application potential.
Large-Margin Determinantal Point Processes
Extensive empirical studies validate our contributions, including applications on challenging document and video summarization, where flexibility in modeling the kernel matrix and balancing different errors is indispensable.
Reshaping Visual Datasets for Domain Adaptation
By maximum distinctiveness, we require the underlying distributions of the identified domains to be different from each other; by maximum learnability, we ensure that a strong discriminative model can be learned from the domain.
