Search Results for author:
Found 45 papers, 29 papers with code
A Unified Model for Tracking and Image-Video Detection Has More Power
Objection detection (OD) has been one of the most fundamental tasks in computer vision.
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Vision-language (VL) pre-training has recently received considerable attention.
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Dev
GLIPv2: Unifying Localization and Vision-Language Understanding
We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e. g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e. g., VQA, image captioning).
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Test
(using extra training data)
Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding
Detection Hub further achieves SoTA performance on UODB benchmark with wide variety of datasets.
K-LITE: Learning Transferable Visual Models with External Knowledge
We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts.
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets and tasks.
Ranked #1 on
Zero-Shot Image Classification
on ODinW
Missingness Bias in Model Debugging
Missingness, or the absence of features from an input, is a concept fundamental to many model debugging tools.
Unified Contrastive Learning in Image-Text-Label Space
Particularly, it attains gains up to 9. 2% and 14. 5% in average on zero-shot recognition benchmarks over the language-image contrastive learning and supervised learning methods, respectively.
Parameter-efficient Model Adaptation for Vision Transformers
In this paper, we aim to study parameter-efficient model adaptation strategies for vision transformers on the image classification task.
Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
This survey is inspired by the remarkable progress in both computer vision and natural language processing, and recent trends shifting from single modality processing to multiple modality comprehension.
RegionCLIP: Region-based Language-Image Pretraining
However, we show that directly applying such models to recognize image regions for object detection leads to poor performance due to a domain shift: CLIP was trained to match an image as a whole to a text description, without capturing the fine-grained alignment between image regions and text spans.
Ranked #4 on
Open Vocabulary Object Detection
on MSCOCO
(using extra training data)
Grounded Language-Image Pre-training
The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representation semantic-rich.
Ranked #1 on
2D object detection
on RF100
Focal Attention for Long-Range Interactions in Vision Transformers
With focal attention, we propose a new variant of Vision Transformer models, called Focal Transformers, which achieve superior performance over the state-of-the-art (SoTA) Vision Transformers on a range of public image classification and object detection benchmarks.
Florence: A New Foundation Model for Computer Vision
Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
Ranked #1 on
Action Recognition In Videos
on Kinetics-600
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks.
Ranked #15 on
Cross-Modal Retrieval
on COCO 2014
Image Scene Graph Generation (SGG) Benchmark
There is a surge of interest in image scene graph generation (object, attribute and relationship detection) due to the need of building fine-grained image understanding models that go beyond object detection.
Focal Self-attention for Local-Global Interactions in Vision Transformers
With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks.
Ranked #14 on
Instance Segmentation
on COCO test-dev
Efficient Self-supervised Vision Transformers for Representation Learning
This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning.
Ranked #3 on
Self-Supervised Image Classification
on ImageNet
Representation Learning
Self-Supervised Image Classification
3DB: A Framework for Debugging Computer Vision Models
We introduce 3DB: an extendable, unified framework for testing and debugging vision models using photorealistic simulation.
Multiscale Invertible Generative Networks for High-Dimensional Bayesian Inference
We propose a Multiscale Invertible Generative Network (MsIGN) and associated training algorithm that leverages multiscale structure to solve high-dimensional Bayesian inference.
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding
This paper presents a new Vision Transformer (ViT) architecture Multi-Scale Vision Longformer, which significantly enhances the ViT of \cite{dosovitskiy2020image} for encoding high-resolution images using two techniques.
Ranked #32 on
Instance Segmentation
on COCO minival
Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix
Nevertheless, we find that the performance of IRM can be dramatically degraded under \emph{strong $\Lambda$ spuriousness} -- when the spurious correlation between the spurious features and the class label is strong due to the strong causal influence of their common cause, the domain label, on both of them (see Fig.
VinVL: Revisiting Visual Representations in Vision-Language Models
In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model \oscar \cite{li2020oscar}, and utilize an improved approach \short\ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks.
Ranked #9 on
Image Captioning
on nocaps-val-overall
Dynamic DETR: End-to-End Object Detection With Dynamic Attention
To mitigate the second limitation of learning difficulty, we introduce a dynamic decoder by replacing the cross-attention module with a ROI-based dynamic attention in the Transformer decoder.
MiniVLM: A Smaller and Faster Vision-Language Model
We design a Two-stage Efficient feature Extractor (TEE), inspired by the one-stage EfficientDet network, to significantly reduce the time cost of visual feature extraction by $95\%$, compared to a baseline model.
MagGAN: High-Resolution Face Attribute Editing with Mask-Guided Generative Adversarial Network
We present Mask-guided Generative Adversarial Network (MagGAN) for high-resolution face attribute editing, in which semantic facial masks from a pre-trained face parser are used to guide the fine-grained image editing process.
Training Sparse Neural Networks using Compressed Sensing
The adaptive weighting we introduce corresponds to a novel regularizer based on the logarithm of the absolute value of the weights.
Novel Human-Object Interaction Detection via Adversarial Domain Generalization
We study in this paper the problem of novel human-object interaction (HOI) detection, aiming at improving the generalization ability of the model to unseen scenarios.
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks.
Ranked #1 on
Image Retrieval
on COCO
(Recall@10 metric)
Object-Centric Image Generation from Layouts
In this paper, we start with the idea that a model must be able to understand individual objects and relationships between objects in order to generate complex scenes well.
Ranked #1 on
Layout-to-Image Generation
on COCO-Stuff 256x256
Statistical Adaptive Stochastic Gradient Methods
We propose a statistical adaptive procedure called SALSA for automatically scheduling the learning rate (step size) in stochastic gradient methods.
Understanding the Role of Momentum in Stochastic Gradient Methods
The use of momentum in stochastic gradient methods has become a widespread practice in machine learning.
Statistical Adaptive Stochastic Optimization
We investigate statistical methods for automatically scheduling the learning rate (step size) in stochastic optimization.
Using Statistics to Automate Stochastic Optimization
Despite the development of numerous adaptive optimizers, tuning the learning rate of stochastic gradient methods remains a major roadblock to obtaining good practical performance in machine learning.
TIGEr: Text-to-Image Grounding for Image Caption Evaluation
This paper presents a new metric called TIGEr for the automatic evaluation of image captioning systems.
Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
In this paper, we employ adversarial training to improve the performance of randomized smoothing.
Object-driven Text-to-Image Synthesis via Adversarial Training
In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow object-centered text-to-image synthesis for complex scenes.
A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks
This framework works for neural networks with diverse architectures and nonlinearities and covers both primal and dual views of robustness verification.
Towards Coherent and Cohesive Long-form Text Generation
Generating coherent and cohesive long-form texts is a challenging task.
RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications
The Jacobian matrix (or the gradient for single-output networks) is directly related to many important properties of neural networks, such as the function landscape, stationary points, (local) Lipschitz constants and robustness to adversarial attacks.
A bird's eye view on coherence, and a worm's eye view on cohesion
Generating coherent and cohesive long-form texts is a challenging problem in natural language generation.
Turbo Learning for Captionbot and Drawingbot
We study in this paper the problems of both image captioning and text-to-image generation, and present a novel turbo learning approach to jointly training an image-to-text generator (a. k. a.
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
In this paper, we propose an Attentional Generative Adversarial Network (AttnGAN) that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation.
Ranked #7 on
Text-to-Image Generation
on Multi-Modal-CelebA-HQ
On the Discrimination-Generalization Tradeoff in GANs
When evaluated with neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set.
A sparse decomposition of low rank symmetric positive semi-definite matrices
In this paper, we partition the indices from 1 to $N$ into several patches and propose to quantify the sparseness of a vector by the number of patches on which it is nonzero, which is called patch-wise sparseness.
Numerical Analysis
