Search Results for author:
Found 40 papers, 28 papers with code
What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases
Vision-language (VL) models, pretrained on colossal image-text datasets, have attained broad VL competence that is difficult to evaluate.
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning
Vision-language pre-training and instruction tuning have demonstrated general-purpose capabilities in 2D visual reasoning tasks by aligning visual encoders with state-of-the-art large language models (LLMs).
CodeTF: One-stop Transformer Library for State-of-the-art Code LLM
In this paper, we present CodeTF, an open-source Transformer-based library for state-of-the-art Code LLMs and code intelligence.
BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing
Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions.
ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
Recent advancements in multimodal pre-training methods have shown promising efficacy in 3D representation learning by aligning multimodal features across 3D shapes, their 2D counterparts, and language descriptions.
Ranked #6 on
3D Point Cloud Classification
on ScanObjectNN
(using extra training data)
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
To address these limitations, we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks.
Ranked #1 on
Code Search
on CodeXGLUE - AdvTest
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence.
Ranked #5 on
visual instruction following
on LLaVA-Bench
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models.
Ranked #1 on
Image Retrieval
on Flickr30k
From Images to Textual Prompts: Zero-Shot Visual Question Answering With Frozen Large Language Models
To address this issue, we propose Img2Prompt, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections, so that LLMs can perform zero-shot VQA tasks without end-to-end training.
From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models
To address this issue, we propose \emph{Img2Prompt}, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections, so that LLMs can perform zero-shot VQA tasks without end-to-end training.
Tackling Data Heterogeneity in Federated Learning with Class Prototypes
We propose FedNH, a novel method that improves the local models' performance for both personalization and generalization by combining the uniformity and semantics of class prototypes.
BotSIM: An End-to-End Bot Simulation Toolkit for Commercial Task-Oriented Dialog Systems
BotSIM adopts a layered design comprising the infrastructure layer, the adaptor layer and the application layer.
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training
Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting.
Ranked #2 on
Visual Question Answering (VQA)
on VQA v2 val
LAVIS: A Library for Language-Vision Intelligence
We introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications.
Masked Unsupervised Self-training for Label-free Image Classification
We demonstrate the efficacy of MUST on a variety of downstream tasks, where it improves upon CLIP by a large margin.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision.
Ranked #3 on
Open Vocabulary Attribute Detection
on OVAD-Box benchmark
(using extra training data)
Align and Prompt: Video-and-Language Pre-training with Entity Prompts
To achieve this, we first introduce an entity prompter module, which is trained with VTC to produce the similarity between a video crop and text prompts instantiated with entity names.
Ranked #19 on
Zero-Shot Video Retrieval
on DiDeMo
Open Vocabulary Object Detection with Pseudo Bounding-Box Labels
To enlarge the set of base classes, we propose a method to automatically generate pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs.
Improving Tail-Class Representation with Centroid Contrastive Learning
ICCL interpolates two images from a class-agnostic sampler and a class-aware sampler, and trains the model such that the representation of the interpolative image can be used to retrieve the centroids for both source classes.
Ranked #22 on
Long-tail Learning
on CIFAR-10-LT (ρ=10)
Cascaded Fast and Slow Models for Efficient Semantic Code Search
The goal of natural language semantic code search is to retrieve a semantically relevant code snippet from a fixed set of candidates using a natural language query.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens.
Ranked #5 on
Open Vocabulary Attribute Detection
on OVAD-Box benchmark
(using extra training data)
Learning From Noisy Data With Robust Representation Learning
In contrast to most existing methods, we combat noise by learning robust representation.
Noise-Robust Contrastive Learning
In contrast to most existing methods, we combat noise by learning robust representation.
CoMatch: Semi-supervised Learning with Contrastive Graph Regularization
CoMatch jointly learns two representations of the training data, their class probabilities and low-dimensional embeddings.
MoPro: Webly Supervised Learning with Momentum Prototypes
We propose momentum prototypes (MoPro), a simple contrastive learning method that achieves online label noise correction, out-of-distribution sample removal, and representation learning.
Ranked #12 on
Image Classification
on OmniBenchmark
(using extra training data)
The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation
Specifically, we systematically investigate performance drop of the state-of-the-art two-stage instance segmentation model Mask R-CNN on the recent long-tail LVIS dataset, and unveil that a major cause is the inaccurate classification of object proposals.
Prototypical Contrastive Learning of Unsupervised Representations
This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that addresses the fundamental limitations of instance-wise contrastive learning.
Ranked #5 on
Contrastive Learning
on imagenet-1k
Improving out-of-distribution generalization via multi-task self-supervised pretraining
Self-supervised feature representations have been shown to be useful for supervised classification, few-shot learning, and adversarial robustness.
Ranked #116 on
Domain Generalization
on PACS
Towards Noise-resistant Object Detection with Noisy Annotations
We address the challenging problem of training object detectors with noisy annotations, where the noise contains a mixture of label noise and bounding box noise.
DivideMix: Learning with Noisy Labels as Semi-supervised Learning
Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data.
Ranked #3 on
Learning with noisy labels
on CIFAR-100N
Weakly-Supervised Multi-Person Action Recognition in 360$^{\circ}$ Videos
To enable research in this direction, we introduce 360Action, the first omnidirectional video dataset for multi-person action recognition.
GradMix: Multi-source Transfer across Domains and Tasks
Therefore, it is important to develop algorithms that can leverage off-the-shelf labeled dataset to learn useful knowledge for the target task.
Classification Calibration for Long-tail Instance Segmentation
In this report, we investigate the performance drop phenomenon of state-of-the-art two-stage instance segmentation models when processing extreme long-tail training data based on the LVIS [5] dataset, and find a major cause is the inaccurate classification of object proposals.
Visual Social Relationship Recognition
Social relationships form the basis of social structure of humans.
Learning to Learn from Noisy Labeled Data
Despite the success of deep neural networks (DNNs) in image classification tasks, the human-level performance relies on massive training data with high-quality manual annotations, which are expensive and time-consuming to collect.
Ranked #26 on
Image Classification
on Clothing1M
(using extra training data)
Unsupervised Learning of View-invariant Action Representations
Different from previous works in video representation learning, our unsupervised learning task is to predict 3D motion in multiple target views using video representation from a source view.
Interact as You Intend: Intention-Driven Human-Object Interaction Detection
The recent advances in instance-level detection tasks lay strong foundation for genuine comprehension of the visual scenes.
Video Storytelling: Textual Summaries for Events
Video storytelling introduces new challenges, mainly due to the diversity of the story and the length and complexity of the video.
Attention Transfer from Web Images for Video Recognition
However, due to the domain shift problem, the performance of Web images trained deep classifiers tend to degrade when directly deployed to videos.
Dual-Glance Model for Deciphering Social Relationships
Since the beginning of early civilizations, social relationships derived from each individual fundamentally form the basis of social structure in our daily life.
Ranked #3 on
Visual Social Relationship Recognition
on PIPA
