Search Results for author:
Found 112 papers, 70 papers with code
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion.
Ranked #1 on
Zero-Shot Object Detection
on MSCOCO
Improved Baselines with Visual Instruction Tuning
Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning.
Ranked #3 on
visual instruction following
on LLaVA-Bench
Factual Inconsistency Detection in Chart Captioning
visual instruction following
+1
Visual Instruction Tuning
Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field.
Ranked #4 on
Visual Question Answering
on BenchLMM
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
We find that scaling LMM consistently enhances model performance and improves language capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to the performance of full-model fine-tuning.
Ranked #45 on
Visual Question Answering
on MM-Vet
Focal Modulation Networks
For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2. 4, and beats Swin at multi-scale (50. 5 v. s.
Ranked #8 on
Object Detection
on COCO minival
(using extra training data)
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
We present Set-of-Mark (SoM), a new visual prompting method, to unleash the visual grounding abilities of large multimodal models (LMMs), such as GPT-4V.
LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing
LLaVA-Interactive is a research prototype for multimodal human-AI interaction.
Instruction Tuning with GPT-4
Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed.
MIMIC-IT: Multi-Modal In-Context Instruction Tuning
We release the MIMIC-IT dataset, instruction-response collection pipeline, benchmarks, and the Otter model.
Ranked #81 on
Visual Question Answering
on MM-Vet
Grounded Language-Image Pre-training
The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representation semantic-rich.
Ranked #1 on
2D Object Detection
on RF100
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets and tasks.
Ranked #1 on
Object Detection
on ELEVATER
Semantic-SAM: Segment and Recognize Anything at Any Granularity
In this paper, we introduce Semantic-SAM, a universal image segmentation model to enable segment and recognize anything at any desired granularity.
Visual In-Context Prompting
In-context prompting in large language models (LLMs) has become a prevalent approach to improve zero-shot capabilities, but this idea is less explored in the vision domain.
GLIGEN: Open-Set Grounded Text-to-Image Generation
Large-scale text-to-image diffusion models have made amazing advances.
Ranked #4 on
Conditional Text-to-Image Synthesis
on COCO-MIG
Generalized Decoding for Pixel, Image, and Language
We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly.
Ranked #4 on
Instance Segmentation
on ADE20K val
(using extra training data)
A Simple Framework for Open-Vocabulary Segmentation and Detection
We present OpenSeeD, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets.
Ranked #2 on
Instance Segmentation
on ADE20K val
(using extra training data)
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks.
Ranked #1 on
Image Retrieval
on MS COCO
(Recall@10 metric)
Focal Self-attention for Local-Global Interactions in Vision Transformers
With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks.
Ranked #17 on
Instance Segmentation
on COCO test-dev
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years.
Multimodal Foundation Models: From Specialists to General-Purpose Assistants
This paper presents a comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities, focusing on the transition from specialist models to general-purpose assistants.
RegionCLIP: Region-based Language-Image Pretraining
However, we show that directly applying such models to recognize image regions for object detection leads to poor performance due to a domain shift: CLIP was trained to match an image as a whole to a text description, without capturing the fine-grained alignment between image regions and text spans.
Ranked #11 on
Open Vocabulary Object Detection
on MSCOCO
(using extra training data)
LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
LLaVA-Plus is a general-purpose multimodal assistant that expands the capabilities of large multimodal models.
Ranked #1 on
LMM real-life tasks
on Leaderboard
Focal Attention for Long-Range Interactions in Vision Transformers
With focal attention, we propose a new variant of Vision Transformer models, called Focal Transformers, which achieve superior performance over the state-of-the-art (SoTA) Vision Transformers on a range of public image classification and object detection benchmarks.
Measuring the Intrinsic Dimension of Objective Landscapes
A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times.
Efficient Self-supervised Vision Transformers for Representation Learning
This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning.
Ranked #16 on
Self-Supervised Image Classification
on ImageNet
Representation Learning
Self-Supervised Image Classification
Florence: A New Foundation Model for Computer Vision
Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
Ranked #1 on
Action Recognition In Videos
on Kinetics-600
Unified Contrastive Learning in Image-Text-Label Space
Particularly, it attains gains up to 9. 2% and 14. 5% in average on zero-shot recognition benchmarks over the language-image contrastive learning and supervised learning methods, respectively.
K-LITE: Learning Transferable Visual Models with External Knowledge
We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts.
Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
We hope that our first pre-trained big VAE language model itself and results can help the NLP community renew the interests of deep generative models in the era of large-scale pre-training, and make these principled methods more practical.
Joint Embedding of Words and Labels for Text Classification
Word embeddings are effective intermediate representations for capturing semantic regularities between words, when learning the representations of text sequences.
Ranked #11 on
Text Classification
on DBpedia
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
Many deep learning architectures have been proposed to model the compositionality in text sequences, requiring a substantial number of parameters and expensive computations.
Ranked #1 on
Named Entity Recognition (NER)
on CoNLL 2000
On the Hidden Mystery of OCR in Large Multimodal Models
In this paper, we conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks including Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER).
TrustLLM: Trustworthiness in Large Language Models
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
The posteriors over neural network weights are high dimensional and multimodal.
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
To address this issue, we have created GVC data that allows for the combination of grounding and chat capabilities.
Few-shot Natural Language Generation for Task-Oriented Dialog
It is pre-trained on a large set of annotated NLG corpus to acquire the controllable generation ability, and fine-tuned with only a few domain-specific labels to adapt to new domains.
Ranked #4 on
Data-to-Text Generation
on MULTIWOZ 2.1
Towards Building the Federated GPT: Federated Instruction Tuning
This repository offers a foundational framework for exploring federated fine-tuning of LLMs using heterogeneous instructions across diverse categories.
LAFITE: Towards Language-Free Training for Text-to-Image Generation
One of the major challenges in training text-to-image generation models is the need of a large number of high-quality image-text pairs.
Ranked #2 on
Text-to-Image Generation
on Multi-Modal-CelebA-HQ
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
To bridge this gap, we present MathVista, a benchmark designed to combine challenges from diverse mathematical and visual tasks.
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
Variational autoencoders (VAEs) with an auto-regressive decoder have been applied for many natural language processing (NLP) tasks.
Feature Quantization Improves GAN Training
The instability in GAN training has been a long-standing problem despite remarkable research efforts.
Ranked #1 on
Conditional Image Generation
on CIFAR-100
Learning Customized Visual Models with Retrieval-Augmented Knowledge
Image-text contrastive learning models such as CLIP have demonstrated strong task transfer ability.
Ranked #1 on
Semi-Supervised Image Classification
on ImageNet - 1% labeled data
(using extra training data)
POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
Large-scale pre-trained language models, such as BERT and GPT-2, have achieved excellent performance in language representation learning and free-form text generation.
Parameter-efficient Model Adaptation for Vision Transformers
In this paper, we aim to study parameter-efficient model adaptation strategies for vision transformers on the image classification task.
ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching
We investigate the non-identifiability issues associated with bidirectional adversarial training for joint distribution matching.
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
By training on a large amount of image-text-action triplets in a self-supervised learning manner, the pre-trained model provides generic representations of visual environments and language instructions.
Ranked #1 on
Visual Navigation
on Help, Anna! (HANNA)
SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching
We present a new method SOLOIST that uses transfer learning and machine teaching to build task bots at scale.
Ranked #4 on
End-To-End Dialogue Modelling
on MULTIWOZ 2.0
Triangle Generative Adversarial Networks
The generators are designed to learn the two-way conditional distributions between the two domains, while the discriminators implicitly define a ternary discriminative function, which is trained to distinguish real data pairs and two kinds of fake data pairs.
Implicit Deep Latent Variable Models for Text Generation
Deep latent variable models (LVM) such as variational auto-encoder (VAE) have recently played an important role in text generation.
Few-Shot Named Entity Recognition: A Comprehensive Study
This paper presents a comprehensive study to efficiently build named entity recognition (NER) systems when a small number of in-domain labeled data is available.
Seeds Cleansing CNMF for Spatiotemporal Neural Signals Extraction of Miniscope Imaging Data
Miniscope calcium imaging is increasingly being used to monitor large populations of neuronal activities in freely behaving animals.
Neurons and Cognition Quantitative Methods
Twin Auxiliary Classifiers GAN
One of the popular conditional models is Auxiliary Classifier GAN (AC-GAN), which generates highly discriminative images by extending the loss function of GAN with an auxiliary classifier.
Ranked #2 on
Conditional Image Generation
on CIFAR-100
Twin Auxilary Classifiers GAN
One of the popular conditional models is Auxiliary Classifier GAN (AC-GAN) that generates highly discriminative images by extending the loss function of GAN with an auxiliary classifier.
Ranked #2 on
Image Generation
on CIFAR-100
Deep Temporal Sigmoid Belief Networks for Sequence Modeling
Deep dynamic generative models are developed to learn sequential dependencies in time-series data.
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward
Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM).
Structure-Aware Human-Action Generation
Generating long-range skeleton-based human actions has been a challenging problem since small deviations of one frame can cause a malformed action sequence.
Ranked #2 on
Human action generation
on NTU RGB+D 2D
Adversarial Time-to-Event Modeling
Modern health data science applications leverage abundant molecular and electronic health data, providing opportunities for machine learning to build statistical models to support clinical practice.
Towards Amortized Ranking-Critical Training for Collaborative Filtering
In this paper we investigate new methods for training collaborative filtering models based on actor-critic reinforcement learning, to directly optimize the non-differentiable quality metrics of interest.
Ranked #4 on
Recommendation Systems
on Million Song Dataset
Symmetric Variational Autoencoder and Connections to Adversarial Learning
A new form of the variational autoencoder (VAE) is proposed, based on the symmetric Kullback-Leibler divergence.
Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization
Stochastic gradient Markov chain Monte Carlo (SG-MCMC) methods are Bayesian analogs to popular stochastic optimization methods; however, this connection is not well studied.
Adversarial Learning of a Sampler Based on an Unnormalized Distribution
We investigate adversarial learning in the case when only an unnormalized form of the density can be accessed, rather than samples.
Survival Cluster Analysis
As a result, there is an unmet need in survival analysis for identifying subpopulations with distinct risk profiles, while jointly accounting for accurate individualized time-to-event predictions.
Partition-Guided GANs
We formulate two desired criteria for the space partitioner that aid the training of our mixture of generators: 1) to produce connected partitions and 2) provide a proxy of distance between partitions and data samples, along with a direction for reducing that distance.
Ranked #9 on
Image Generation
on STL-10
Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentation
Recent studies imply that deep neural networks are vulnerable to adversarial examples -- inputs with a slight but intentional perturbation are incorrectly classified by the network.
Hierarchical Graph Capsule Network
Graph Neural Networks (GNNs) draw their strength from explicitly modeling the topological information of structured data.
HallE-Control: Controlling Object Hallucination in Large Multimodal Models
Current Large Multimodal Models (LMMs) achieve remarkable progress, yet there remains significant uncertainty regarding their ability to accurately apprehend visual details, that is, in performing detailed captioning.
Learning Structural Weight Uncertainty for Sequential Decision-Making
Learning probability distributions on the weights of neural networks (NNs) has recently proven beneficial in many applications.
Robust Navigation with Language Pretraining and Stochastic Sampling
Core to the vision-and-language navigation (VLN) challenge is building robust instruction representations and action decoding schemes, which can generalize well to previously unseen instructions and environments.
Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics
Learning good representation of giga-pixel level whole slide pathology images (WSI) for downstream tasks is critical.
Contrastive Attraction and Contrastive Repulsion for Representation Learning
We realize this strategy with contrastive attraction and contrastive repulsion (CACR), which makes the query not only exert a greater force to attract more distant positive samples but also do so to repel closer negative samples.
Continuous-Time Flows for Efficient Inference and Density Estimation
Distinct from normalizing flows and GANs, CTFs can be adopted to achieve the above two goals in one framework, with theoretical guarantees.
Adversarial Symmetric Variational Autoencoder
A new form of variational autoencoder (VAE) is developed, in which the joint distribution of data and codes is considered in two (symmetric) forms: ($i$) from observed data fed through the encoder to yield codes, and ($ii$) from latent codes drawn from a simple prior and propagated through the decoder to manifest data.
VAE Learning via Stein Variational Gradient Descent
A new method for learning variational autoencoders (VAEs) is developed, based on Stein variational gradient descent.
Learning Generic Sentence Representations Using Convolutional Neural Networks
We propose a new encoder-decoder approach to learn distributed sentence representations that are applicable to multiple purposes.
Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling
Recurrent neural networks (RNNs) have shown promising performance for language modeling.
Unsupervised Learning with Truncated Gaussian Graphical Models
Gaussian graphical models (GGMs) are widely used for statistical modeling, because of ease of inference and the ubiquitous use of the normal distribution in practical approximations.
Stochastic Gradient MCMC with Stale Gradients
In this paper we develop theory to show that while the bias and MSE of an SG-MCMC algorithm depend on the staleness of stochastic gradients, its estimation variance (relative to the expected estimate, based on a prescribed number of samples) is independent of it.
Variational Autoencoder for Deep Learning of Images, Labels and Captions
A novel variational autoencoder is developed to model images, as well as associated labels or captions.
Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
Pytorch implementations of Bayes By Backprop, MC Dropout, SGLD, the Local Reparametrization Trick, KF-Laplace and more
High-Order Stochastic Gradient Thermostats for Bayesian Learning of Deep Models
Stochastic gradient MCMC algorithms (SG-MCMC) are a family of diffusion-based sampling methods for large-scale Bayesian learning.
A Deep Generative Deconvolutional Image Model
A deep generative model is developed for representation and analysis of images, based on a hierarchical convolutional dictionary-learning framework.
Policy Optimization as Wasserstein Gradient Flows
Policy optimization is a core component of reinforcement learning (RL), and most existing RL methods directly optimize parameters of a policy based on maximizing the expected total reward, or its surrogate.
Generative Adversarial Network Training is a Continual Learning Problem
We hypothesize that this is at least in part due to the evolution of the generator distribution and the catastrophic forgetting tendency of neural networks, which leads to the discriminator losing the ability to remember synthesized samples from previous instantiations of the generator.
Persistence-based Structural Recognition
This paper presents a framework for object recognition using topological persistence.
Learning Weight Uncertainty With Stochastic Gradient MCMC for Shape Classification
Learning the representation of shape cues in 2D & 3D objects for recognition is a fundamental task in computer vision.
DoubleTransfer at MEDIQA 2019: Multi-Source Transfer Learning for Natural Language Understanding in the Medical Domain
We use a multi-source transfer learning approach to transfer the knowledge from MT-DNN and SciBERT to natural language understanding tasks in the medical domain.
Straight-Through Estimator as Projected Wasserstein Gradient Flow
The Straight-Through (ST) estimator is a widely used technique for back-propagating gradients through discrete random variables.
Multi-View Learning for Vision-and-Language Navigation
Learning to navigate in a visual environment following natural language instructions is a challenging task because natural language instructions are highly variable, ambiguous, and under-specified.
Weakly supervised cross-domain alignment with optimal transport
Cross-domain alignment between image objects and text sequences is key to many visual-language tasks, and it poses a fundamental challenge to both computer vision and natural language processing.
Robust Conversational AI with Grounded Text Generation
This article provides an overview of this progress and discusses related methods and technologies that can be incorporated for building robust conversational AI systems.
Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference
The neural attention mechanism plays an important role in many natural language processing applications.
Improving Text Generation with Student-Forcing Optimal Transport
An extension is further proposed to improve the OT learning, based on the structural and contextual information of the text sequences.
ReMP: Rectified Metric Propagation for Few-Shot Learning
Few-shot learning features the capability of generalizing from a few examples.
Self-supervised Pre-training with Hard Examples Improves Visual Representations
Self-supervised pre-training (SSP) employs random image transformations to generate training data for visual representation learning.
Ranked #69 on
Self-Supervised Image Classification
on ImageNet
RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems
For task-oriented dialog systems to be maximally useful, it must be able to process conversations in a way that is (1) generalizable with a small number of training examples for new task domains, and (2) robust to user input in various styles, modalities or domains.
SDA: Improving Text Generation with Self Data Augmentation
Data augmentation has been widely used to improve deep neural networks in many research fields, such as computer vision.
Leveraging User Behavior History for Personalized Email Search
Given these observations, we propose to leverage user search history as query context to characterize users and build a context-aware ranking model for email search.
SYNERGY: Building Task Bots at Scale Using Symbolic Knowledge and Machine Teaching
We propose SYNERGY, a hybrid learning framework where a task bot is developed in two steps: (i) Symbolic knowledge to neural networks: Large amounts of simulated dialog sessions are generated based on task-specific symbolic knowledge which is represented as a task schema consisting of dialog flows and task-oriented databases.
Rethinking Sentiment Style Transfer
To overcome this issue, we propose a graph-based method to extract attribute content and attribute-independent content from input sentences in the YELP dataset and IMDB dataset.
Few-Shot Named Entity Recognition: An Empirical Baseline Study
This paper presents an empirical study to efficiently build named entity recognition (NER) systems when a small amount of in-domain labeled data is available.
A Generic Approach for Enhancing GANs by Regularized Latent Optimization
With the rapidly growing model complexity and data volume, training deep generative models (DGMs) for better performance has becoming an increasingly more important challenge.
Towards Language-Free Training for Text-to-Image Generation
One of the major challenges in training text-to-image generation models is the need of a large number of high-quality text-image pairs.
STT: Soft Template Tuning for Few-Shot Adaptation
Significantly, it can even outperform the time- and resource-consuming fine-tuning method on sentiment classification tasks.
Lafite2: Few-shot Text-to-Image Generation
The low requirement of the proposed method yields high flexibility and usability: it can be beneficial to a wide range of settings, including the few-shot, semi-supervised and fully-supervised learning; it can be applied on different models including generative adversarial networks (GANs) and diffusion models.
Scaling Vision-Language Models with Sparse Mixture of Experts
The field of natural language processing (NLP) has made significant strides in recent years, particularly in the development of large-scale vision-language models (VLMs).
Large Multimodal Models: Notes on CVPR 2023 Tutorial
This tutorial note summarizes the presentation on ``Large Multimodal Models: Towards Building and Surpassing Multimodal GPT-4'', a part of CVPR 2023 tutorial on ``Recent Advances in Vision Foundation Models''.
Benchmarking and Analyzing Generative Data for Visual Recognition
Advancements in large pre-trained generative models have expanded their potential as effective data generators in visual recognition.
Aligning Large Multimodal Models with Factually Augmented RLHF
Large Multimodal Models (LMM) are built across modalities and the misalignment between two modalities can result in "hallucination", generating textual outputs that are not grounded by the multimodal information in context.
BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys
In a comprehensive battery of tests on counterfactual medical image generation, BiomedJourney substantially outperforms prior state-of-the-art methods in instruction image editing and medical image generation such as InstructPix2Pix and RoentGen.
Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images
In particular, we instruction tune vision-language models to generate detailed visual descriptions of camera trap images using similar terminology to experts.
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images.
Training Small Multimodal Models to Bridge Biomedical Competency Gap: A Case Study in Radiology Imaging
Frontier models such as GPT-4V still have major competency gaps in multimodal capabilities for biomedical applications.
