Search Results for author:
Found 67 papers, 30 papers with code
MURAL: Multimodal, Multitask Representations Across Languages
Both image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages.
MultiReQA: A Cross-Domain Evaluation forRetrieval Question Answering Models
Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus (Ahmad et al., 2019). This dataset paper presents MultiReQA, a new multi-domain ReQA evaluation suite composed of eight retrieval QA tasks drawn from publicly available QA datasets.
Multi-stage Training with Improved Negative Contrast for Neural Passage Retrieval
In the context of neural passage retrieval, we study three promising techniques: synthetic data generation, negative sampling, and fusion.
Understanding Alignment in Multimodal LLMs: A Comprehensive Study
A primary objective of alignment for MLLMs is to encourage these models to align responses more closely with image information.
MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs
We introduce MIA-Bench, a new benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to strictly adhere to complex instructions.
Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks.
Ranked #83 on
Visual Question Answering
on MM-Vet
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks.
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons.
Ranked #30 on
Visual Question Answering
on MM-Vet
How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts
The remarkable advancements in Multimodal Large Language Models (MLLMs) have not rendered them immune to challenges, particularly in the context of handling deceptive information in prompts, thus producing hallucinated responses under such conditions.
VeCLIP: Improving CLIP Training via Visual-enriched Captions
For example, VeCLIP achieves up to +25. 2% gain in COCO and Flickr30k retrieval tasks under the 12M setting.
Ferret: Refer and Ground Anything Anywhere at Any Granularity
We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions.
Compressing LLMs: The Truth is Rarely Pure and Never Simple
Recently, several works have shown significant success in training-free and data-free compression (pruning and quantization) of LLMs that achieve 50 - 60% sparsity and reduce the bit width to 3 or 4 bits per weight, with negligible degradation of perplexity over the uncompressed baseline.
Guiding Instruction-based Image Editing via Multimodal Large Language Models
Extensive experimental results demonstrate that expressive instructions are crucial to instruction-based image editing, and our MGIE can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.
Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts
We empirically show that our sparse Mobile Vision MoEs (V-MoEs) can achieve a better trade-off between performance and efficiency than the corresponding dense ViTs.
MOFI: Learning Image Representations from Noisy Entity Annotated Images
Our approach involves employing a named entity recognition model to extract entities from the alt-text, and then using a CLIP model to select the correct entities as labels of the paired image.
Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustness
In this paper, we discuss two effective approaches to improve the efficiency and robustness of CLIP training: (1) augmenting the training dataset while maintaining the same number of optimization steps, and (2) filtering out samples that contain text regions in the image.
On Robustness in Multimodal Learning
Multimodal learning is defined as learning over multiple heterogeneous input modalities such as video, audio, and text.
STAIR: Learning Sparse Text and Image Representation in Grounded Tokens
We extend the CLIP model and build a sparse text and image representation (STAIR), where the image and text are mapped to a sparse token space.
Masked Autoencoding Does Not Help Natural Language Supervision at Scale
Self supervision and natural language supervision have emerged as two exciting ways to train general purpose image encoders which excel at a variety of downstream tasks.
DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely
Automated summary quality assessment falls into two categories: reference-based and reference-free.
Perceptual Grouping in Contrastive Vision-Language Models
In this work we examine how well vision-language models are able to understand where objects reside within an image and group together visually related parts of the imagery.
A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning
Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments, as a step towards robots that can follow human instructions.
Ranked #1 on
Vision and Language Navigation
on RxR
(using extra training data)
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge.
Ranked #1 on
Text-to-Image Generation
on LAION COCO
Simple and Effective Synthesis of Indoor 3D Scenes
We study the problem of synthesizing immersive 3D indoor scenes from one or more images.
LongT5: Efficient Text-To-Text Transformer for Long Sequences
Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models.
Ranked #1 on
Text Summarization
on BigPatent
Large Dual Encoders Are Generalizable Retrievers
With multi-stage training, surprisingly, scaling up the model size brings significant improvement on a variety of retrieval tasks, especially for out-of-domain generalization.
Ranked #9 on
Zero-shot Text Search
on BEIR
A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations
A simple but highly effective method "Language Information Removal (LIR)" factors out language identity information from semantic related components in multilingual representations pre-trained on multi-monolingual data.
MURAL: Multimodal, Multitask Retrieval Across Languages
Both image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages.
Ranked #1 on
Semantic Image-Text Similarity
on CxC
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
To support our investigation, we establish a new sentence representation transfer benchmark, SentGLUE, which extends the SentEval toolkit to nine tasks from the GLUE benchmark.
Pathdreamer: A World Model for Indoor Navigation
People navigating in unfamiliar buildings take advantage of myriad visual, spatial and semantic cues to efficiently achieve their navigation goals.
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset.
Ranked #1 on
Image Classification
on VTAB-1k
(using extra training data)
Cross-Modal Contrastive Learning for Text-to-Image Generation
The quality of XMC-GAN's output is a major step up from previous models, as we show on three challenging datasets.
Ranked #27 on
Text-to-Image Generation
on MS COCO
(using extra training data)
Universal Sentence Representations Learning with Conditional Masked Language Model
This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations on large scale unlabeled corpora.
Universal Sentence Representation Learning with Conditional Masked Language Model
This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations on large scale unlabeled corpora.
Self-Supervised Learning for Pairwise Data Refinement
We evaluate our methods on de-noising parallel texts and training neural machine translation models.
Neural Passage Retrieval with Improved Negative Contrast
In this paper we explore the effects of negative sampling in dual encoder models used to retrieve passages for automatic question answering.
Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation
Independently computing embeddings for questions and answers results in late fusion of information related to matching questions to their answers.
Language-agnostic BERT Sentence Embedding
While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning (Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings have yet to be explored.
SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling
Canonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain.
MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models
Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus (Ahmad et al., 2019). This paper presents MultiReQA, anew multi-domain ReQA evaluation suite com-posed of eight retrieval QA tasks drawn from publicly available QA datasets.
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
By supporting multi-modal retrieval training and evaluation, image captioning datasets have spurred remarkable progress on representation learning.
Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation
The question generation system is trained on general domain data, but is applied to documents in the targeted domain.
LAReQA: Language-agnostic answer retrieval from a multilingual pool
We present LAReQA, a challenging new benchmark for language-agnostic answer retrieval from a multilingual candidate pool.
Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improve
We examine these questions by contrasting the performance of several variants of LSTM-CRF architectures for named entity recognition, with some provided only representations of the context as features.
Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models
We propose a method for auditing the in-domain robustness of systems, focusing specifically on differences in performance due to the national origin of entities.
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification
Most existing work on adversarial data generation focuses on English.
Learning a Multi-Domain Curriculum for Neural Machine Translation
Most data selection research in machine translation focuses on improving a single domain.
ReQA: An Evaluation for End-to-End Answer Retrieval Models
Popular QA benchmarks like SQuAD have driven progress on the task of identifying answer spans within a specific passage, with models now surpassing human performance.
Multilingual Universal Sentence Encoder for Semantic Retrieval
We introduce two pre-trained retrieval focused multilingual sentence encoding models, respectively based on the Transformer and CNN model architectures.
Hierarchical Document Encoder for Parallel Corpus Mining
We explore using multilingual document embeddings for nearest neighbor mining of parallel data.
Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction
Experiments on a complex biomedical information extraction task using expert and lay annotators show that: (i) simply excluding from the training data instances predicted to be difficult yields a small boost in performance; (ii) using difficulty scores to weight instances during training provides further, consistent gains; (iii) assigning instances predicted to be difficult to domain experts is an effective strategy for task routing.
Universal Sentence Encoder for English
We present easy-to-use TensorFlow Hub sentence embedding models having good task transfer performance.
Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model
A significant roadblock in multilingual neural language modeling is the lack of labeled non-English data.
Review Helpfulness Prediction with Embedding-Gated CNN
Product reviews, in the form of texts dominantly, significantly help consumers finalize their purchasing decisions.
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings
This paper presents an effective approach for parallel corpus mining using bilingual sentence embeddings.
A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature
We present a corpus of 5, 000 richly annotated abstracts of medical articles describing clinical randomized controlled trials.
Cross-Domain Review Helpfulness Prediction Based on Convolutional Neural Networks with Auxiliary Domain Discriminators
With the growing amount of reviews in e-commerce websites, it is critical to assess the helpfulness of reviews and recommend them accordingly to consumers.
Syntactic Patterns Improve Information Extraction for Medical Search
Medical professionals search the published literature by specifying the type of patients, the medical intervention(s) and the outcome measure(s) of interest.
Learning Semantic Textual Similarity from Conversations
We present a novel approach to learn representations for sentence-level semantic similarity using conversational data.
Universal Sentence Encoder
For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance.
Ranked #2 on
Text Classification
on TREC-6
Conversational Response Selection
Semantic Textual Similarity
+7
Combining Lexical and Syntactic Features for Detecting Content-dense Texts in News
On manually annotated data, we compare the performance of domain-specific classifiers, trained on data only from a given news domain and a general classifier in which data from all four domains is pooled together.
Aspect Extraction from Product Reviews Using Category Hierarchy Information
Recent work on aspect extraction is leveraging the hierarchical relationship between products and their categories.
Detecting (Un)Important Content for Single-Document News Summarization
We present a robust approach for detecting intrinsic sentence importance in news, by training on two corpora of document-summary pairs.
