Search Results for author:
Found 104 papers, 48 papers with code
Improving Factuality with Explicit Working Memory
Large language models can generate factually inaccurate content, a problem known as hallucination.
Memory Layers at Scale
We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters.
OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
Scientific progress depends on researchers' ability to synthesize the growing body of literature.
Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models
In the Transfusion setting, where text and image are trained with different objectives, a 7B MoT model matches the image modality performance of the dense baseline with one third of the FLOPs, and a 760M MoT model outperforms a 1. 4B dense baseline across key image generation metrics.
Altogether: Image Captioning via Re-aligning Alt-text
This paper focuses on creating synthetic data to improve the quality of image captions.
CRAG -- Comprehensive RAG Benchmark
To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual question answering benchmark of 4, 409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search.
Nearest Neighbor Speculative Decoding for LLM Generation and Attribution
Large language models (LLMs) often hallucinate and lack the ability to provide attribution for their generations.
FLAME: Factuality-Aware Alignment for Large Language Models
Furthermore, reward functions used in standard RL can also encourage hallucination, because it guides the LLM to provide more helpful responses on a diverse set of instructions, often preferring longer and more detailed responses.
MoDE: CLIP Data Experts via Clustering
The success of contrastive language-image pretraining (CLIP) relies on the supervision from the pairing between images and captions, which tends to be noisy in web-crawled data.
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge.
Ranked #38 on
Common Sense Reasoning
on WinoGrande
Reliable, Adaptable, and Attributable Language Models with Retrieval
Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability.
Instruction-tuned Language Models are Better Knowledge Learners
The standard recipe for doing so involves continued pre-training on new documents followed by instruction-tuning on question-answer (QA) pairs.
Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering
We propose EAR, a query Expansion And Reranking approach for improving passage retrieval, with the application to open-domain question answering.
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate, and (2) human evaluation is time-consuming and costly.
Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering
Few-shot learning for open domain multi-hop question answering typically relies on the incontext learning capability of large language models (LLMs).
Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing
In this work, we propose a new task of simulating NL feedback for interactive semantic parsing.
Large Language Model Programs
In recent years, large pre-trained language models (LLMs) have demonstrated the ability to follow instructions and perform novel tasks from a few examples.
VideoOFA: Two-Stage Pre-Training for Video-to-Text Generation
We propose a new two-stage pre-training framework for video-to-text generation tasks such as video captioning and video question answering: A generative encoder-decoder model is first jointly pre-trained on massive image-text data to learn fundamental vision-language concepts, and then adapted to video data in an intermediate video-text pre-training stage to learn video-specific skills such as spatio-temporal reasoning.
LEVER: Learning to Verify Language-to-Code Generation with Execution
The advent of large language models trained on code (code LLMs) has led to significant progress in language-to-code generation.
Ranked #2 on
Semantic Parsing
on spider
How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval
We hence propose a new DA approach with diverse queries and sources of supervision to progressively train a generalizable DR. As a result, DRAGON, our dense retriever trained with diverse augmentation, is the first BERT-base-sized DR to achieve state-of-the-art effectiveness in both supervised and zero-shot evaluations and even competes with models using more complex late interaction (ColBERTv2 and SPLADE++).
REPLUG: Retrieval-Augmented Black-Box Language Models
We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model.
Ranked #15 on
Question Answering
on Natural Questions
One Embedder, Any Task: Instruction-Finetuned Text Embeddings
Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets.
Nonparametric Masked Language Modeling
Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases.
Coder Reviewer Reranking for Code Generation
Sampling diverse programs from a code language model and reranking with model likelihood is a popular method for code generation but it is prone to preferring degenerate solutions.
Ranked #30 on
Code Generation
on MBPP
Retrieval-Augmented Multimodal Language Modeling
To integrate knowledge in a more scalable and modular way, we propose a retrieval-augmented multimodal model, which enables a base multimodal model (generator) to refer to relevant text and images fetched by a retriever from external memory (e. g., documents on the web).
Ranked #8 on
Image Captioning
on MS COCO
CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval
In this paper, we unify different multi-vector retrieval models from a token routing viewpoint and propose conditional token interaction via dynamic lexical routing, namely CITADEL, for efficient and effective multi-vector retrieval.
Task-aware Retrieval with Instructions
We study the problem of retrieval with instructions, where users of a retrieval system explicitly describe their intent along with their queries.
RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering
Moreover, existing models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions.
Adapting Pretrained Text-to-Text Models for Long Text Sequences
We present an empirical study of adapting an existing pretrained text-to-text model for long-sequence inputs.
Ranked #1 on
Text Summarization
on QMSum
Structured Prompt Tuning
We propose structured prompt tuning, a simple and effective method to improve prompt tuning.
On Continual Model Refinement in Out-of-Distribution Data Streams
Real-world natural language processing (NLP) models need to be continually updated to fix the prediction errors in out-of-distribution (OOD) data streams while overcoming catastrophic forgetting.
Autoregressive Search Engines: Generating Substrings as Document Identifiers
Knowledge-intensive language tasks require NLP systems to both provide the correct answer and retrieve supporting evidence for it in a given corpus.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings.
Ranked #13 on
Semantic Textual Similarity
on STS16
Improving Passage Retrieval with Zero-Shot Question Generation
We propose a simple and effective re-ranking method for improving passage retrieval in open question answering.
InCoder: A Generative Model for Code Infilling and Synthesis
Our model is the first generative model that is able to directly perform zero-shot code infilling, which we evaluate on challenging tasks such as type inference, comment generation, and variable re-naming.
Ranked #92 on
Code Generation
on MBPP
The Web Is Your Oyster -- Knowledge-Intensive NLP against a Very Large Web Corpus
In order to address increasing demands of real-world applications, the research for knowledge-intensive NLP (KI-NLP) should advance by capturing the challenges of a truly open-domain environment: web-scale knowledge, lack of structure, inconsistent quality and noise.
Boosted Dense Retriever
DrBoost is trained in stages: each component model is learned sequentially and specialized by focusing only on retrieval mistakes made by the current ensemble.
Simple Local Attentions Remain Competitive for Long-Context Tasks
Many NLP tasks require processing long contexts beyond the length limit of pretrained models.
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning
Recent parameter-efficient language model tuning (PELT) methods manage to match the performance of fine-tuning with much fewer trainable parameters and perform especially well when training data is limited.
CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training
Consequently, we propose a novel QA dataset based on the Common Crawl project in this paper.
Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?
Despite their recent popularity and well-known advantages, dense retrievers still lag behind sparse methods such as BM25 in their ability to reliably match salient phrases and rare entities in the query and to generalize to out-of-domain data.
Ranked #2 on
Passage Retrieval
on EntityQuestions
Domain-matched Pre-training Tasks for Dense Retrieval
Pre-training on larger datasets with ever increasing model size is now a proven recipe for increased performance across almost all NLP tasks.
Ranked #2 on
Passage Retrieval
on Natural Questions
(using extra training data)
On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study
In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions.
RECONSIDER: Improved Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering
State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain Question Answering (QA) are typically trained for span selection using distantly supervised positive examples and heuristically retrieved negative examples.
On the Influence of Masking Policies in Intermediate Pre-training
Current NLP models are predominantly trained through a two-stage "pre-train then fine-tune" pipeline.
On Unifying Misinformation Detection
In this paper, we introduce UnifiedM2, a general-purpose misinformation model that jointly models multiple domains of misinformation with a single, unified setup.
Multi-task Retrieval for Knowledge-Intensive Tasks
Retrieving relevant contexts from a large corpus is a crucial step for tasks such as open-domain question answering and fact checking.
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
FiD-Ex: Improving Sequence-to-Sequence Models for Extractive Rationale Generation
Natural language (NL) explanations of model predictions are gaining popularity as a means to understand and verify decisions made by large black-box pre-trained models, for NLP tasks such as Question Answering (QA) and Fact Verification.
Studying Strategically: Learning to Mask for Closed-book QA
Thus, our policy packs task-relevant knowledge into the parameters of a language model.
Joint Verification and Reranking for Open Fact Checking Over Tables
Structured information is an important knowledge source for automatic verification of factual claims.
RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering
State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain Question Answering (QA) are typically trained for span selection using distantly supervised positive examples and heuristically retrieved negative examples.
Efficient One-Pass End-to-End Entity Linking for Questions
We present ELQ, a fast end-to-end entity linking model for questions, which uses a biencoder to jointly perform mention detection and linking in one pass.
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
We propose a simple and efficient multi-hop dense retrieval approach for answering complex open-domain questions, which achieves state-of-the-art performance on two multi-hop datasets, HotpotQA and multi-evidence FEVER.
Ranked #14 on
Question Answering
on HotpotQA
Open-Domain Question Answering
This tutorial provides a comprehensive and coherent overview of cutting-edge research in open-domain question answering (QA), the task of answering questions using a large collection of documents of diversified topics.
Language Models as Fact Checkers?
Recent work has suggested that language models (LMs) store both common-sense and factual knowledge learned from pre-training data.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks.
Ranked #4 on
Question Answering
on WebQuestions
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks.
Ranked #12 on
Text-To-SQL
on spider
(Exact Match Accuracy (Dev) metric)
An Imitation Game for Learning Semantic Parsers from User Interaction
Despite the widely successful applications, bootstrapping and fine-tuning semantic parsers are still a tedious process with challenges such as costly data annotation and privacy risks.
Dense Passage Retrieval for Open-Domain Question Answering
Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method.
Ranked #1 on
Question Answering
on NaturalQA
Unsupervised Question Decomposition for Question Answering
We aim to improve question answering (QA) by decomposing hard questions into simpler sub-questions that existing QA systems are capable of answering.
Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study
As a promising paradigm, interactive semantic parsing has shown to improve both semantic parsing accuracy and user confidence in the results.
Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text
Our goal is to better comprehend procedural text, e. g., a paragraph about photosynthesis, by not only predicting what happens, but why some actions need to happen before others.
Be Consistent! Improving Procedural Text Comprehension using Label Consistency
Our goal is procedural text comprehension, namely tracking how the properties of entities (e. g., their location) change with time given a procedural text (e. g., a paragraph about photosynthesis, a recipe).
QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships
Many natural language questions require recognizing and reasoning with qualitative relationships (e. g., in science, economics, and medicine), but are challenging to answer with corpus-based methods.
FlowQA: Grasping Flow in History for Conversational Machine Comprehension
Conversational machine comprehension requires the understanding of the conversation history, such as previous question/answer pairs, the document context, and the current question.
Ranked #1 on
Question Answering
on QuAC
QuAC: Question Answering in Context
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).
Policy Shaping and Generalized Update Equations for Semantic Parsing from Denotations
Semantic parsing from denotations faces two key challenges in model training: (1) given only the denotations (e. g., answers), search for good candidate semantic parses, and (2) choose the best model update algorithm.
Reasoning about Actions and State Changes by Injecting Commonsense Knowledge
Comprehending procedural text, e. g., a paragraph describing photosynthesis, requires modeling actions and the state changes they produce, so that questions about entities at different timepoints can be answered.
Dissecting Contextual Word Embeddings: Architecture and Representation
Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks.
QuAC : Question Answering in Context
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).
Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension
The new dataset, ProPara, is the first to contain natural (rather than machine-generated) text about a changing world along with a full annotation of entity states (location and existence) during those changes (81k datapoints).
Ranked #4 on
Procedural Text Understanding
on ProPara
Natural Language to Structured Query Generation via Meta-Learning
In conventional supervised training, a model is trained to fit all the training examples.
Ranked #7 on
Code Generation
on WikiSQL
Maximum Margin Reward Networks for Learning from Explicit and Implicit Supervision
Neural networks have achieved state-of-the-art performance on several structured-output prediction tasks, trained in a fully supervised fashion.
Cross-Sentence N-ary Relation Extraction with Graph LSTMs
Past work in relation extraction has focused on binary relations in single sentences.
NLP for Precision Medicine
We will introduce precision medicine and showcase the vast opportunities for NLP in this burgeoning field with great societal impact.
Search-based Neural Structured Learning for Sequential Question Answering
Recent work in semantic parsing for question answering has focused on long and complicated questions, many of which would seem unnatural if asked in a normal conversation between two humans.
A Knowledge-Grounded Neural Conversation Model
We generalize the widely-used Seq2Seq approach by conditioning responses on both conversation history and external "facts", allowing the model to be versatile and applicable in an open-domain setting.
Answering Complicated Question Intents Expressed in Decomposed Question Sequences
Recent work in semantic parsing for question answering has focused on long and complicated questions, many of which would seem unnatural if asked in a normal conversation between two humans.
Basic Reasoning with Tensor Product Representations
In this paper we present the initial development of a general theory for mapping inference in predicate logic to computation over Tensor Product Representations (TPRs; Smolensky (1990), Smolensky & Legendre (2006)).
Reasoning in Vector Space: An Exploratory Study of Question Answering
Question answering tasks have shown remarkable progress with distributed vector representation.
Embedding Entities and Relations for Learning and Inference in Knowledge Bases
We consider learning representations of entities and relations in KBs using the neural-embedding approach.
Ranked #10 on
Link Prediction
on UMLS
Learning Multi-Relational Semantics Using Neural-Embedding Models
In this paper we present a unified framework for modeling multi-relational representations, scoring, and learning, and conduct an empirical study of several recent multi-relational embedding models under the framework.
Learning Semantic Representations for the Phrase Translation Model
The results show that the new semantic-based phrase translation model significantly improves the performance of a state-of-the-art phrase-based statistical machine translation sys-tem, leading to a gain of 0. 7-1. 0 BLEU points.
Dual Coordinate Descent Algorithms for Efficient Large Margin Structured Prediction
Due to the nature of complex NLP problems, structured prediction algorithms have been important modeling tools for a wide range of tasks.
