Search Results for author:
Found 80 papers, 40 papers with code
Misinfo Reaction Frames: Reasoning about Readers’ Reactions to News Headlines
Even to a simple and short news headline, readers react in a multitude of ways: cognitively (e. g. inferring the writer’s intent), emotionally (e. g. feeling distrust), and behaviorally (e. g. sharing the news with their friends).
Can NLI Models Verify QA Systems’ Predictions?
To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just “good enough” in the context of imperfect QA datasets.
longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks.
Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability.
Future of Information Retrieval Research in the Age of Generative AI
In the fast-evolving field of information retrieval (IR), the integration of generative AI technologies such as large language models (LLMs) is transforming how users search for and interact with information.
Recycled Attention: Efficient inference for long-context language models
Generating long sequences of tokens given a long-context input imposes a heavy computational burden for large language models (LLMs).
RARe: Retrieval Augmented Retrieval with In-Context Examples
Our approach, RARe, finetunes a pre-trained model with in-context examples whose query is semantically similar to the target query.
Diverging Preferences: When do Annotators Disagree and do Models Know?
We find that the majority of disagreements are in opposition with standard reward modeling approaches, which are designed with the assumption that annotator disagreement is noise.
Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions
This allows LLMs to learn to ask clarifying questions when it can generate responses that are tailored to each user interpretation in future turns.
Contrastive Learning to Improve Retrieval for Real-world Fact Checking
By leveraging the AVeriTeC dataset, which annotates subquestions for claims with human written answers from evidence documents, we fine-tune Contriever with a contrastive objective based on multiple training signals, including distillation from GPT-4, evaluating subquestion answers, and gold labels in the dataset.
Open-World Evaluation for Retrieving Diverse Perspectives
On this data, retrievers paired with a corpus are evaluated to surface a document set that contains diverse perspectives.
Long-Form Answers to Visual Questions from Blind and Low Vision People
Vision language models can now generate long-form answers to questions about images - long-form visual question answers (LFVQA).
CodeUpdateArena: Benchmarking Knowledge Editing on API Updates
An instance in our benchmark consists of a synthetic API function update paired with a program synthesis example that uses the updated functionality; our goal is to update an LLM to be able to solve this program synthesis example without providing documentation of the update at inference time.
CaLMQA: Exploring culturally specific long-form question answering across 23 languages
To bridge this gap, we introduce CaLMQA, a collection of 1. 5K complex culturally specific questions spanning 23 languages and 51 culturally agnostic questions translated from English into 22 other languages.
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment
Alignment suppresses irrelevant and unhelpful content while shifting the output distribution toward longer responses that cover information spanning several responses from the base LLM, essentially presenting diverse information in a single response.
Exploring Design Choices for Building Language-Specific LLMs
In this paper, we study building language-specific LLMs by adapting monolingual and multilingual LLMs.
SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
Extensive experiments on language and vision benchmarks show that SVFT recovers up to 96% of full fine-tuning performance while training only 0. 006 to 0. 25% of parameters, outperforming existing methods that only recover up to 85% performance using 0. 03 to 0. 8% of the trainable parameter budget.
AmbigDocs: Reasoning across Documents on Different Entities under the Same Name
and a set of documents discussing different people named Michael Jordan, can LMs distinguish entity mentions to generate a cohesive answer to the question?
KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions
To evaluate the capabilities of current LLMs on this task, we construct KIWI, a dataset of knowledge-intensive writing instructions in the scientific domain.
BAT: Learning to Reason about Spatial Sounds with Large Language Models
By integrating Spatial-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment.
Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs
In this work, we study such behavior in LMs by proposing a task-agnostic framework for resolving ambiguity by asking users clarifying questions.
Crafting In-context Examples according to LMs' Parametric Knowledge
We perform analysis on three multi-answer question answering datasets, which allows us to further study answer set ordering strategies based on the LM's knowledge of each answer.
Understanding Retrieval Augmentation for Long-Form Question Answering
Our study provides new insights on how retrieval augmentation impacts long, knowledge-rich text generation of LMs.
RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
Retrieving documents and prepending them in-context at inference time improves performance of language model (LMs) on a wide range of tasks.
Development and Evaluation of Three Chatbots for Postpartum Mood and Anxiety Disorders
In collaboration with Postpartum Support International (PSI), a non-profit organization dedicated to supporting caregivers with postpartum mood and anxiety disorders, we developed three chatbots to provide context-specific empathetic support to postpartum caregivers, leveraging both rule-based and generative models.
Propagating Knowledge Updates to LMs Through Distillation
Then, we update the model parameters so that the distribution of the LM (the student) matches the distribution of the LM conditioned on the definition (the teacher) on the transfer set.
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants
We present the first unified study of the efficiency of self-attention-based Transformer variants spanning text, speech and vision.
Concise Answers to Complex Questions: Summarization of Long-form Answers
Long-form question answering systems provide rich information by presenting paragraph-level answers, often containing optional background or auxiliary information.
A Critical Evaluation of Evaluations for Long-form Question Answering
We present a careful analysis of experts' evaluation, which focuses on new aspects such as the comprehensiveness of the answer.
Textless Speech-to-Speech Translation With Limited Parallel Data
We first pretrain a model on large-scale monolingual speech data, finetune it with a small amount of parallel speech data (20-60 hours), and lastly train with an unsupervised backtranslation objective.
Mitigating Temporal Misalignment by Discarding Outdated Facts
While large language models are able to retain vast amounts of world knowledge seen during pretraining, such knowledge is prone to going out of date and is nontrivial to update.
Continually Improving Extractive QA via Human Feedback
We study continually improving an extractive question answering (QA) system via human user feedback.
Complex Claim Verification with Evidence Retrieved in the Wild
Evidence retrieval is a core part of automatic fact-checking.
Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge
Pre-trained language models (LMs) are used for knowledge intensive tasks like question answering, but their knowledge gets continuously outdated as the world changes.
DIFFQG: Generating Questions to Summarize Factual Changes
We propose representing factual changes between paired documents as question-answer pairs, where the answer to the same question differs between two versions.
Learning to Reject with a Fixed Predictor: Application to Decontextualization
We study the problem of classification with a reject option for a fixed predictor, applicable in natural language processing.
Understanding Postpartum Parents' Experiences via Two Digital Platforms
In the current work, we provide a descriptive analysis of the concerns, psychological states, and motivations shared by healthy and distressed postpartum support seekers on two digital platforms, a one-on-one digital helpline and a publicly available online forum.
Continual Learning for On-Device Speech Recognition using Disentangled Conformers
Additionally, current speech recognition models and continual learning algorithms are not optimized to be compute-efficient.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages
We study politeness phenomena in nine typologically diverse languages.
Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources
While the NLP community is generally aware of resource disparities among languages, we lack research that quantifies the extent and types of such disparity.
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality
Recent visuolinguistic pre-trained models show promising progress on various end tasks such as image retrieval and video captioning.
Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence
Question answering models can use rich knowledge sources -- up to one hundred retrieved passages and parametric knowledge in the large-scale language model (LM).
MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languages
We present the results of the Workshop on Multilingual Information Access (MIA) 2022 Shared Task, evaluating cross-lingual open-retrieval question answering (QA) systems in 16 typologically diverse languages.
longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks
Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability.
Modeling Exemplification in Long-form Question Answering via Retrieval
We show that not only do state-of-the-art LFQA models struggle to generate relevant examples, but also that standard evaluation metrics such as ROUGE are insufficient to judge exemplification quality.
Generating Literal and Implied Subquestions to Fact-check Complex Claims
Verifying complex political claims is a challenging task, especially when politicians use various tactics to subtly misrepresent the facts.
Entity Cloze By Date: What LMs Know About Unseen Entities
Given its wide coverage on entity knowledge and temporal indexing, our dataset can be used to evaluate LMs and techniques designed to modify or extend their knowledge.
How Do We Answer Complex Questions: Discourse Structure of Long-form Answers
Long-form answers, consisting of multiple sentences, can provide nuanced and comprehensive answers to a broader set of questions.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We study learning from user feedback for extractive question answering by simulating feedback using supervised data.
SituatedQA: Incorporating Extra-Linguistic Contexts into QA
To construct SituatedQA, we first identify such questions in existing QA datasets.
Learning with Different Amounts of Annotation: From Zero to Many Labels
Introducing such multi label examples at the cost of annotating fewer examples brings clear gains on natural language inference task and entity typing task, even when we simply first train with a single label data and then fine tune with multi label examples.
CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge
We introduce CREAK, a testbed for commonsense reasoning about entity knowledge, bridging fact-checking about entities (Harry Potter is a wizard and is skilled at riding a broomstick) with commonsense inferences (if you're good at a skill you can teach others how to do it).
Knowing More About Questions Can Help: Improving Calibration in Question Answering
We study calibration in question answering, estimating whether model correctly predicts answer for each question.
Can NLI Models Verify QA Systems' Predictions?
To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just "good enough" in the context of imperfect QA datasets.
Misinfo Reaction Frames: Reasoning about Readers' Reactions to News Headlines
We propose Misinfo Reaction Frames (MRF), a pragmatic formalism for modeling how readers might react to a news headline.
Capturing Label Distribution: A Case Study in NLI
We depart from the standard practice of collecting a single reference per each training example, and find that collecting multiple references can achieve better accuracy under the fixed annotation budget.
Decontextualization: Making Sentences Stand-Alone
Models for question answering, dialogue agents, and summarization often interpret the meaning of a sentence in a rich context and use that meaning in a new context.
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval
However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging.
XOR QA: Cross-lingual Open-Retrieval Question Answering
Multilingual question answering tasks typically assume answers exist in the same language as the question.
QED: A Framework and Dataset for Explanations in Question Answering
A question answering system that in addition to providing an answer provides an explanation of the reasoning that leads to that answer has potential advantages in terms of debuggability, extensibility and trust.
Entities as Experts: Sparse Memory Access with Entity Supervision
We introduce a new model - Entities as Experts (EAE) - that can access distinct memories of the entities mentioned in a piece of text.
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations.
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems.
No Permanent Friends or Enemies: Tracking Relationships between Nations from News
Understanding the dynamics of international politics is important yet challenging for civilians.
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
Reasoning about implied relationships (e. g., paraphrastic, common sense, encyclopedic) between pairs of words is crucial for many cross-sentence inference problems.
FlowQA: Grasping Flow in History for Conversational Machine Comprehension
Conversational machine comprehension requires the understanding of the conversation history, such as previous question/answer pairs, the document context, and the current question.
Ranked #1 on
Question Answering
on QuAC
QuAC: Question Answering in Context
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).
Neural Metaphor Detection in Context
We present end-to-end neural models for detecting metaphorical word use in context.
QuAC : Question Answering in Context
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).
Ultra-Fine Entity Typing
We introduce a new entity typing task: given a sentence with an entity mention, the goal is to predict a set of free-form phrases (e. g. skyscraper, songwriter, or criminal) that describe appropriate types for the target entity.
Ranked #4 on
Entity Typing
on Ontonotes v5 (English)
Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking
We present an analytic study on the language of news media in the context of political fact-checking and fake news detection.
Coarse-to-Fine Question Answering for Long Documents
We present a framework for question answering that can efficiently scale to longer documents while maintaining or even improving performance of state-of-the-art models.
Zero-Shot Relation Extraction via Reading Comprehension
We show that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot.
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples.
Hierarchical Question Answering for Long Documents
We present a framework for question answering that can efficiently scale to longer documents while maintaining or even improving performance of state-of-the-art models.
Extracting Structured Scholarly Information from the Machine Translation Literature
Understanding the experimental results of a scientific paper is crucial to understanding its contribution and to comparing it with related work.
