Search Results for author:
Found 118 papers, 32 papers with code
Eliciting Bias in Question Answering Models through Ambiguity
We develop two sets of questions for closed and open domain questions respectively, which use ambiguous questions to probe QA models for bias.
Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy
The goal of question answering (QA) is to answer _any_ question.
Distantly-Supervised Dense Retrieval Enables Open-Domain Question Answering without Evidence Annotation
This paper investigates whether models can learn to find evidence from a large corpus, with only distant supervision from answer labels for model training, thereby generating no additional annotation cost.
Evaluation Paradigms in Question Answering
Question answering (QA) primarily descends from two branches of research: (1) Alan Turing’s investigation of machine intelligence at Manchester University and (2) Cyril Cleverdon’s comparison of library card catalog indices at Cranfield University.
What’s in a Name? Answer Equivalence For Open-Domain Question Answering
We incorporate answers for two settings: evaluation with additional answers and model training with equivalent answers.
Human-Centered Evaluation of Explanations
The NLP community are increasingly interested in providing explanations for NLP models to help people make sense of model behavior and potentially improve human interaction with models.
Adapting Entities across Languages and Cultures
He is associated with founding a company in the United States, so perhaps the German founder Carl Benz could stand in for Gates in those contexts.
Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?
Question answering (QA)-producing correct answers for input questions-is popular, but we test a reverse question answering (RQA) task: given an input answer, generate a question with that answer.
Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA
Recent advancements of large language models (LLMs) have led to claims of AI surpassing humans in natural language processing (NLP) tasks such as textual understanding and reasoning.
SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinement
Automating the creation of scientific diagrams from academic papers can significantly streamline the development of tutorials, presentations, and posters, thereby saving time and accelerating the process.
A SMART Mnemonic Sounds like "Glue Tonic": Mixing LLMs with Student Feedback to Make Mnemonic Learning Stick
To train SMART, we first fine-tune LLaMA-2 on a curated set of user-written mnemonics.
KARL: Knowledge-Aware Retrieval and Representations aid Retention and Learning in Students
Flashcard schedulers rely on 1) student models to predict the flashcards a student knows; and 2) teaching policies to pick which cards to show next via these predictions.
Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis
Topic models are a popular tool for understanding text collections, but their evaluation has been a point of contention.
CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert Judgments For Open-Domain Question Answering
Question answering (QA) can only make progress if we know if an answer is correct, but for many of the most challenging and interesting QA examples, current evaluation metrics to determine answer equivalence (AE) often do not align with human judgments, particularly more verbose, free-form answers from large language models (LLM).
How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generation
Dynamic adversarial question generation, where humans write examples to stump a model, aims to create examples that are realistic and informative.
Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering
Questions posed by information-seeking users often contain implicit false or potentially harmful assumptions.
Labeled Interactive Topic Models
To facilitate user interaction with these neural topic models, we have developed an interactive interface.
Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition
We also present a comprehensive taxonomical ontology of the types of adversarial prompts.
Not all Fake News is Written: A Dataset and Analysis of Misleading Video Headlines
Polarization and the marketplace for impressions have conspired to make navigating information online difficult for users, and while there has been a significant effort to detect false or misleading text, multimodal datasets have received considerably less attention.
Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrong
To reduce over-reliance on LLMs, we ask LLMs to provide contrastive information - explain both why the claim is true and false, and then we present both sides of the explanation to users.
MegaWika: Millions of reports and their sources across 50 diverse languages
To foster the development of new models for collaborative AI-assisted report generation, we introduce MegaWika, consisting of 13 million Wikipedia articles in 50 diverse languages, along with their 71 million referenced source materials.
Getting MoRE out of Mixture of Language Model Reasoning Experts
Beyond generalizability, the interpretable design of MoRE improves selective question answering results compared to baselines without incorporating inter-expert agreement.
InteractiveIE: Towards Assessing the Strength of Human-AI Collaboration in Improving the Performance of Information Extraction
Learning template based information extraction from documents is a crucial yet difficult task.
Cheater's Bowl: Human vs. Computer Search Strategies for Open-Domain Question Answering
For humans and computers, the first step in answering an open-domain question is retrieving a set of relevant documents from a large corpus.
Prompting GPT-3 To Be Reliable
While reliability is a broad and vaguely defined term, we decompose reliability into four main facets that correspond to the existing framework of ML safety and are well-recognized to be important: generalizability, social biases, calibration, and factuality.
Improving Question Answering with Generation of NQ-like Questions
We demonstrate that in a low resource setting, using the generated data improves the QA performance over the baseline system on both NQ and QB data.
Re-Examining Calibration: The Case of Question Answering
Building on those observations, we propose a new calibration metric, MacroCE, that better captures whether the model assigns low confidence to wrong predictions and high confidence to correct predictions.
Automatic Song Translation for Tonal Languages
This paper develops automatic song translation (AST) for tonal languages and addresses the unique challenge of aligning words' tones with melody of a song in addition to conveying the original meaning.
Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability
(2) Does the answer to that question change with model adaptation?
Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotation
Open-domain question answering answers a question based on evidence retrieved from a large corpus.
What's in a Name? Answer Equivalence For Open-Domain Question Answering
We incorporate answers for two settings: evaluation with additional answers and model training with equivalent answers.
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items (examples) and subjects (NLP models).
Picard understanding Darmok: A Dataset and Model for Metaphor-Rich Translation in a Constructed Language
Tamarian, a fictional language introduced in the Star Trek episode Darmok, communicates meaning through utterances of metaphorical references, such as "Darmok and Jalad at Tanagra" instead of "We should work together."
Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence
To address the standardization gap, we systematically evaluate a dominant classical model and two state-of-the-art neural models on two commonly used datasets.
Toward Deconfounding the Influence of Entity Demographics for Question Answering Accuracy
The goal of question answering (QA) is to answer any question.
Adapting Coreference Resolution Models through Active Learning
Active learning mitigates this problem by sampling a small subset of data for annotators to label.
Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval
Complex question answering often requires finding a reasoning chain that consists of multiple evidence pieces.
Fool Me Twice: Entailment from Wikipedia Gamification
We release FoolMeTwice (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game.
Complex Factoid Question Answering with a Free-Text Knowledge Graph
DELFT's advantage comes from both the high coverage of its free-text knowledge graph-more than double that of dbpedia relations-and the novel graph neural network which reasons on the rich but noisy free-text evidence.
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims
We introduce CLIMATE-FEVER, a new publicly available dataset for verification of climate change-related claims.
ClimaText: A Dataset for Climate Change Topic Detection
Climate change communication in the mass media and other textual sources may affect and shape public perception.
An Attentive Recurrent Model for Incremental Prediction of Sentence-final Verbs
Verb prediction is important for understanding human processing of verb-final languages, with practical applications to real-time simultaneous interpretation from verb-final to verb-medial languages.
On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries
Large-scale semantic parsing datasets annotated with logical forms have enabled major advances in supervised approaches.
Cold-start Active Learning through Self-supervised Language Modeling
Typically, the active learning strategy is contingent on the classification model.
It Takes Two to Lie: One to Lie, and One to Listen
Trust is implicit in many online text conversations{---}striking up new friendships, or asking for tech support.
Why Overfitting Isn't Always Bad: Retrofitting Cross-Lingual Word Embeddings to Dictionaries
Cross-lingual word embeddings (CLWE) are often evaluated on bilingual lexicon induction (BLI).
Bilingual Lexicon Induction
Cross-Lingual Word Embeddings
+2
Which Evaluations Uncover Sense Representations that Actually Make Sense?
Text representations are critical for modern natural language processing.
Meta Answering for Machine Reading
We investigate a framework for machine reading, inspired by real world information-seeking problems, where a meta question answering system interacts with a black box environment.
Interactive Refinement of Cross-Lingual Word Embeddings
Cross-lingual word embeddings transfer knowledge between languages: models trained on high-resource languages can predict in low-resource languages.
How Pre-trained Word Representations Capture Commonsense Physical Comparisons
One type of common sense is how two objects compare on physical properties such as size and weight: e. g., {`}is a house bigger than a person?{'}.
A Multilingual Topic Model for Learning Weighted Topic Links Across Corpora with Low Comparability
Multilingual topic models (MTMs) learn topics on documents in multiple languages.
Can You Unpack That? Learning to Rewrite Questions-in-Context
Question answering is an AI-complete problem, but existing datasets lack key elements of language understanding such as coreference and ellipsis resolution.
What Question Answering can Learn from Trivia Nerds
In addition to the traditional task of getting machines to answer questions, a major research question in question answering is to create interesting, challenging questions that can help systems learn how to answer questions and also reveal which systems are the best at answering questions.
Mitigating Noisy Inputs for Question Answering
We investigate and mitigate the effects of noise from Automatic Speech Recognition systems on two factoid Question Answering (QA) tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Are Girls Neko or Sh\=ojo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization
Cross-lingual word embeddings (CLWE) underlie many multilingual natural language processing systems, often through orthogonal transformations of pre-trained monolingual embeddings.
A Resource-Free Evaluation Metric for Cross-Lingual Word Embeddings Based on Graph Modularity
Cross-lingual word embeddings encode the meaning of words from different languages into a shared low-dimensional space.
Are Girls Neko or Shōjo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization
Cross-lingual word embeddings (CLWE) underlie many multilingual natural language processing systems, often through orthogonal transformations of pre-trained monolingual embeddings.
Why Didn't You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models
To address the lack of comparative evaluation of Human-in-the-Loop Topic Modeling (HLTM) systems, we implement and evaluate three contrasting HLTM modeling approaches using simulation experiments.
Automatic Evaluation of Local Topic Quality
Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments.
Misleading Failures of Partial-input Baselines
Recent work establishes dataset difficulty and removes annotation artifacts via partial-input baselines (e. g., hypothesis-only models for SNLI or question-only models for VQA).
Quizbowl: The Case for Incremental Question Answering
Throughout this paper, we show that collaborations with the vibrant trivia community have contributed to the quality of our dataset, spawned new research directions, and doubled as an exciting way to engage the public with research in machine learning and natural language processing.
Exploiting Cross-Lingual Subword Similarities in Low-Resource Document Classification
Text classification must sometimes be applied in a low-resource language with no labeled training data.
Cross-Lingual Document Classification
Document Classification
+3
What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play
Machine learning is an important tool for decision making, but its ethical and responsible application requires rigorous vetting of its interpretability and utility: an understudied problem, particularly for natural language processing models.
A dataset and baselines for sequential open-domain question answering
Previous work on question-answering systems mainly focuses on answering individual questions, assuming they are independent and devoid of context.
A Differentiable Self-disambiguated Sense Embedding Model via Scaled Gumbel Softmax
We present a differentiable multi-prototype word representation model that disentangles senses of polysemous words and produces meaningful sense-specific embeddings without external resources.
Interpreting Neural Networks With Nearest Neighbors
However, the confidence of neural networks is not a robust measure of model uncertainty.
Trick Me If You Can: Human-in-the-loop Generation of Adversarial Examples for Question Answering
We propose human-in-the-loop adversarial generation, where human authors are guided to break models.
Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types
Annotated corpora enable supervised machine learning and data analysis.
Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions
Modern question answering systems have been touted as approaching human performance.
Automatic Estimation of Simultaneous Interpreter Performance
Simultaneous interpretation, translation of the spoken word in real-time, is both highly challenging and physically demanding.
Lessons from the Bible on Modern Topics: Low-Resource Multilingual Topic Model Evaluation
Multilingual topic models enable document analysis across languages through coherent multilingual summaries of the data.
Inducing and Embedding Senses with Scaled Gumbel Softmax
Methods for learning word sense embeddings represent a single word with multiple sense-specific vectors.
Pathologies of Neural Models Make Interpretations Difficult
In existing interpretation methods for NLP, a word's importance is determined by either input perturbation---measuring the decrease in model confidence when that word is removed---or by the gradient with respect to that word.
Learning to Color from Language
Automatic colorization is the process of adding color to greyscale images.
Why ADAGRAD Fails for Online Topic Modeling
For online topic modeling, the magnitude of gradients is very large.
Adapting Topic Models using Lexical Associations with Tree Priors
Models work best when they are optimized taking into account the evaluation criteria that people care about.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve.
Tandem Anchoring: a Multiword Anchor Approach for Interactive Topic Modeling
We propose combinations of words as anchors, going beyond existing single word anchor algorithms{---}an approach we call {``}Tandem Anchors{''}.
Evaluating Visual Representations for Topic Understanding and Their Effects on Manually Generated Topic Labels
Probabilistic topic models are important tools for indexing, summarizing, and analyzing large document collections by their themes.
The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
While computers can now describe what is explicitly depicted in natural images, in this paper we examine whether they can understand the closure-driven narratives conveyed by stylized artwork and dialogue in comic book panels.
Opponent Modeling in Deep Reinforcement Learning
Opponent modeling is necessary in multi-agent settings where secondary agents with competing goals also adapt their strategies, yet it remains challenging because strategies interact with each other and change.
Linguistic Harbingers of Betrayal: A Case Study on an Online Strategy Game
Interpersonal relations are fickle, with close friendships often dissolving into enmity.
Online Adaptor Grammars with Hybrid Inference
Adaptor grammars are a flexible, powerful formalism for defining nonparametric, unsupervised models of grammar productions.
