Search Results for author:
Found 114 papers, 30 papers with code
On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries
Large-scale semantic parsing datasets annotated with logical forms have enabled major advances in supervised approaches.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve.
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items (examples) and subjects (NLP models).
Prompting GPT-3 To Be Reliable
While reliability is a broad and vaguely defined term, we decompose reliability into four main facets that correspond to the existing framework of ML safety and are well-recognized to be important: generalizability, social biases, calibration, and factuality.
Opponent Modeling in Deep Reinforcement Learning
Opponent modeling is necessary in multi-agent settings where secondary agents with competing goals also adapt their strategies, yet it remains challenging because strategies interact with each other and change.
Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence
To address the standardization gap, we systematically evaluate a dominant classical model and two state-of-the-art neural models on two commonly used datasets.
Cold-start Active Learning through Self-supervised Language Modeling
Typically, the active learning strategy is contingent on the classification model.
Interpreting Neural Networks With Nearest Neighbors
However, the confidence of neural networks is not a robust measure of model uncertainty.
Trick Me If You Can: Human-in-the-loop Generation of Adversarial Examples for Question Answering
We propose human-in-the-loop adversarial generation, where human authors are guided to break models.
The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
While computers can now describe what is explicitly depicted in natural images, in this paper we examine whether they can understand the closure-driven narratives conveyed by stylized artwork and dialogue in comic book panels.
Learning to Color from Language
Automatic colorization is the process of adding color to greyscale images.
Picard understanding Darmok: A Dataset and Model for Metaphor-Rich Translation in a Constructed Language
Tamarian, a fictional language introduced in the Star Trek episode Darmok, communicates meaning through utterances of metaphorical references, such as "Darmok and Jalad at Tanagra" instead of "We should work together."
Fool Me Twice: Entailment from Wikipedia Gamification
We release FoolMeTwice (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game.
Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval
Complex question answering often requires finding a reasoning chain that consists of multiple evidence pieces.
Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotation
Open-domain question answering answers a question based on evidence retrieved from a large corpus.
Distantly-Supervised Dense Retrieval Enables Open-Domain Question Answering without Evidence Annotation
This paper investigates whether models can learn to find evidence from a large corpus, with only distant supervision from answer labels for model training, thereby generating no additional annotation cost.
Are Girls Neko or Shōjo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization
Cross-lingual word embeddings (CLWE) underlie many multilingual natural language processing systems, often through orthogonal transformations of pre-trained monolingual embeddings.
A Resource-Free Evaluation Metric for Cross-Lingual Word Embeddings Based on Graph Modularity
Cross-lingual word embeddings encode the meaning of words from different languages into a shared low-dimensional space.
Adapting Coreference Resolution Models through Active Learning
Active learning mitigates this problem by sampling a small subset of data for annotators to label.
Automatic Estimation of Simultaneous Interpreter Performance
Simultaneous interpretation, translation of the spoken word in real-time, is both highly challenging and physically demanding.
Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition
We also present a comprehensive taxonomical ontology of the types of adversarial prompts.
Interactive Refinement of Cross-Lingual Word Embeddings
Cross-lingual word embeddings transfer knowledge between languages: models trained on high-resource languages can predict in low-resource languages.
What's in a Name? Answer Equivalence For Open-Domain Question Answering
We incorporate answers for two settings: evaluation with additional answers and model training with equivalent answers.
What’s in a Name? Answer Equivalence For Open-Domain Question Answering
We incorporate answers for two settings: evaluation with additional answers and model training with equivalent answers.
Re-Examining Calibration: The Case of Question Answering
Building on those observations, we propose a new calibration metric, MacroCE, that better captures whether the model assigns low confidence to wrong predictions and high confidence to correct predictions.
Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis
Topic models are a popular tool for understanding text collections, but their evaluation has been a point of contention.
Lessons from the Bible on Modern Topics: Low-Resource Multilingual Topic Model Evaluation
Multilingual topic models enable document analysis across languages through coherent multilingual summaries of the data.
Inducing and Embedding Senses with Scaled Gumbel Softmax
Methods for learning word sense embeddings represent a single word with multiple sense-specific vectors.
Pathologies of Neural Models Make Interpretations Difficult
In existing interpretation methods for NLP, a word's importance is determined by either input perturbation---measuring the decrease in model confidence when that word is removed---or by the gradient with respect to that word.
Linguistic Harbingers of Betrayal: A Case Study on an Online Strategy Game
Interpersonal relations are fickle, with close friendships often dissolving into enmity.
What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play
Machine learning is an important tool for decision making, but its ethical and responsible application requires rigorous vetting of its interpretability and utility: an understudied problem, particularly for natural language processing models.
A dataset and baselines for sequential open-domain question answering
Previous work on question-answering systems mainly focuses on answering individual questions, assuming they are independent and devoid of context.
Evaluating Visual Representations for Topic Understanding and Their Effects on Manually Generated Topic Labels
Probabilistic topic models are important tools for indexing, summarizing, and analyzing large document collections by their themes.
Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions
Modern question answering systems have been touted as approaching human performance.
Tandem Anchoring: a Multiword Anchor Approach for Interactive Topic Modeling
We propose combinations of words as anchors, going beyond existing single word anchor algorithms{---}an approach we call {``}Tandem Anchors{''}.
Why ADAGRAD Fails for Online Topic Modeling
For online topic modeling, the magnitude of gradients is very large.
Adapting Topic Models using Lexical Associations with Tree Priors
Models work best when they are optimized taking into account the evaluation criteria that people care about.
Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types
Annotated corpora enable supervised machine learning and data analysis.
Exploiting Cross-Lingual Subword Similarities in Low-Resource Document Classification
Text classification must sometimes be applied in a low-resource language with no labeled training data.
Cross-Lingual Document Classification
Document Classification
+3
Online Adaptor Grammars with Hybrid Inference
Adaptor grammars are a flexible, powerful formalism for defining nonparametric, unsupervised models of grammar productions.
Quizbowl: The Case for Incremental Question Answering
Throughout this paper, we show that collaborations with the vibrant trivia community have contributed to the quality of our dataset, spawned new research directions, and doubled as an exciting way to engage the public with research in machine learning and natural language processing.
Misleading Failures of Partial-input Baselines
Recent work establishes dataset difficulty and removes annotation artifacts via partial-input baselines (e. g., hypothesis-only models for SNLI or question-only models for VQA).
Why Didn't You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models
To address the lack of comparative evaluation of Human-in-the-Loop Topic Modeling (HLTM) systems, we implement and evaluate three contrasting HLTM modeling approaches using simulation experiments.
Automatic Evaluation of Local Topic Quality
Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments.
Are Girls Neko or Sh\=ojo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization
Cross-lingual word embeddings (CLWE) underlie many multilingual natural language processing systems, often through orthogonal transformations of pre-trained monolingual embeddings.
Mitigating Noisy Inputs for Question Answering
We investigate and mitigate the effects of noise from Automatic Speech Recognition systems on two factoid Question Answering (QA) tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
What Question Answering can Learn from Trivia Nerds
In addition to the traditional task of getting machines to answer questions, a major research question in question answering is to create interesting, challenging questions that can help systems learn how to answer questions and also reveal which systems are the best at answering questions.
How Pre-trained Word Representations Capture Commonsense Physical Comparisons
One type of common sense is how two objects compare on physical properties such as size and weight: e. g., {`}is a house bigger than a person?{'}.
A Multilingual Topic Model for Learning Weighted Topic Links Across Corpora with Low Comparability
Multilingual topic models (MTMs) learn topics on documents in multiple languages.
Can You Unpack That? Learning to Rewrite Questions-in-Context
Question answering is an AI-complete problem, but existing datasets lack key elements of language understanding such as coreference and ellipsis resolution.
Meta Answering for Machine Reading
We investigate a framework for machine reading, inspired by real world information-seeking problems, where a meta question answering system interacts with a black box environment.
Why Overfitting Isn't Always Bad: Retrofitting Cross-Lingual Word Embeddings to Dictionaries
Cross-lingual word embeddings (CLWE) are often evaluated on bilingual lexicon induction (BLI).
Bilingual Lexicon Induction
Cross-Lingual Word Embeddings
+2
Which Evaluations Uncover Sense Representations that Actually Make Sense?
Text representations are critical for modern natural language processing.
It Takes Two to Lie: One to Lie, and One to Listen
Trust is implicit in many online text conversations{---}striking up new friendships, or asking for tech support.
CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims
We introduce CLIMATE-FEVER, a new publicly available dataset for verification of climate change-related claims.
ClimaText: A Dataset for Climate Change Topic Detection
Climate change communication in the mass media and other textual sources may affect and shape public perception.
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
Complex Factoid Question Answering with a Free-Text Knowledge Graph
DELFT's advantage comes from both the high coverage of its free-text knowledge graph-more than double that of dbpedia relations-and the novel graph neural network which reasons on the rich but noisy free-text evidence.
Toward Deconfounding the Influence of Entity Demographics for Question Answering Accuracy
The goal of question answering (QA) is to answer any question.
An Attentive Recurrent Model for Incremental Prediction of Sentence-final Verbs
Verb prediction is important for understanding human processing of verb-final languages, with practical applications to real-time simultaneous interpretation from verb-final to verb-medial languages.
Adapting Entities across Languages and Cultures
He is associated with founding a company in the United States, so perhaps the German founder Carl Benz could stand in for Gates in those contexts.
Eliciting Bias in Question Answering Models through Ambiguity
We develop two sets of questions for closed and open domain questions respectively, which use ambiguous questions to probe QA models for bias.
Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy
The goal of question answering (QA) is to answer _any_ question.
Evaluation Paradigms in Question Answering
Question answering (QA) primarily descends from two branches of research: (1) Alan Turing’s investigation of machine intelligence at Manchester University and (2) Cyril Cleverdon’s comparison of library card catalog indices at Cranfield University.
A Differentiable Self-disambiguated Sense Embedding Model via Scaled Gumbel Softmax
We present a differentiable multi-prototype word representation model that disentangles senses of polysemous words and produces meaningful sense-specific embeddings without external resources.
Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability
(2) Does the answer to that question change with model adaptation?
Automatic Song Translation for Tonal Languages
This paper develops automatic song translation (AST) for tonal languages and addresses the unique challenge of aligning words' tones with melody of a song in addition to conveying the original meaning.
Human-Centered Evaluation of Explanations
The NLP community are increasingly interested in providing explanations for NLP models to help people make sense of model behavior and potentially improve human interaction with models.
Improving Question Answering with Generation of NQ-like Questions
We demonstrate that in a low resource setting, using the generated data improves the QA performance over the baseline system on both NQ and QB data.
Cheater's Bowl: Human vs. Computer Search Strategies for Open-Domain Question Answering
For humans and computers, the first step in answering an open-domain question is retrieving a set of relevant documents from a large corpus.
Getting MoRE out of Mixture of Language Model Reasoning Experts
Beyond generalizability, the interpretable design of MoRE improves selective question answering results compared to baselines without incorporating inter-expert agreement.
InteractiveIE: Towards Assessing the Strength of Human-AI Collaboration in Improving the Performance of Information Extraction
Learning template based information extraction from documents is a crucial yet difficult task.
MegaWika: Millions of reports and their sources across 50 diverse languages
To foster the development of new models for collaborative AI-assisted report generation, we introduce MegaWika, consisting of 13 million Wikipedia articles in 50 diverse languages, along with their 71 million referenced source materials.
Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrong
To reduce over-reliance on LLMs, we ask LLMs to provide contrastive information - explain both why the claim is true and false, and then we present both sides of the explanation to users.
Not all Fake News is Written: A Dataset and Analysis of Misleading Video Headlines
Polarization and the marketplace for impressions have conspired to make navigating information online difficult for users, and while there has been a significant effort to detect false or misleading text, multimodal datasets have received considerably less attention.
Labeled Interactive Topic Models
To facilitate user interaction with these neural topic models, we have developed an interactive interface.
Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering
Questions posed by information-seeking users often contain implicit false or potentially harmful assumptions.
How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generation
Dynamic adversarial question generation, where humans write examples to stump a model, aims to create examples that are realistic and informative.
CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert Judgments For Open-Domain Question Answering
Question answering (QA) can only make progress if we know if an answer is correct, but for many of the most challenging and interesting QA examples, current evaluation metrics to determine answer equivalence (AE) often do not align with human judgments, particularly more verbose, free-form answers from large language models (LLM).
KARL: Knowledge-Aware Retrieval and Representations aid Retention and Learning in Students
Flashcard schedulers are tools that rely on 1) student models to predict the flashcards a student knows; and 2) teaching policies to schedule cards based on these predictions.
