Search Results for author:
Found 45 papers, 24 papers with code
Robustness and Adversarial Examples in Natural Language Processing
In particular, we will review recent studies on analyzing the weakness of NLP systems when facing adversarial inputs and data with a distribution shift.
IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations
Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs.
Proving membership in LLM pretraining data via data watermarks
Detecting whether copyright holders' works were used in LLM pretraining is poised to be an important problem.
Does VLN Pretraining Work with Nonsensical or Irrelevant Instructions?
Data augmentation via back-translation is common when pretraining Vision-and-Language Navigation (VLN) models, even though the generated instructions are noisy.
Efficient End-to-End Visual Document Understanding with Rationale Distillation
Pre-processing tools, such as optical character recognition (OCR), can map document image inputs to textual tokens, then large language models (LLMs) can reason over text.
Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks
On the other hand, even successful methods identify neurons that are not specific to a single memorized sequence.
Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models
In this paper, we instead demonstrate that Transformers learn to implement higher-order optimization methods to perform ICL.
Estimating Large Language Model Capabilities without Labeled Test Data
In this paper, we propose the task of ICL accuracy estimation, in which we predict the accuracy of an LLM when doing in-context learning on a new task given only unlabeled test data for that task.
Chain-of-Questions Training with Latent Answers for Robust Multistep Question Answering
We train a language model (LM) to robustly answer multistep questions by generating and answering sub-questions.
How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench
We investigate the predictability of large language model (LLM) capabilities: given records of past experiments using different model families, numbers of parameters, tasks, and numbers of in-context examples, can we accurately predict LLM performance on new experiment configurations?
SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples
In this work, we propose Self-labeled Counterfactuals for Extrapolating to Negative Examples (SCENE), an automatic method for synthesizing training data that greatly improves models' ability to detect challenging negative examples.
Data Curation Alone Can Stabilize In-context Learning
Across five tasks and two LLMs, sampling from stable subsets selected by CondAcc and Datamodels improves average accuracy over sampling from the entire training set by 7. 7% and 6. 3%, respectively.
Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large Language Models
In many task settings, text classification models are likely to encounter examples from novel classes on which they cannot predict correctly.
Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems
For vision-and-language reasoning tasks, both fully connectionist, end-to-end methods and hybrid, neuro-symbolic methods have achieved high in-distribution performance.
Benchmarking Long-tail Generalization with Likelihood Splits
In order to reliably process natural language, NLP systems must generalize to the long tail of rare utterances.
Are Sample-Efficient NLP Models More Robust?
Recent results in image classification and extractive question answering have observed that pre-trained models trained on less in-distribution data have better out-of-distribution performance.
On Continual Model Refinement in Out-of-Distribution Data Streams
Real-world natural language processing (NLP) models need to be continually updated to fix the prediction errors in out-of-distribution (OOD) data streams while overcoming catastrophic forgetting.
Knowledge Base Question Answering by Case-based Reasoning over Subgraphs
Question answering (QA) over knowledge bases (KBs) is challenging because of the diverse, essentially unbounded, types of reasoning patterns needed.
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants
We collect training datasets in twenty experimental settings and perform a detailed analysis of this approach for the task of extractive question answering (QA) for both standard and adversarial data collection.
Analyzing Dynamic Adversarial Training Data in the Limit
To create models that are robust across a wide range of test inputs, training datasets should include diverse examples that span numerous phenomena.
On the Robustness of Reading Comprehension Models to Entity Renaming
We study the robustness of machine reading comprehension (MRC) models to entity renaming -- do models make more wrong predictions when the same questions are asked about an entity whose name has been changed?
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items (examples) and subjects (NLP models).
Question Answering Infused Pre-training of General-Purpose Contextualized Representations
We propose a pre-training objective based on question answering (QA) for learning general-purpose contextual representations, motivated by the intuition that the representation of a phrase in a passage should encode all questions that the phrase can answer in context.
Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality
We release a new benchmark for lexical substitution, the task of finding appropriate substitutes for a target word in a context.
The statistical advantage of automatic NLG metrics at the system level
Our analysis compares the adjusted error of metrics to humans and a derived, perfect segment-level annotator, both of which are unbiased estimators dependent on the number of judgments collected.
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform.
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation
We further conduct a novel human-in-the-loop evaluation to show that our models are considerably more robust to new human-written adversarial examples: crowdworkers can fool our model only 8. 8% of the time on average, compared to 17. 6% for a model trained without synthetic data.
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines.
Dynabench: Rethinking Benchmarking in NLP
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking.
Do Question Answering Modeling Improvements Hold Across Benchmarks?
Do question answering (QA) modeling improvements (e. g., choice of architecture and training procedure) hold consistently across the diverse landscape of QA benchmarks?
Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA
While research on explaining predictions of open-domain QA systems (ODQA) to users is gaining momentum, most works have failed to evaluate the extent to which explanations improve user trust.
To what extent do human explanations of model behavior align with actual model behavior?
Given the increasingly prominent role NLP models (will) play in our lives, it is important for human expectations of model behavior to align with actual model behavior.
With Little Power Comes Great Responsibility
Despite its importance to experimental design, statistical power (the probability that, given a real effect, an experiment will reject the null hypothesis) has largely been ignored by the NLP community.
On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks
Many pairwise classification tasks, such as paraphrase detection and open-domain question answering, naturally have extreme label imbalance (e. g., $99. 99\%$ of examples are negatives).
Selective Question Answering under Domain Shift
In this work, we propose the setting of selective question answering under domain shift, in which a QA model is tested on a mixture of in-domain and out-of-domain data, and must answer (i. e., not abstain on) as many questions as possible while maintaining high accuracy.
Robust Encodings: A Framework for Combating Adversarial Typos
We instantiate RobEn to defend against a large family of adversarial typos.
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems.
Certified Robustness to Adversarial Word Substitutions
We train the first models that are provably robust to all word substitutions in this family.
Document-Level N-ary Relation Extraction with Multiscale Representation Learning
Most information extraction methods focus on binary relations expressed within single sentences.
Document-Level $N$-ary Relation Extraction with Multiscale Representation Learning
Widening the system's purview to the entire document maximizes potential recall.
Know What You Don't Know: Unanswerable Questions for SQuAD
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.
Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer
We consider the task of text attribute transfer: transforming a sentence to alter a specific attribute (e. g., sentiment) while preserving its attribute-independent content (e. g., changing "screen is just the right size" to "screen is too small").
Ranked #1 on
Unsupervised Text Style Transfer
on Yelp2018
Adversarial Examples for Evaluating Reading Comprehension Systems
Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear.
Data Recombination for Neural Semantic Parsing
Modeling crisp logical regularities is crucial in semantic parsing, making it difficult for neural models with no task-specific prior knowledge to achieve good results.
