Search Results for author:
Found 14 papers, 6 papers with code
Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
Large language models (LLMs) have achieved widespread success on a variety of in-context few-shot tasks, but this success is typically evaluated via correctness rather than consistency.
Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models
To address this need, we introduce the Adversarial Nibbler challenge.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questions
These findings suggest that, in the case of answering reading comprehension questions, debate is not a helpful format.
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey
We present the results of the NLP Community Metasurvey.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions
We use long contexts -- humans familiar with the context write convincing explanations for pre-selected correct and incorrect answers, and we test if those explanations allow humans who have not read the full context to more accurately determine the correct answer.
QuALITY: Question Answering with Long Input Texts, Yes!
To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5, 000 tokens, much longer than typical current models can process.
BBQ: A Hand-Built Bias Benchmark for Question Answering
It is well documented that NLP models learn social biases, but little work has been done on how these biases manifest in model outputs for applied tasks like question answering (QA).
NOPE: A Corpus of Naturally-Occurring Presuppositions in English
Understanding language requires grasping not only the overtly stated content, but also making inferences about things that were left unsaid.
Does Putting a Linguist in the Loop Improve NLU Data Collection?
We take natural language inference as a test case and ask whether it is beneficial to put a linguist `in the loop' during data collection to dynamically identify and address gaps in the data by introducing novel constraints on the task.
BLiMP: The Benchmark of Linguistic Minimal Pairs for English
We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLiMP), a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English.
Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
We conclude that a variety of methods is necessary to reveal all relevant aspects of a model's grammatical knowledge in a given domain.
