Search Results for author:
Found 24 papers, 16 papers with code
CapWAP: Image Captioning with a Purpose
In this task, we use question-answer (QA) pairs{---}a natural expression of information need{---}from users, instead of reference captions, for both training and post-inference evaluation.
Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
However, even pretrain reward ensembles do not eliminate reward hacking: we show several qualitative reward hacking phenomena that are not mitigated by ensembling because all reward models in the ensemble exhibit similar error patterns.
Risk-Controlling Model Selection via Guided Bayesian Optimization
Adjustable hyperparameters of machine learning models typically impact various key trade-offs such as accuracy, fairness, robustness, or inference cost.
Towards Robust and Efficient Continual Language Learning
As the application space of language models continues to evolve, a natural question to ask is how we can quickly adapt models to new tasks.
Conformal Language Modeling
Translating this process to conformal prediction, we calibrate a stopping rule for sampling different outputs from the LM that get added to a growing set of candidates until we are confident that the output set is sufficient.
Efficiently Controlling Multiple Risks with Pareto Testing
Machine learning applications frequently come with multiple diverse objectives and constraints that can change over time.
Calibrated Selective Classification
Providing calibrated uncertainty estimates alongside predictions -- probabilities that correspond to true frequencies -- can be as important as having predictions that are simply accurate on average.
Conformal Risk Control
We extend conformal prediction to control the expected value of any monotone loss function.
Confident Adaptive Language Modeling
Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Conformal Prediction Sets with Limited False Positives
We propose to trade coverage for a notion of precision by enforcing that the presence of incorrect candidates in the predicted conformal sets (i. e., the total number of false positives) is bounded according to a user-specified tolerance.
Trading Coverage for Precision: Conformal Prediction with Limited False Discoveries
In this paper, we develop a new approach to conformal prediction in which we aim to output a precise set of promising prediction candidates that is guaranteed to contain a limited number of incorrect answers.
Consistent Accelerated Inference via Confident Adaptive Transformers
In this work, we present CATs -- Confident Adaptive Transformers -- in which we simultaneously increase computational efficiency, while guaranteeing a specifiable degree of consistency with the original model with high confidence.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Typical fact verification models use retrieved written evidence to verify claims.
Few-shot Conformal Prediction with Auxiliary Tasks
We develop a novel approach to conformal prediction when the target task has limited data available for training.
Making Pre-trained Language Models Better Few-shot Learners
We present LM-BFF--better few-shot fine-tuning of language models--a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples.
Ranked #6 on
Zero-Shot Text Classification
on AG News
CapWAP: Captioning with a Purpose
In this task, we use question-answer (QA) pairs---a natural expression of information need---from users, instead of reference captions, for both training and post-inference evaluation.
Efficient Conformal Prediction via Cascaded Inference with Expanded Admission
This set is guaranteed to contain a correct answer with high probability, and is well-suited for many open-ended classification tasks.
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems.
Working Hard or Hardly Working: Challenges of Integrating Typology into Neural Dependency Parsers
This paper explores the task of leveraging typology in the context of cross-lingual dependency parsing.
StarSpace: Embed All The Things!
A framework for training and evaluating AI models on a variety of openly available dialogue datasets.
ParlAI: A Dialog Research Software Platform
We introduce ParlAI (pronounced "par-lay"), an open-source software platform for dialog research implemented in Python, available at http://parl. ai.
Reading Wikipedia to Answer Open-Domain Questions
This paper proposes to tackle open- domain question answering using Wikipedia as the unique knowledge source: the answer to any factoid question is a text span in a Wikipedia article.
Ranked #1 on
Open-Domain Question Answering
on SQuAD1.1
Key-Value Memory Networks for Directly Reading Documents
Directly reading documents and being able to answer questions from them is an unsolved challenge.
Ranked #12 on
Question Answering
on WikiQA
