Search Results for author:
Found 25 papers, 18 papers with code
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
Even simple, known confounders such as preference for longer outputs remain in existing automated evaluation metrics.
Benchmarking Multi-Domain Active Learning on Image Classification
Active learning aims to enhance model performance by strategically labeling informative data points.
Proving Test Set Contamination in Black Box Language Models
In contrast, the tendency for language models to memorize example order means that a contaminated language model will find certain canonical orderings to be much more likely than others.
One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention
Then, we find that changing the distribution of the covariates and weight vector to a non-isotropic Gaussian distribution has a strong impact on the learned algorithm: the global minimizer of the pre-training loss now implements a single step of $\textit{pre-conditioned}$ GD.
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a +10% improvement in win-rate against Davinci003.
Benchmarking Large Language Models for News Summarization
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood.
Coder Reviewer Reranking for Code Generation
Sampling diverse programs from a code language model and reranking with model likelihood is a popular method for code generation but it is prone to preferring degenerate solutions.
Ranked #23 on
Code Generation
on MBPP
Data Feedback Loops: Model-driven Amplification of Dataset Biases
Datasets scraped from the internet have been critical to the successes of large-scale machine learning.
Diffusion-LM Improves Controllable Text Generation
Controlling the behavior of language models (LMs) without re-training is a major open problem in natural language generation.
TempLM: Distilling Language Models into Template-Based Generators
While pretrained language models (PLMs) have greatly improved text generation, they have also been known to produce unfaithful or inappropriate content.
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference
Meta-learning promises few-shot learners that can adapt to new distributions by repurposing knowledge acquired from previous training.
Distributionally Robust Neural Networks
Distributionally robust optimization (DRO) allows us to learn models that instead minimize the worst-case training loss over a set of pre-defined groups.
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
Distributionally robust optimization (DRO) allows us to learn models that instead minimize the worst-case training loss over a set of pre-defined groups.
Ranked #1 on
Out-of-Distribution Generalization
on UrbanCars
Learning Autocomplete Systems as a Communication Game
We study textual autocomplete---the task of predicting a full sentence from a partial sentence---as a human-machine communication game.
Distributionally Robust Language Modeling
Language models are generally trained on data spanning a wide range of topics (e. g., news, reviews, fiction), but they might be applied to an a priori unknown target distribution (e. g., restaurant reviews).
Unifying Human and Statistical Evaluation for Natural Language Generation
How can we measure whether a natural language generation system produces both high quality and diverse outputs?
A Retrieve-and-Edit Framework for Predicting Structured Outputs
For the task of generating complex outputs such as source code, editing existing outputs can be easier than generating complex outputs from scratch.
Fairness Without Demographics in Repeated Loss Minimization
Machine learning models (e. g., speech recognizers) are usually trained to minimize average loss, which results in representation disparity---minority groups (e. g., non-native speakers) contribute less to the training objective and thus tend to suffer higher loss.
Derivative free optimization via repeated classification
We develop an algorithm for minimizing a function using $n$ batched function value measurements at each of $T$ rounds by using classifiers to identify a function's sublevel set.
Unsupervised Transformation Learning via Convex Relaxations
Our goal is to extract meaningful transformations from raw images, such as varying the thickness of lines in handwriting or the lighting in a portrait.
Generating Sentences by Editing Prototypes
We propose a new generative model of sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence.
Word Embeddings as Metric Recovery in Semantic Spaces
Continuous word representations have been remarkably useful across NLP tasks but remain poorly understood.
From random walks to distances on unweighted graphs
Using these techniques we generalize results on the degeneracy of hitting times and analyze a metric based on the Laplace transformed hitting time (LTHT).
Word, graph and manifold embedding from Markov processes
Continuous vector representations of words and objects appear to carry surprisingly rich semantic content.
Metric recovery from directed unweighted graphs
We demonstrate empirically that the estimator performs well on simulated examples as well as on real-world co-purchasing graphs even with a small number of points and degree scaling as low as $\log(n)$.
