Search Results for author:
Found 69 papers, 23 papers with code
Lawma: The Power of Specialization for Legal Tasks
Annotation and classification of legal text are central components of empirical legal research.
Evaluating language models as risk scores
In this work, we focus on the use of language models as risk scores for unrealizable prediction tasks.
Training on the Test Task Confounds Evaluation and Emergence
This also applies to reported instances of emergent behavior that cannot be explained by the choice of evaluation metric.
Limits to Predicting Online Speech Using Large Language Models
Across three large language models ranging in size from 1 billion to 70 billion parameters, we find that predicting a user's posts from their peers' posts performs poorly.
Allocation Requires Prediction Only if Inequality Is Low
Algorithmic predictions are emerging as a promising solution concept for efficiently allocating societal resources.
An engine not a camera: Measuring performative power of online search
The power of digital platforms is at the center of major ongoing policy and regulatory efforts.
Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks
Sensitivity quantifies the impact that irrelevant changes to tasks have on a benchmark.
ImageNot: A contrast with ImageNet preserves model rankings
We introduce ImageNot, a dataset designed to match the scale of ImageNet while differing drastically in other aspects.
Predictors from causal features do not generalize better to new domains
Our goal is to test the hypothesis that models trained on causal features generalize better across domains.
Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget
We study how to best spend a budget of noisy labels to compare the accuracy of two binary classifiers.
Performative Prediction: Past and Future
A consequence of performative prediction is a natural equilibrium notion that gives rise to new optimization challenges.
What Makes ImageNet Look Unlike LAION
ImageNet was famously created from Flickr image search results.
Questioning the Survey Responses of Large Language Models
As large language models increase in capability, researchers have started to conduct surveys of all kinds on these models in order to investigate the population represented by their responses.
Unprocessing Seven Years of Algorithmic Fairness
Unprocessing allows for a direct comparison of methods using different underlying models and levels of relaxation.
Test-Time Training on Nearest Neighbors for Large Language Models
Surprisingly, retrieving and training on as few as 20 neighbors, each for only one gradient iteration, drastically improves performance across more than 20 language modeling tasks in the Pile.
Ranked #6 on
Language Modelling
on The Pile
Difficult Lessons on Social Prediction from Wisconsin Public Schools
These systems assist in targeting interventions to individual students by predicting which students are at risk of dropping out.
Causal Inference out of Control: Estimating the Steerability of Consumption
Regulators and academics are increasingly interested in the causal effect that algorithmic actions of a digital platform have on consumption.
Algorithmic Collective Action in Machine Learning
We initiate a principled study of algorithmic collective action on digital platforms that deploy machine learning algorithms.
A Theory of Dynamic Benchmarks
Dynamic benchmarks interweave model fitting and data collection in an attempt to mitigate the limitations of static benchmarks.
Is your model predicting the past?
When does a machine learning model predict the future of individuals and when does it recite patterns that predate the individuals?
Adversarial Scrutiny of Evidentiary Statistical Software
Responding to this need to adversarially scrutinize output from such software, we propose robust adversarial testing as an audit framework to examine the validity of evidentiary statistical software.
Performative Power
We introduce the notion of performative power, which measures the ability of a firm operating an algorithmic system, such as a digital content recommendation platform, to cause change in a population of participants.
Retiring Adult: New Datasets for Fair Machine Learning
Our primary contribution is a suite of new datasets derived from US Census surveys that extend the existing data ecosystem for research on fair machine learning.
Causal Inference Struggles with Agency on Online Platforms
Our results suggest that observational studies derived from user self-selection are a poor alternative to randomized experimentation on online platforms.
Alternative Microfoundations for Strategic Classification
When reasoning about strategic behavior in a machine learning context it is tempting to combine standard microfoundations of rational agents with the statistical decision theory underlying classification.
Patterns, predictions, and actions: A story about machine learning
This graduate textbook on machine learning tells a story of how patterns in data support predictions and consequential actions.
Revisiting Design Choices in Proximal Policy Optimization
We explain why standard design choices are problematic in these cases, and show that alternative choices of surrogate objectives and policy parameterizations can prevent the failure modes.
From Optimizing Engagement to Measuring Value
Most recommendation engines today are based on predicting user engagement, e. g. predicting whether a user will click on an item or not.
Stochastic Optimization for Performative Prediction
In performative prediction, the choice of a model influences the distribution of future data, typically through actions taken based on the model's predictions.
Balancing Competing Objectives with Noisy Data: Score-Based Classifiers for Welfare-Aware Machine Learning
Our theoretical results characterize the optimal strategies in this class, bound the Pareto errors due to inaccuracies in the scores, and show an equivalence between optimal strategies and a rich class of fairness-constrained profit-maximizing policies.
Performative Prediction
When predictions support decisions they may influence the outcome they aim to predict.
A Meta-Analysis of Overfitting in Machine Learning
By systematically comparing the public ranking with the final ranking, we assess how much participants adapted to the holdout set over the course of a competition.
Strategic Classification is Causal Modeling in Disguise
Moreover, we show a similar result holds for designing cost functions that satisfy the requirements of previous work.
Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
In this paper, we propose Test-Time Training, a general approach for improving the performance of predictive models when training and test data come from different distributions.
Ranked #34 on
Language Modelling
on LAMBADA
Building change detection for remote sensing images
CARLA MAP Leaderboard
+6
Test-Time Training for Out-of-Distribution Generalization
We introduce a general approach, called test-time training, for improving the performance of predictive models when test and training data come from different distributions.
Linear Dynamics: Clustering without identification
Linear dynamical systems are a fundamental and powerful parametric model class.
Explaining an increase in predicted risk for clinical alerts
We consider explanations in a temporal setting where a stateful dynamical model produces a sequence of risk estimates given an input at each time step.
Model Similarity Mitigates Test Set Overuse
Excessive reuse of test data has become commonplace in today's machine learning workflows.
The advantages of multiple classes for reducing overfitting from test set reuse
We show a new upper bound of $\tilde O(\max\{\sqrt{k\log(n)/(mn)}, k/n\})$ on the worst-case bias that any attack can achieve in a prediction problem with $m$ classes.
Identity Crisis: Memorization and Generalization under Extreme Overparameterization
We study the interplay between memorization and generalization of overparameterized networks in the extreme case of a single training example and an identity-mapping task.
Natural Analysts in Adaptive Data Analysis
The source of these pessimistic bounds is a model that permits arbitrary, possibly adversarial analysts that optimally use information to bias results.
A System for Massively Parallel Hyperparameter Tuning
Modern learning models are characterized by large hyperparameter spaces and long training times.
Sanity Checks for Saliency Maps
We find that reliance, solely, on visual assessment can be misleading.
The implicit fairness criterion of unconstrained learning
We show that under reasonable conditions, the deviation from satisfying group calibration is upper bounded by the excess risk of the learned score relative to the Bayes optimal score function.
The Social Cost of Strategic Classification
Consequential decision-making typically incentivizes individuals to behave strategically, tailoring their behavior to the specifics of the decision rule.
Model Reconstruction from Model Explanations
We show through theory and experiment that gradient-based explanations of a model quickly reveal the model itself.
Stable Recurrent Models
Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks.
Delayed Impact of Fair Machine Learning
Fairness in machine learning has predominantly been studied in static classification settings without concern for how decisions change the underlying population over time.
Massively Parallel Hyperparameter Tuning
Modern machine learning models are characterized by large hyperparameter search spaces and prohibitively expensive training costs.
Avoiding Discrimination through Causal Reasoning
Going beyond observational criteria, we frame the problem of discrimination based on protected attributes in the language of causal reasoning.
Climbing a shaky ladder: Better adaptive risk estimation
We revisit the \emph{leaderboard problem} introduced by Blum and Hardt (2015) in an effort to reduce overfitting in machine learning benchmarks.
Identity Matters in Deep Learning
An emerging design principle in deep learning is that each layer of a deep artificial neural network should be able to easily express the identity transformation.
Understanding deep learning requires rethinking generalization
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Equality of Opportunity in Supervised Learning
We propose a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features.
Gradient Descent Learns Linear Dynamical Systems
We prove that stochastic gradient descent efficiently converges to the global optimizer of the maximum likelihood objective of an unknown linear time-invariant dynamical system from a sequence of noisy observations generated by the system.
Differentially Private Learning of Structured Discrete Distributions
We investigate the problem of learning an unknown probability distribution over a discrete population from random samples.
Train faster, generalize better: Stability of stochastic gradient descent
In the non-convex case, we give a new interpretation of common practices in neural networks, and formally show that popular techniques for training large deep models are indeed stability-promoting.
Strategic Classification
Jury designs a classifier, and Contestant receives an input to the classifier, which he may change at some cost.
Generalization in Adaptive Data Analysis and Holdout Reuse
We also formalize and address the general problem of data reuse in adaptive data analysis.
The Ladder: A Reliable Leaderboard for Machine Learning Competitions
In this work, we introduce a notion of "leaderboard accuracy" tailored to the format of a competition.
Preserving Statistical Validity in Adaptive Data Analysis
We show that, surprisingly, there is a way to estimate an exponential in $n$ number of expectations accurately even if the functions are chosen adaptively.
Preventing False Discovery in Interactive Data Analysis is Hard
In particular, our result suggests that the perceived difficulty of preventing false discovery in today's collaborative research environment may be inherent.
Fast matrix completion without the condition number
We give the first algorithm for Matrix Completion whose running time and sample complexity is polynomial in the rank of the unknown target matrix, linear in the dimension of the matrix, and logarithmic in the condition number of the matrix.
Tight bounds for learning a mixture of two gaussians
Denoting by $\sigma^2$ the variance of the unknown mixture, we prove that $\Theta(\sigma^{12})$ samples are necessary and sufficient to estimate each parameter up to constant additive error when $d=1.$ Our upper bound extends to arbitrary dimension $d>1$ up to a (provably necessary) logarithmic loss in $d$ using a novel---yet simple---dimensionality reduction technique.
Computational Limits for Matrix Completion
Matrix Completion is the problem of recovering an unknown real-valued low-rank matrix from a subsample of its entries.
Understanding Alternating Minimization for Matrix Completion
In addition, we contribute a new technique for controlling the coherence of intermediate solutions arising in iterative algorithms based on a smoothed analysis of the QR factorization.
The Noisy Power Method: A Meta Algorithm with Applications
The noisy power method can be seen as a meta-algorithm that has recently found a number of important applications in a broad range of machine learning problems including alternating minimization for matrix completion, streaming principal component analysis (PCA), and privacy-preserving spectral analysis.
A Simple and Practical Algorithm for Differentially Private Data Release
We present a new algorithm for differentially private data release, based on a simple combination of the Exponential Mechanism with the Multiplicative Weights update rule.
Algorithms and Hardness for Robust Subspace Recovery
We give an algorithm that finds $T$ when it contains more than a $\frac{d}{n}$ fraction of the points.
