Search Results for author:
Found 29 papers, 6 papers with code
Discovering Data Structures: Nearest Neighbor Search and Beyond
We also adapt our framework to the problem of estimating frequencies over a data stream, and believe it could also be a powerful discovery tool for new problems.
When is Multicalibration Post-Processing Necessary?
Calibration is a well-studied property of predictors which guarantees meaningful uncertainty estimates.
Pre-trained Large Language Models Use Fourier Features to Compute Addition
This paper shows that pre-trained LLMs add numbers using Fourier features -- dimensions in the hidden state that represent numbers via a set of features sparse in the frequency domain.
Optimal Multiclass U-Calibration Error and Beyond
We consider the problem of online multiclass U-calibration, where a forecaster aims to make sequential distributional predictions over $K$ classes with low U-calibration error, that is, low regret with respect to all bounded proper losses simultaneously.
Simplicity Bias of Transformers to Learn Low Sensitivity Functions
Transformers achieve state-of-the-art accuracy and robustness across many tasks, but an understanding of the inductive biases that they have and how those biases are different from other neural network architectures remains elusive.
Transductive Learning Is Compact
We demonstrate a compactness result holding broadly across supervised learning with a general class of loss functions: Any hypothesis class $H$ is learnable with transductive sample complexity $m$ precisely when all of its finite projections are learnable with sample complexity $m$.
Stability and Multigroup Fairness in Ranking with Uncertain Predictions
However, when predictors trained for classification tasks have intrinsic uncertainty, it is not obvious how this uncertainty should be represented in the derived rankings.
Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
Transformers excel at in-context learning (ICL) -- learning from demonstrations without parameter updates -- but how they do so remains a mystery.
Mitigating Simplicity Bias in Deep Learning for Improved OOD Generalization and Robustness
Neural networks (NNs) are known to exhibit simplicity bias where they tend to prefer learning 'simple' features over more 'complex' ones, even when the latter may be more informative.
Regularization and Optimal Multiclass Learning
We demonstrate that an agnostic version of the Hall complexity again characterizes error rates exactly, and exhibit an optimal learner using maximum entropy programs.
Fairness in Matching under Uncertainty
The prevalence and importance of algorithmic two-sided marketplaces has drawn attention to the issue of fairness in such settings.
Efficient Convex Optimization Requires Superlinear Memory
We show that any memory-constrained, first-order algorithm which minimizes $d$-dimensional, $1$-Lipschitz convex functions over the unit ball to $1/\mathrm{poly}(d)$ accuracy using at most $d^{1. 25 - \delta}$ bits of memory must make at least $\tilde{\Omega}(d^{1 + (4/3)\delta})$ first-order queries (for any constant $\delta \in [0, 1/4]$).
KL Divergence Estimation with Multi-group Attribution
Estimating the Kullback-Leibler (KL) divergence between two distributions given samples from them is well-studied in machine learning and information theory.
On the Statistical Complexity of Sample Amplification
The ``sample amplification'' problem formalizes the following question: Given $n$ i. i. d.
Big-Step-Little-Step: Efficient Gradient Methods for Objectives with Multiple Scales
We consider the problem of minimizing a function $f : \mathbb{R}^d \rightarrow \mathbb{R}$ which is implicitly decomposable as the sum of $m$ unknown non-interacting smooth, strongly convex functions and provide a method which solves this problem with a number of gradient evaluations that scales (up to logarithmic factors) as the product of the square-root of the condition numbers of the components.
Omnipredictors
We suggest a rigorous new paradigm for loss minimization in machine learning where the loss function can be ignored at the time of learning and only be taken into account when deciding an action.
One Network Fits All? Modular versus Monolithic Task Formulations in Neural Networks
Can deep learning solve multiple tasks simultaneously, even when they are unrelated and very different?
Multicalibrated Partitions for Importance Weights
We significantly strengthen previous work that use the MaxEntropy approach, that define the importance weights based on a distribution $Q$ closest to $P$, that looks the same as $R$ on every set $C \in \mathcal{C}$, where $\mathcal{C}$ may be a huge collection of sets.
PIDForest: Anomaly Detection via Partial Identification
We consider the problem of detecting anomalies in a large dataset.
Sample Amplification: Increasing Dataset Size even when Learning is Impossible
In the Gaussian case, we show that an $\left(n, n+\Theta(\frac{n}{\sqrt{d}} )\right)$ amplifier exists, even though learning the distribution to small constant total variation distance requires $\Theta(d)$ samples.
Memory-Sample Tradeoffs for Linear Regression with Small Error
We consider the problem of performing linear regression over a stream of $d$-dimensional examples, and show that any algorithm that uses a subquadratic amount of memory exhibits a slower rate of convergence than can be achieved without memory constraints.
A Spectral View of Adversarially Robust Features
This connection can be leveraged to provide both robust features, and a lower bound on the robustness of any function that has significant variance across the dataset.
Recovery Guarantees for Quadratic Tensors with Sparse Observations
We consider the tensor completion problem of predicting the missing entries of a tensor.
Efficient Anomaly Detection via Matrix Sketching
We consider the problem of finding anomalies in high-dimensional data using popular PCA based anomaly scores.
Learning Overcomplete HMMs
On the other hand, we show that learning is impossible given only a polynomial number of samples for HMMs with a small output alphabet and whose transition matrices are random regular graphs with large degree.
Sketching Linear Classifiers over Data Streams
We introduce a new sub-linear space sketch---the Weight-Median Sketch---for learning compressed linear classifiers over data streams while supporting the efficient recovery of large-magnitude weights in the model.
Compressed Factorization: Fast and Accurate Low-Rank Factorization of Compressively-Sensed Data
What learning algorithms can be run directly on compressively-sensed data?
Orthogonalized ALS: A Theoretically Principled Tensor Decomposition Algorithm for Practical Use
The popular Alternating Least Squares (ALS) algorithm for tensor decomposition is efficient and easy to implement, but often converges to poor local optima---particularly when the weights of the factors are non-uniform.
Prediction with a Short Memory
For a Hidden Markov Model with $n$ hidden states, $I$ is bounded by $\log n$, a quantity that does not depend on the mixing time, and we show that the trivial prediction algorithm based on the empirical frequencies of length $O(\log n/\epsilon)$ windows of observations achieves this error, provided the length of the sequence is $d^{\Omega(\log n/\epsilon)}$, where $d$ is the size of the observation alphabet.
