Search Results for author:
Found 37 papers, 13 papers with code
Gradient Equilibrium in Online Learning: Theory and Applications
We present a new perspective on online learning that we refer to as gradient equilibrium: a sequence of iterates achieves gradient equilibrium if the average of gradients of losses along the sequence converges to zero.
Revisiting Optimism and Model Complexity in the Wake of Overparameterized Machine Learning
Common practice in modern machine learning involves fitting a large number of parameters relative to the number of observations.
Optimal Ridge Regularization for Out-of-Distribution Prediction
We study the behavior of optimal ridge regularization and optimal ridge risk for out-of-distribution prediction, where the test distribution deviates arbitrarily from the train distribution.
Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent
We analyze the statistical properties of generalized cross-validation (GCV) and leave-one-out cross-validation (LOOCV) applied to early-stopped gradient descent (GD) in high-dimensional least squares regression.
Integral Probability Metrics Meet Neural Networks: The Radon-Kolmogorov-Smirnov Test
Integral probability metrics (IPMs) constitute a general class of nonparametric two-sample tests that are based on maximizing the mean difference between samples from one distribution $P$ versus another $Q$, over all choices of data transformations $f$ living in some function space $\mathcal{F}$.
Conformal PID Control for Time Series Prediction
We study the problem of uncertainty quantification for time series prediction, with the goal of providing easy-to-use algorithms with formal guarantees.
Class-Conditional Conformal Prediction with Many Classes
Standard conformal prediction methods provide a marginal coverage guarantee, which means that for a random test point, the conformal prediction set contains the true label with a user-specified probability.
The Voronoigram: Minimax Estimation of Bounded Variation Functions From Scattered Data
We study an estimator that forms the Voronoi diagram of the design points, and then solves an optimization problem that regularizes according to a certain discrete notion of total variation (TV): the sum of weighted absolute differences of parameters $\theta_i,\theta_j$ (which estimate the function values $f_0(x_i), f_0(x_j)$) at all neighboring cells $i, j$ in the Voronoi diagram.
Multivariate Trend Filtering for Lattice Data
We study a multivariate version of trend filtering, called Kronecker trend filtering or KTF, for the case in which the design points form a lattice in $d$ dimensions.
Recalibrating probabilistic forecasts of epidemics
We apply this recalibration method to the 27 influenza forecasters in the FluSight Network and show that recalibration reliably improves forecast accuracy and calibration.
Minimax Optimal Regression over Sobolev Spaces via Laplacian Eigenmaps on Neighborhood Graphs
We also show that PCR-LE is \emph{manifold adaptive}: that is, we consider the situation where the design is supported on a manifold of small intrinsic dimension $m$, and give upper bounds establishing that PCR-LE achieves the faster minimax estimation ($n^{-2s/(2s + m)}$) and testing ($n^{-4s/(4s + m)}$) rates of convergence.
Minimax Optimal Regression over Sobolev Spaces via Laplacian Regularization on Neighborhood Graphs
In this paper we study the statistical properties of Laplacian smoothing, a graph-based approach to nonparametric regression.
Flexible Model Aggregation for Quantile Regression
Quantile regression is a fundamental problem in statistical learning motivated by a need to quantify uncertainty in predictions, or to model a diverse population without being overly reductive.
The Implicit Regularization of Stochastic Gradient Flow for Least Squares
We study the implicit regularization of mini-batch stochastic gradient descent, when applied to the fundamental problem of least squares regression.
Divided Differences, Falling Factorials, and Discrete Splines: Another Look at Trend Filtering and Related Problems
This paper reviews a class of univariate piecewise polynomial functions known as discrete splines, which share properties analogous to the better-known class of spline functions, but where continuity in derivatives is replaced by (a suitable notion of) continuity in divided differences.
Statistics Theory Numerical Analysis Numerical Analysis Methodology Statistics Theory
Modelling High-Dimensional Categorical Data Using Nonconvex Fusion Penalties
We provide an algorithm for exact and efficient computation of the global minimum of the resulting nonconvex objective in the case with a single variable with potentially many levels, and use this within a block coordinate descent procedure in the multivariate case.
Predictive inference with the jackknife+
This paper introduces the jackknife+, which is a novel method for constructing predictive confidence intervals.
Methodology
Conformal Prediction Under Covariate Shift
We extend conformal prediction methodology beyond the case of exchangeable data.
Methodology
A Higher-Order Kolmogorov-Smirnov Test
We present an extension of the Kolmogorov-Smirnov (KS) two-sample test, which can be more sensitive to differences in the tails.
Surprises in High-Dimensional Ridgeless Least Squares Interpolation
Interpolators -- estimators that achieve zero training error -- have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type.
The limits of distribution-free conditional predictive inference
We consider the problem of distribution-free predictive inference, with the goal of producing predictive coverage guarantees that hold conditionally rather than marginally.
Statistics Theory Statistics Theory
A Continuous-Time View of Early Stopping for Least Squares
Our primary focus is to compare the risk of gradient flow to that of ridge regression.
Higher-Order Total Variation Classes on Grids: Minimax Theory and Trend Filtering Methods
To move past this, we define two new higher-order TV classes, based on two ways of compiling the discrete derivatives of a parameter across the nodes.
A Sharp Error Analysis for the Fused Lasso, with Application to Approximate Changepoint Screening
In the 1-dimensional multiple changepoint detection problem, we derive a new fast error rate for the fused lasso estimator, under the assumption that the mean vector has a sparse number of changepoints.
Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso
In exciting new work, Bertsimas et al. (2016) showed that the classical best subset selection problem in regression modeling can be formulated as a mixed integer optimization (MIO) problem.
Methodology Computation
Additive Models with Trend Filtering
We study additive models built with trend filtering, i. e., additive models whose components are each regularized by the (discrete) total variation of their $k$th (discrete) derivative, for a chosen integer $k \geq 0$.
Exact Post-Selection Inference for Changepoint Detection and Other Generalized Lasso Problems
Leveraging a sequential characterization of this path from Tibshirani & Taylor (2011), and recent advances in post-selection inference from Lee et al. (2016), Tibshirani et al. (2016), we develop exact hypothesis tests and confidence intervals for linear contrasts of the underlying mean vector, conditioned on any model selection event along the generalized lasso path (assuming Gaussian errors in the observations).
Methodology
Distribution-Free Predictive Inference For Regression
In the spirit of reproducibility, all of our empirical results can also be easily (re)generated using this package.
High-Dimensional Longitudinal Classification with the Multinomial Fused Lasso
We study regularized estimation in high-dimensional longitudinal classification problems, using the lasso and fused lasso regularizers.
Nonparametric modal regression
Modal regression estimates the local modes of the distribution of $Y$ given $X=x$, instead of the mean, as in the usual regression sense, and can hence reveal important structure missed by usual regression methods.
Trend Filtering on Graphs
We introduce a family of adaptive estimators on graphs, based on penalizing the $\ell_1$ norm of discrete graph differences.
A General Framework for Fast Stagewise Algorithms
Forward stagewise regression follows a very simple strategy for constructing a sequence of sparse regression estimates: it starts with all coefficients equal to zero, and iteratively updates the coefficient (by a small amount $\epsilon$) of the variable that achieves the maximal absolute inner product with the current residual.
Fast and Flexible ADMM Algorithms for Trend Filtering
This paper presents a fast and robust algorithm for trend filtering, a recently developed nonparametric regression tool.
The Falling Factorial Basis and Its Statistical Applications
We study a novel spline-like basis, which we name the "falling factorial basis", bearing many similarities to the classic truncated power basis.
Exact Post-Selection Inference for Sequential Regression Procedures
We propose new inference tools for forward stepwise regression, least angle regression, and the lasso.
Methodology 62F03, 62G15
Adaptive piecewise polynomial estimation via trend filtering
We also provide theoretical support for these empirical findings; most notably, we prove that (with the right choice of tuning parameter) the trend filtering estimate converges to the true underlying function at the minimax rate for functions whose $k$th derivative is of bounded variation.
A significance test for the lasso
We propose a simple test statistic based on lasso fitted values, called the covariance test statistic, and show that when the true model is linear, this statistic has an $\operatorname {Exp}(1)$ asymptotic distribution under the null hypothesis (the null being that all truly active variables are contained in the current lasso model).
Statistics Theory Methodology Statistics Theory
