Search Results for author:
Found 100 papers, 42 papers with code
Online Control of the False Coverage Rate and False Sign Rate
Here, we consider the general problem of FCR control in the online setting, where there is an infinite sequence of fixed unknown parameters ordered by time.
Logarithmic Neyman Regret for Adaptive Estimation of the Average Treatment Effect
Estimation of the Average Treatment Effect (ATE) is a core problem in causal inference with strong connections to Off-Policy Evaluation in Reinforcement Learning.
Sharp Matrix Empirical Bernstein Inequalities
We present two sharp empirical Bernstein inequalities for symmetric random matrices with bounded eigenvalues.
Conformalized Interactive Imitation Learning: Handling Expert Shift and Intermittent Feedback
From the interactive IL side, we develop ConformalDAgger, a new approach wherein the robot uses prediction intervals calibrated by IQT as a reliable measure of deployment-time uncertainty to actively query for more expert feedback.
$β$-calibration of Language Model Confidence Scores for Generative QA
To use generative question-and-answering (QA) systems for decision-making and in any critical application, these systems need to provide well-calibrated confidence scores that reflect the correctness of their answers.
Sequential Kernelized Stein Discrepancy
We present a sequential version of the kernelized Stein discrepancy, which allows for conducting goodness-of-fit tests for unnormalized densities that are continuously monitored and adaptively stopped.
Anytime-Valid Inference for Double/Debiased Machine Learning of Causal Parameters
This can be of particular concern in large scale experimental studies with huge financial costs or human lives at stake, as well as in observational studies where the length of confidence of intervals do not shrink to zero even with increasing sample size due to partial identifiability of a structural parameter.
Conformal online model aggregation
Conformal prediction equips machine learning models with a reasonable notion of uncertainty quantification without making strong distributional assumptions.
Combining Evidence Across Filtrations Using Adjusters
We first establish that a class of functions called adjusters allows us to lift e-processes from a coarser filtration into any finer filtration.
Positive Semidefinite Supermartingales and Randomized Matrix Concentration Inequalities
We present new concentration inequalities for either martingale dependent or exchangeable random symmetric matrices under a variety of tail conditions, encompassing now-standard Chernoff bounds to self-normalized heavy-tailed settings.
Semiparametric Efficient Inference in Adaptive Experiments
This central limit theorem enables efficient inference at fixed sample sizes.
Time-Uniform Confidence Spheres for Means of Random Vectors
Our results include a dimension-free CSS for log-concave random vectors, a dimension-free CSS for sub-Gaussian random vectors, and CSSs for sub-$\psi$ random vectors (which includes sub-gamma, sub-Poisson, and sub-exponential distributions).
Online multiple testing with e-values
A scientist tests a continuous stream of hypotheses over time in the course of her investigation -- she does not test a predetermined, fixed number of hypotheses.
Deep anytime-valid hypothesis testing
We propose a general framework for constructing powerful, sequential hypothesis tests for a large class of nonparametric testing problems.
Anytime-valid t-tests and confidence sequences for Gaussian means with unknown variance
These are respectively obtained by swapping Lai's flat mixture for a Gaussian mixture, and swapping the right Haar mixture over $\sigma$ with the maximum likelihood estimate under the null, as done in universal inference.
On the near-optimality of betting confidence sets for bounded means
Constructing nonasymptotic confidence intervals (CIs) for the mean of a univariate distribution from independent and identically distributed (i. i. d.)
Reducing sequential change detection to sequential estimation
We consider the problem of sequential change detection, where the goal is to design a scheme for detecting any changes in a parameter or functional $\theta$ of the data stream distribution that has small detection delay, but guarantees control on the frequency of false alarms in the absence of changes.
Adaptive Privacy Composition for Accuracy-first Mechanisms
In many practical applications of differential privacy, practitioners seek to provide the best privacy guarantees subject to a target level of accuracy.
Differentially Private Conditional Independence Testing
We provide theoretical guarantees on the performance of our tests and validate them empirically.
Auditing Fairness by Betting
Whereas previous work relies on a fixed-sample size, our methods are sequential and allow for the continuous monitoring of incoming data, making them highly amenable to tracking the fairness of real-world systems.
Counterfactually Comparing Abstaining Classifiers
When evaluating black-box abstaining classifier(s), however, we lack a principled approach that accounts for what the classifier would have predicted on its abstentions.
Risk-limiting Financial Audits via Weighted Sampling without Replacement
Next, we develop methods to improve the quality of CSs by incorporating side information about the unknown values associated with each item.
Online Platt Scaling with Calibeating
We present an online post-hoc calibration method, called Online Platt Scaling (OPS), which combines the Platt scaling technique with online logistic regression.
The extended Ville's inequality for nonintegrable nonnegative supermartingales
Following the initial work by Robbins, we rigorously present an extended theory of nonnegative supermartingales, requiring neither integrability nor finiteness.
A unified recipe for deriving (time-uniform) PAC-Bayes bounds
We present a unified framework for deriving PAC-Bayesian generalization bounds.
Sequential change detection via backward confidence sequences
We present a simple reduction from sequential estimation to sequential changepoint detection (SCD).
Huber-Robust Confidence Sequences
Confidence sequences are confidence intervals that can be sequentially tracked, and are valid at arbitrary data-dependent stopping times.
A Permutation-Free Kernel Independence Test
In nonparametric independence testing, we observe i. i. d.\ data $\{(X_i, Y_i)\}_{i=1}^n$, where $X \in \mathcal{X}, Y \in \mathcal{Y}$ lie in any general spaces, and we wish to test the null that $X$ is independent of $Y$.
Sequential Kernelized Independence Testing
Independence testing is a classical statistical problem that has been extensively studied in the batch setting when one fixes the sample size before collecting data.
A Permutation-free Kernel Two-Sample Test
The usual kernel-MMD test statistic is a degenerate U-statistic under the null, and thus it has an intractable limiting distribution.
Anytime-valid off-policy inference for contextual bandits
Importantly, our methods can be employed while the original experiment is still running (that is, not necessarily post-hoc), when the logging policy may be itself changing (due to learning), and even if the context distributions are a highly dependent time-series (such as if they are drifting over time).
QuTE: decentralized multiple testing on sensor networks with false discovery rate control
We consider the setting where distinct agents reside on the nodes of an undirected graph, and each agent possesses p-values corresponding to one or more hypotheses local to its node.
Brownian Noise Reduction: Maximizing Privacy Subject to Accuracy Constraints
In this work, we generalize noise reduction to the setting of Gaussian noise, introducing the Brownian mechanism.
Faster online calibration without randomization: interval forecasts and the power of two choices
We study the problem of making calibrated probabilistic forecasts for a binary sequence generated by an adversarial nature.
Fully Adaptive Composition in Differential Privacy
However, these results require that the privacy parameters of all algorithms be fixed before interacting with the data.
E-detectors: a nonparametric framework for sequential change detection
Sequential change detection is a classical problem with a variety of applications.
Nonparametric extensions of randomized response for private confidence sets
This work derives methods for performing nonparametric, nonasymptotic statistical inference for population means under the constraint of local differential privacy (LDP).
Data fission: splitting a single data point
Rasines and Young (2022) offers an alternative approach that uses additive Gaussian noise -- this enables post-selection inference in finite samples for Gaussian distributed data and asymptotically when errors are non-Gaussian.
Best Arm Identification under Additive Transfer Bandits
We consider a variant of the best arm identification (BAI) problem in multi-armed bandits (MAB) in which there are two sets of arms (source and target), and the objective is to determine the best target arm while only pulling source arms.
Universal Inference Meets Random Projections: A Scalable Test for Log-concavity
Shape constraints yield flexible middle grounds between fully nonparametric and fully parametric approaches to modeling distributions of data.
Tracking the risk of a deployed model and detecting harmful distribution shifts
When deployed in the real world, machine learning models inevitably encounter changes in the data distribution, and certain -- but not all -- distribution shifts could result in significant performance degradation.
Comparing Sequential Forecasters
Consider two forecasters, each making a single prediction for a sequence of events over time.
Top-label calibration and multiclass-to-binary reductions
We propose top-label calibration as a rectification of confidence calibration.
A unified framework for bandit multiple testing
In bandit multiple hypothesis testing, each arm corresponds to a different null hypothesis that we wish to test, and the goal is to design adaptive algorithms that correctly identify large set of interesting arms (true discoveries), while only mistakenly identifying a few uninteresting ones (false discoveries).
Martingale Methods for Sequential Estimation of Convex Functionals and Divergences
We present a unified technique for sequential estimation of convex divergences between distributions, including integral probability metrics like the kernel maximum mean discrepancy, $\varphi$-divergences like the Kullback-Leibler divergence, and optimal transport costs, such as powers of Wasserstein distances.
Time-uniform central limit theory and asymptotic confidence sequences
This paper introduces time-uniform analogues of such asymptotic confidence intervals, adding to the literature on confidence sequences (CS) -- sequences of confidence intervals that are uniformly valid over time -- which provide valid inference at arbitrary stopping times and incur no penalties for "peeking" at the data, unlike classical confidence intervals which require the sample size to be fixed in advance.
Distribution-free uncertainty quantification for classification under label shift
Piggybacking on recent progress in addressing label shift (for better prediction), we examine the right way to achieve UQ by reweighting the aforementioned conformal and calibration procedures whenever some unlabeled data from the target distribution is available.
Large-scale simultaneous inference under dependence
Simultaneous inference allows for the exploration of data while deciding on criteria for proclaiming discoveries.
Statistics Theory Methodology Statistics Theory
Off-policy Confidence Sequences
We develop confidence bounds that hold uniformly over time for off-policy evaluation in the contextual bandit setting.
Dimension-agnostic inference using cross U-statistics
Classical asymptotic theory for statistical inference usually involves calibrating a statistic by fixing the dimension $d$ while letting the sample size $n$ increase to infinity.
Dynamic Algorithms for Online Multiple Testing
This statistical advance is enabled by the development of new algorithmic ideas: earlier algorithms are more "static" while our new ones allow for the dynamical adjustment of testing levels based on the amount of wealth the algorithm has accumulated.
Estimating means of bounded random variables by betting
This paper derives confidence intervals (CI) and time-uniform confidence sequences (CS) for the classical problem of estimating an unknown mean from bounded observations.
Distribution-free binary classification: prediction sets, confidence intervals and calibration
We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting, that is without making any distributional assumptions on the data.
The leave-one-covariate-out conditional randomization test
Conditional independence testing is an important problem, yet provably hard without assumptions.
Uncertainty quantification using martingales for misspecified Gaussian processes
There is a necessary cost to achieving robustness: if the prior was correct, posterior GP bands are narrower than our CS.
Confidence sequences for sampling without replacement
We then present Hoeffding- and empirical-Bernstein-type time-uniform CSs and fixed-time confidence intervals for sampling WoR, which improve on previous bounds in the literature and explicitly quantify the benefit of WoR sampling.
Fast and Powerful Conditional Randomization Testing via Distillation
We propose the distilled CRT, a novel approach to using state-of-the-art machine learning algorithms in the CRT while drastically reducing the number of times those algorithms need to be run, thereby taking advantage of their power and the CRT's statistical guarantees without suffering the usual computational expense.
Methodology
On the power of conditional independence testing under model-X
For testing conditional independence (CI) of a response Y and a predictor X given covariates Z, the recently introduced model-X (MX) framework has been the subject of active methodological research, especially in the context of MX knockoffs and their successful application to genome-wide association studies.
Familywise Error Rate Control by Interactive Unmasking
We propose a method for multiple hypothesis testing with familywise error rate (FWER) control, called the i-FWER test.
Methodology
On conditional versus marginal bias in multi-armed bandits
The bias of the sample means of the arms in multi-armed bandits is an important issue in adaptive data analysis that has recently received considerable attention in the literature.
Universal Inference
Constructing tests and confidence sets for such models is notoriously difficult.
The Power of Batching in Multiple Hypothesis Testing
One important partition of algorithms for controlling the false discovery rate (FDR) in multiple testing is into offline and online algorithms.
Online control of the familywise error rate
Biological research often involves testing a growing number of null hypotheses as new data is accumulated over time.
Path Length Bounds for Gradient Descent and Flow
We derive bounds on the path length $\zeta$ of gradient descent (GD) and gradient flow (GF) curves for various classes of smooth convex and nonconvex functions.
Sequential estimation of quantiles with applications to A/B-testing and best-arm identification
We propose confidence sequences -- sequences of confidence intervals which are valid uniformly over time -- for quantiles of any distribution over a complete, fully-ordered set, based on a stream of i. i. d.
Are sample means in multi-armed bandits positively or negatively biased?
It is well known that in stochastic multi-armed bandits (MAB), the sample mean of an arm is typically not an unbiased estimator of its true mean.
ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls
Major internet companies routinely perform tens of thousands of A/B tests each year.
Predictive inference with the jackknife+
This paper introduces the jackknife+, which is a novel method for constructing predictive confidence intervals.
Methodology
Conformal Prediction Under Covariate Shift
We extend conformal prediction methodology beyond the case of exchangeable data.
Methodology
A Higher-Order Kolmogorov-Smirnov Test
We present an extension of the Kolmogorov-Smirnov (KS) two-sample test, which can be more sensitive to differences in the tails.
The limits of distribution-free conditional predictive inference
We consider the problem of distribution-free predictive inference, with the goal of producing predictive coverage guarantees that hold conditionally rather than marginally.
Statistics Theory Statistics Theory
On the bias, risk and consistency of sample means in multi-armed bandits
For example, when is it consistent, how large is its bias, and can we bound its mean squared error?
Asynchronous Online Testing of Multiple Hypotheses
We consider the problem of asynchronous online testing, aimed at providing control of the false discovery rate (FDR) during a continual stream of data collection and testing, where each test may be a sequential test that can start and stop at arbitrary times.
Time-uniform, nonparametric, nonasymptotic confidence sequences
A confidence sequence is a sequence of confidence intervals that is uniformly valid over an unbounded time horizon.
Statistics Theory Probability Methodology Statistics Theory
Towards "simultaneous selective inference": post-hoc bounds on the false discovery proportion
In this paper, we show that the entire path of rejection sets considered by a variety of existing FDR procedures (like BH, knockoffs, and many others) can be endowed with simultaneous high-probability bounds on FDP.
Statistics Theory Statistics Theory
SAFFRON: an adaptive algorithm for online control of the false discovery rate
However, unlike older methods, SAFFRON's threshold sequence is based on a novel estimate of the alpha fraction that it allocates to true null hypotheses.
Online control of the false discovery rate with decaying memory
In the online multiple testing problem, p-values corresponding to different null hypotheses are observed one by one, and the decision of whether or not to reject the current hypothesis must be made immediately, after which the next p-value is observed.
DAGGER: A sequential algorithm for FDR control on DAGs
We propose a linear-time, single-pass, top-down algorithm for multiple testing on directed acyclic graphs (DAGs), where nodes represent hypotheses and edges specify a partial ordering in which hypotheses must be tested.
A framework for Multi-A(rmed)/B(andit) testing with online FDR control
We propose an alternative framework to existing setups for controlling false alarms when multiple A/B tests are run over time.
A unified treatment of multiple testing with prior knowledge using the p-filter
There is a significant literature on methods for incorporating knowledge into multiple testing procedures so as to improve their power and precision.
Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
In this context, the MMD may be used in two roles: first, as a discriminator, either directly on the samples, or on features of the samples.
Function-Specific Mixing Times and Concentration Away from Equilibrium
These results show that it is possible for empirical expectations of functions to concentrate long before the underlying chain has mixed in the classical sense, and we show that the concentration rates we achieve are optimal up to constants.
On kernel methods for covariates that are rankings
This paper studies the use of reproducing kernel Hilbert space methods for learning from permutation-valued features.
Asymptotic behavior of $\ell_p$-based Laplacian regularization in semi-supervised learning
Together, these properties show that $p = d+1$ is an optimal choice, yielding a function estimate $\hat{f}$ that is both smooth and non-degenerate, while remaining maximally sensitive to $P$.
Classification accuracy as a proxy for two sample testing
We prove two results that hold for all classifiers in any dimensions: if its true error remains $\epsilon$-better than chance for some $\epsilon>0$ as $d, n \to \infty$, then (a) the permutation-based test is consistent (has power approaching to one), (b) a computationally efficient test based on a Gaussian approximation of the null distribution is also consistent.
Minimax Lower Bounds for Linear Independence Testing
Linear independence testing is a fundamental information-theoretic and statistical problem that can be posed as follows: given $n$ points $\{(X_i, Y_i)\}^n_{i=1}$ from a $p+q$ dimensional multivariate distribution where $X_i \in \mathbb{R}^p$ and $Y_i \in\mathbb{R}^q$, determine whether $a^T X$ and $b^T Y$ are uncorrelated for every $a \in \mathbb{R}^p, b\in \mathbb{R}^q$ or not.
The p-filter: multi-layer FDR control for grouped hypotheses
In many practical applications of multiple hypothesis testing using the False Discovery Rate (FDR), the given hypotheses can be naturally partitioned into groups, and one may not only want to control the number of false discoveries (wrongly rejected null hypotheses), but also the number of falsely discovered groups of hypotheses (we say a group is falsely discovered if at least one hypothesis within that group is rejected, when in reality the group contains only nulls).
On Wasserstein Two Sample Testing and Related Families of Nonparametric Tests
In this work, our central object is the Wasserstein distance, as we form a chain of connections from univariate methods like the Kolmogorov-Smirnov test, PP/QQ plots and ROC/ODC curves, to multivariate tests involving energy statistics and kernel based maximum mean discrepancy.
Adaptivity and Computation-Statistics Tradeoffs for Kernel and Distance based High Dimensional Two Sample Testing
We formally characterize the power of popular tests for GDA like the Maximum Mean Discrepancy with the Gaussian kernel (gMMD) and bandwidth-dependent variants of the Energy Distance with the Euclidean norm (eED) in the high-dimensional MDA regime.
Fast Two-Sample Testing with Analytic Representations of Probability Measures
The new tests are consistent against a larger class of alternatives than the previous linear-time tests based on the (non-smoothed) empirical characteristic functions, while being much faster than the current state-of-the-art quadratic-time kernel-based or energy distance-based tests.
Sequential Nonparametric Testing with the Law of the Iterated Logarithm
It is novel in several ways: (a) it takes linear time and constant space to compute on the fly, (b) it has the same power guarantee as a non-sequential version of the test with the same computational constraints up to a small factor, and (c) it accesses only as many samples as are required - its stopping time adapts to the unknown difficulty of the problem.
An Analysis of Active Learning With Uniform Feature Noise
For larger $\sigma$, the \textit{unflattening} of the regression function on convolution with uniform noise, along with its local antisymmetry around the threshold, together yield a behaviour where noise \textit{appears} to be beneficial.
Margins, Kernels and Non-linear Smoothed Perceptrons
This allows us to give guarantees for a primal-dual algorithm that halts in $\min\{\tfrac{\sqrt n}{|\rho|}, \tfrac{\sqrt n}{\epsilon}\}$ iterations with a perfect separator in the RKHS if the primal is feasible or a dual $\epsilon$-certificate of near-infeasibility.
Algorithmic Connections Between Active Learning and Stochastic Convex Optimization
Combining these two parts yields an algorithm that solves stochastic convex optimization of uniformly convex and smooth functions using only noisy gradient signs by repeatedly performing active learning, achieves optimal rates and is adaptive to all unknown convexity and smoothness parameters.
On the High-dimensional Power of Linear-time Kernel Two-Sample Testing under Mean-difference Alternatives
The current literature is split into two kinds of tests - those which are consistent without any assumptions about how the distributions may differ (\textit{general} alternatives), and those which are designed to specifically test easier alternatives, like a difference in means (\textit{mean-shift} alternatives).
Rows vs Columns for Linear Systems of Equations - Randomized Kaczmarz or Coordinate Descent?
This paper is about randomized iterative algorithms for solving a linear system of equations $X \beta = y$ in different settings.
Towards A Deeper Geometric, Analytic and Algorithmic Understanding of Margins
Given a matrix $A$, a linear feasibility problem (of which linear classification is a special case) aims to find a solution to a primal problem $w: A^Tw > \textbf{0}$ or a certificate for the dual problem which is a probability distribution $p: Ap = \textbf{0}$.
Fast and Flexible ADMM Algorithms for Trend Filtering
This paper presents a fast and robust algorithm for trend filtering, a recently developed nonparametric regression tool.
On the Decreasing Power of Kernel and Distance based Nonparametric Hypothesis Tests in High Dimensions
This paper is about two related decision theoretic problems, nonparametric two-sample testing and independence testing.
Nonparametric Independence Testing for Small Sample Sizes
This paper deals with the problem of nonparametric independence testing, a fundamental decision-theoretic problem that asks if two arbitrary (possibly multivariate) random variables $X, Y$ are independent or not, a question that comes up in many fields like causality and neuroscience.
