Search Results for author:
Found 33 papers, 10 papers with code
On the Relationship Between Binary Classification, Bipartite Ranking, and Binary Class Probability Estimation
It is known that a good binary CPE model can be used to obtain a good binary classification model (by thresholding at 0. 5), and also to obtain a good bipartite ranking model (by using the CPE model directly as a ranking model); it is also known that a binary classification model does not necessarily yield a CPE model.
Online and Stochastic Gradient Methods for Non-decomposable Loss Functions
In this work we initiate a study of online learning techniques for such non-decomposable loss functions with an aim to enable incremental learning as well as design scalable solvers for batch problems.
On the Statistical Consistency of Plug-in Classifiers for Non-decomposable Performance Measures
In this work, we consider plug-in algorithms that learn a classifier by applying an empirically determined threshold to a suitable `estimate' of the class probability, and provide a general methodology to show consistency of these methods for any non-decomposable measure that can be expressed as a continuous function of true positive rate (TPR) and true negative rate (TNR), and for which the Bayes optimal classifier is the class probability function thresholded suitably.
Consistent Classification Algorithms for Multi-class Non-Decomposable Performance Metrics
In this paper, we provide a unified framework for analysing a multi-class non-decomposable performance metric, where the problem of finding the optimal classifier for the performance metric is viewed as an optimization problem over the space of all confusion matrices achievable under the given distribution.
Surrogate Functions for Maximizing Precision at the Top
At the heart of our results is a family of truly upper bounding surrogates for prec@k. These surrogates are motivated in a principled manner and enjoy attractive properties such as consistency to prec@k under various natural margin/noise conditions.
Optimizing Non-decomposable Performance Measures: A Tale of Two Classes
Modern classification problems frequently present mild to severe label imbalance as well as specific requirements on classification characteristics, and require optimizing performance measures that are non-decomposable over the dataset, such as F-measure.
Learnability of Influence in Networks
We establish PAC learnability of influence functions for three common influence models, namely, the Linear Threshold (LT), Independent Cascade (IC) and Voter models, and present concrete sample complexity results in each case.
Online Optimization Methods for the Quantification Problem
The estimation of class prevalence, i. e., the fraction of a population that belongs to a certain class, is a very useful tool in data analytics and learning, and finds applications in many domains such as sentiment analysis, epidemiology, etc.
Support Vector Algorithms for Optimizing the Partial Area Under the ROC Curve
Increasingly, however, in several applications, ranging from ranking to biometric screening to medicine, performance is measured not in terms of the full area under the ROC curve, but in terms of the \emph{partial} area under the ROC curve between two false positive rates.
Optimal Auctions through Deep Learning: Advances in Differentiable Economics
Designing an incentive compatible auction that maximizes expected revenue is an intricate task.
Metric-Optimized Example Weights
Real-world machine learning applications often have complex test metrics, and may have training and test data that are not identically distributed.
Pairwise Fairness for Ranking and Regression
We present pairwise fairness metrics for ranking models and regression models that form analogues of statistical fairness notions such as equal opportunity, equal accuracy, and statistical parity.
Optimizing Generalized Rate Metrics through Game Equilibrium
We present a general framework for solving a large class of learning problems with non-linear functions of classification rates.
Optimizing Generalized Rate Metrics with Three Players
We present a general framework for solving a large class of learning problems with non-linear functions of classification rates.
On Making Stochastic Classifiers Deterministic
Stochastic classifiers arise in a number of machine learning problems, and have become especially prominent of late, as they often result from constrained optimization problems, e. g. for fairness, churn, or custom losses.
Optimizing Black-box Metrics with Adaptive Surrogates
We address the problem of training models with black-box and hard-to-optimize metrics by expressing the metric as a monotonic function of a small number of easy-to-optimize surrogates.
Robust Optimization for Fairness with Noisy Protected Groups
Second, we introduce two new approaches using robust optimization that, unlike the naive approach of only relying on $\hat{G}$, are guaranteed to satisfy fairness criteria on the true protected groups G while minimizing a training objective.
Quadratic Metric Elicitation for Fairness and Beyond
Metric elicitation is a recent framework for eliciting classification performance metrics that best reflect implicit user preferences based on the task and context.
Consistent Plug-in Classifiers for Complex Objectives and Constraints
We present a statistically consistent algorithm for constrained classification problems where the objective (e. g. F-measure, G-mean) and the constraints (e. g. demographic parity, coverage) are defined by general functions of the confusion matrix.
Approximate Heavily-Constrained Learning with Lagrange Multiplier Models
In machine learning applications such as ranking fairness or fairness over intersectional groups, one often encounters optimization problems with an extremely large number of constraints.
Distilling Double Descent
Distillation is the technique of training a "student" model based on examples that are labeled by a separate "teacher" model, which itself is trained on a labeled dataset.
Optimizing Black-box Metrics with Iterative Example Weighting
We consider learning to optimize a classification metric defined by a black-box function of the confusion matrix.
Churn Reduction via Distillation
In real-world systems, models are frequently updated as more data becomes available, and in addition to achieving high accuracy, the goal is to also maintain a low difference in predictions compared to the base model (i. e. predictive "churn").
Training Over-parameterized Models with Non-decomposable Objectives
Many modern machine learning applications come with complex and nuanced design goals such as minimizing the worst-case error, satisfying a given precision or recall target, or enforcing group-fairness constraints.
Implicit Rate-Constrained Optimization of Non-decomposable Objectives
We consider a popular family of constrained optimization problems arising in machine learning that involve optimizing a non-decomposable evaluation metric with a certain thresholded form, while constraining another metric of interest.
Robust Distillation for Worst-class Performance
We show empirically that our robust distillation techniques not only achieve better worst-class performance, but also lead to Pareto improvement in the tradeoff between overall performance and worst-class performance compared to other baseline methods.
Consistent Multiclass Algorithms for Complex Metrics and Constraints
We present consistent algorithms for multiclass learning with complex performance metrics and constraints, where the objective and constraints are defined by arbitrary functions of the confusion matrix.
Plugin estimators for selective classification with out-of-distribution detection
Recent work on selective classification with OOD detection (SCOD) has argued for the unified study of these problems; however, the formal underpinnings of this problem are still nascent, and existing techniques are heuristic in nature.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
When Does Confidence-Based Cascade Deferral Suffice?
Cascades are a classical strategy to enable inference cost to vary adaptively across samples, wherein a sequence of classifiers are invoked in turn.
Distributionally Robust Post-hoc Classifiers under Prior Shifts
We investigate the problem of training models that are robust to shifts caused by changes in the distribution of class-priors or group-priors.
Metric-aware LLM inference for regression and scoring
Large language models (LLMs) have demonstrated strong results on a range of NLP tasks.
Language Model Cascades: Token-level uncertainty and beyond
While the principles underpinning cascading are well-studied for classification tasks - with deferral based on predicted class uncertainty favored theoretically and practically - a similar understanding is lacking for generative LM tasks.
