Search Results for author:
Found 50 papers, 22 papers with code
ECLIPSE: An Extreme-Scale Linear Program Solver for Web-Applications
Experiments on real-world data show that our proposed LP solver, ECLIPSE, can solve problems with $10^{12}$ decision variables -- well beyond the capabilities of current solvers.
FALCON: FLOP-Aware Combinatorial Optimization for Neural Network Pruning
In this paper, we propose FALCON, a novel combinatorial-optimization-based framework for network pruning that jointly takes into account model accuracy (fidelity), FLOPs, and sparsity constraints.
Randomization Can Reduce Both Bias and Variance: A Case Study in Random Forests
We study the often overlooked phenomenon, first noted in \cite{breiman2001random}, that random forests appear to reduce bias compared to bagging.
FAST: An Optimization Framework for Fast Additive Segmentation in Transparent ML
We present FAST, an optimization framework for fast additive segmentation.
FFSplit: Split Feed-Forward Network For Optimizing Accuracy-Efficiency Trade-off in Language Model Inference
This usually results in a trade-off between model accuracy and efficiency.
End-to-end Feature Selection Approach for Learning Skinny Trees
Skinny Trees lead to superior feature selection than many existing toolkits e. g., in terms of AUC performance for $25\%$ feature budget, Skinny Trees outperforms LightGBM by $10. 2\%$ (up to $37. 7\%$), and Random Forests by $3\%$ (up to $12. 5\%$).
QuantEase: Optimization-based Quantization for Language Models
Particularly noteworthy is our outlier-aware algorithm's capability to achieve near or sub-3-bit quantization of LLMs with an acceptable drop in accuracy, obviating the need for non-uniform quantization or grouping techniques, improving upon methods such as SpQR by up to two times in terms of perplexity.
Sparse Gaussian Graphical Models with Discrete Optimization: Computational and Statistical Perspectives
To solve the MIP, we propose a custom nonlinear branch-and-bound (BnB) framework that solves node relaxations with tailored first-order methods.
FIRE: An Optimization Approach for Fast Interpretable Rule Extraction
We present FIRE, Fast Interpretable Rule Extraction, an optimization-based framework to extract a small but useful collection of decision rules from tree ensembles.
COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Search
To deal with this challenge, we propose a novel, permutation-based local search method that can complement first-order methods in training any sparse gate, e. g., Hash routing, Top-k, DSelect-k, and COMET.
Matrix Completion from General Deterministic Sampling Patterns
Most of the existing works on provable guarantees for low-rank matrix completion algorithms rely on some unrealistic assumptions such that matrix entries are sampled randomly or the sampling pattern has a specific structure.
Fast as CHITA: Neural Network Pruning with Combinatorial Optimization
Our approach, CHITA, extends the classical Optimal Brain Surgeon framework and results in significant improvements in speed, memory, and performance over existing optimization-based approaches for network pruning.
On Statistical Properties of Sharpness-Aware Minimization: Provable Guarantees
As SAM has been numerically successful, recent papers have studied the theoretical aspects of the framework and have shown SAM solutions are indeed flat.
mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization
Modern deep learning models are over-parameterized, where different optima can result in widely varying generalization performance.
Multi-Task Learning for Sparsity Pattern Heterogeneity: A Discrete Optimization Approach
Allowing the regression coefficients of tasks to have different sparsity patterns (i. e., different supports), we propose a modeling framework for MTL that encourages models to share information across tasks, for a given covariate, through separately 1) shrinking the coefficient supports together, and/or 2) shrinking the coefficient values together.
Improved Deep Neural Network Generalization Using m-Sharpness-Aware Minimization
To that end, this paper presents a thorough empirical evaluation of mSAM on various tasks and datasets.
Pushing the limits of fairness impossibility: Who's the fairest of them all?
The impossibility theorem of fairness is a foundational result in the algorithmic fairness literature.
Quant-BnB: A Scalable Branch-and-Bound Method for Optimal Decision Trees with Continuous Features
Recently there has been significant interest in learning optimal decision trees using various approaches (e. g., based on integer programming, dynamic programming) -- to achieve computational scalability, most of these approaches focus on classification tasks with binary features.
ForestPrune: Compact Depth-Controlled Tree Ensembles
We present ForestPrune, a novel optimization framework to post-process tree ensembles by pruning depth layers from individual trees.
Flexible Modeling and Multitask Learning using Differentiable Tree Ensembles
We therefore propose a novel tensor-based formulation of differentiable trees that allows for efficient vectorization on GPUs.
L0Learn: A Scalable Package for Sparse Learning using L0 Regularization
We present L0Learn: an open-source package for sparse linear regression and classification using $\ell_0$ regularization.
Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance
In this paper, we show that the statistical problems with covariance estimation drive the poor performance of H-score -- a common baseline for newer metrics -- and propose shrinkage-based estimator.
Optimal Ensemble Construction for Multi-Study Prediction with Applications to COVID-19 Excess Mortality Estimation
Importantly, our approach outperforms multi-study stacking and other standard methods in this application.
Predicting Census Survey Response Rates With Parsimonious Additive Models and Structured Interactions
We discuss and interpret findings from our model on the US Census Planning Database.
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning
State-of-the-art MoE models use a trainable sparse gate to select a subset of the experts for each input example.
Linear regression with partially mismatched data: local search with theoretical guarantees
We prove that under a suitable scaling of the number of mismatched pairs compared to the number of samples and features, and certain assumptions on problem data; our local search algorithm converges to a nearly-optimal solution at a linear rate.
Grouped Variable Selection with Discrete Optimization: Computational and Statistical Perspectives
Our algorithmic framework consists of approximate and exact algorithms.
Sparse NMF with Archetypal Regularization: Computational and Robustness Properties
We consider the problem of sparse nonnegative matrix factorization (NMF) using archetypal regularization.
Subgradient Regularized Multivariate Convex Regression at Scale
We present new large-scale algorithms for fitting a subgradient regularized multivariate convex regression function to $n$ samples in $d$ dimensions -- a key problem in shape constrained nonparametric regression with applications in statistics, engineering and the applied sciences.
Sparse Regression at Scale: Branch-and-Bound rooted in First-Order Optimization
In this work, we present a new exact MIP framework for $\ell_0\ell_2$-regularized regression that can scale to $p \sim 10^7$, achieving speedups of at least $5000$x, compared to state-of-the-art exact methods.
The Tree Ensemble Layer: Differentiability meets Conditional Computation
We aim to combine these advantages by introducing a new layer for neural networks, composed of an ensemble of differentiable decision trees (a. k. a.
Learning Sparse Classifiers: Continuous and Mixed Integer Optimization Perspectives
We aim to bridge this gap in computation times by developing new MIP-based algorithms for $\ell_0$-regularized classification.
Computing Estimators of Dantzig Selector type via Column and Constraint Generation
We consider a class of linear-programming based estimators in reconstructing a sparse signal from linear measurements.
Learning Hierarchical Interactions at Scale: A Convex Optimization Approach
In addition, we introduce a specialized active-set strategy with gradient screening for avoiding costly gradient computations.
Solving L1-regularized SVMs and related linear programs: Revisiting the effectiveness of Column and Constraint Generation
The linear Support Vector Machine (SVM) is a classic classification technique in machine learning.
Randomized Gradient Boosting Machine
Gradient Boosting Machine (GBM) introduced by Friedman is a powerful supervised learning algorithm that is very widely used in practice---it routinely features as a leading algorithm in machine learning competitions such as Kaggle and the KDDCup.
Condition Number Analysis of Logistic Regression, and its Implications for Standard First-Order Solution Methods
When the training data is non-separable, we show that the degree of non-separability naturally enters the analysis and informs the properties and convergence guarantees of two standard first-order methods: steepest descent (for any given norm) and stochastic gradient descent.
Fast Best Subset Selection: Coordinate Descent and Local Combinatorial Optimization Algorithms
In spite of the usefulness of $L_0$-based estimators and generic MIO solvers, there is a steep computational price to pay when compared to popular sparse learning algorithms (e. g., based on $L_1$ regularization).
Hierarchical Modeling and Shrinkage for User Session Length Prediction in Media Streaming
In this work we present a novel framework inspired by hierarchical Bayesian modeling to predict, at the moment of login, the amount of time a user will spend in the streaming service.
Matrix Completion with Nonconvex Regularization: Spectral Operators and Scalable Algorithms
Our contributions herein, enhance our prior work on nuclear norm regularized problems for matrix completion (Mazumder et al., 2010) by incorporating a continuum of nonconvex penalty functions between the convex nuclear norm and nonconvex rank functions.
Computation of the Maximum Likelihood estimator in low-rank Factor Analysis
Factor analysis, a classical multivariate statistical technique is popularly used as a fundamental tool for dimensionality reduction in statistics, econometrics and data science.
The Trimmed Lasso: Sparsity and Robustness
Nonconvex penalty methods for sparse modeling in linear regression have been a topic of fervent interest in recent years.
Subset Selection with Shrinkage: Sparse Linear Modeling when the SNR is low
We conduct an extensive theoretical analysis of the predictive properties of the proposed approach and provide justification for its superior predictive performance relative to best subset selection when the noise-level is high.
An Extended Frank-Wolfe Method with "In-Face" Directions, and its Application to Low-Rank Matrix Completion
Motivated principally by the low-rank matrix completion problem, we present an extension of the Frank-Wolfe method that is designed to induce near-optimal solutions on low-dimensional faces of the feasible region.
The Discrete Dantzig Selector: Estimating Sparse Linear Models via Mixed Integer Linear Optimization
We propose new discrete first-order methods, which when paired with state-of-the-art MILO solvers, lead to good solutions for the Discrete Dantzig Selector problem for a given computational budget.
Best Subset Selection via a Modern Optimization Lens
In the last twenty-five years (1990-2014), algorithmic advances in integer optimization combined with hardware improvements have resulted in an astonishing 200 billion factor speedup in solving Mixed Integer Optimization (MIO) problems.
A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives
Furthermore, we show that these new algorithms for the Lasso may also be interpreted as the same master algorithm (subgradient descent), applied to a regularized version of the maximum absolute correlation loss function.
Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares
The matrix-completion problem has attracted a lot of attention, largely as a result of the celebrated Netflix competition.
Flexible Low-Rank Statistical Modeling with Side Information
We propose a general framework for reduced-rank modeling of matrix-valued data.
AdaBoost and Forward Stagewise Regression are First-Order Convex Optimization Methods
Boosting methods are highly popular and effective supervised learning methods which combine weak learners into a single accurate model with good statistical performance.
