Search Results for author:
Found 74 papers, 13 papers with code
Better Schedules for Low Precision Training of Deep Neural Networks
From these experiments, we discover alternative CPT schedules that offer further improvements in training efficiency and model performance, as well as derive a set of best practices for choosing CPT schedules.
On the Error-Propagation of Inexact Deflation for Principal Component Analysis
Principal Component Analysis (PCA) is a popular tool in data analysis, especially when the data is high-dimensional.
CrysFormer: Protein Structure Prediction via 3d Patterson Maps and Partial Structure Attention
Determining the structure of a protein has been a decades-long open question.
Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation
Large Language Models (LLMs) have the ability to solve a variety of tasks, such as text summarization and mathematical questions, just out of the box, but they are often trained with a single task in mind.
Fast FixMatch: Faster Semi-Supervised Learning with Curriculum Batch Size
Advances in Semi-Supervised Learning (SSL) have almost entirely closed the gap between SSL and Supervised Learning at a fraction of the number of labels.
Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beat
We carefully evaluate a number of algorithms for learning in a federated environment, and test their utility for a variety of image classification tasks.
Adaptive Federated Learning with Auto-Tuned Clients
Federated learning (FL) is a distributed machine learning framework where the global model of a central server is trained via multiple collaborative steps by participating clients without sharing their data.
FedJETs: Efficient Just-In-Time Personalization with Federated Mixture of Experts
Our gating function harnesses the knowledge of a pretrained model common expert to enhance its routing decisions on-the-fly.
Provable Accelerated Convergence of Nesterov's Momentum for Deep ReLU Neural Networks
Current state-of-the-art analyses on the convergence of gradient descent for training neural networks focus on characterizing properties of the loss landscape, such as the Polyak-Lojaciewicz (PL) condition and the restricted strong convexity.
When is Momentum Extragradient Optimal? A Polynomial-Based Analysis
The extragradient method has gained popularity due to its robust convergence properties for differentiable games.
Cold Start Streaming Learning for Deep Networks
To mitigate these shortcomings, we propose Cold Start Streaming Learning (CSSL), a simple, end-to-end approach for streaming learning with deep networks that uses a combination of replay and data augmentation to avoid catastrophic forgetting.
Strong Lottery Ticket Hypothesis with $\varepsilon$--perturbation
The strong Lottery Ticket Hypothesis (LTH) claims the existence of a subnetwork in a sufficiently large, randomly initialized neural network that approximates some target neural network without the need of training.
LOFT: Finding Lottery Tickets through Filter-wise Training
\textsc{LoFT} is a model-parallel pretraining algorithm that partitions convolutional layers by filters to train them independently in a distributed setting, resulting in reduced memory and communication costs during pretraining.
Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropout
Asynchronous learning protocols have regained attention lately, especially in the Federated Learning (FL) setup, where slower clients can severely impede the learning process.
DPMS: An ADD-Based Symbolic Approach for Generalized MaxSAT Solving
They lack, however, the ability to handle 1) (non-CNF) hybrid constraints, such as XORs and 2) generalized MaxSAT problems natively.
Local Stochastic Factored Gradient Descent for Distributed Quantum State Tomography
We propose a distributed Quantum State Tomography (QST) protocol, named Local Stochastic Factored Gradient Descent (Local SFGD), to learn the low-rank factor of a density matrix over a set of local machines.
PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication
Notably, little is known regarding the convergence rate of GCN training with both stale features and stale feature gradients.
No More Than 6ft Apart: Robust K-Means via Radius Upper Bounds
We propose to remedy such a scenario by introducing a maximal radius constraint $r$ on the clusters formed by the centroids, i. e., samples from the same cluster should not be more than $2r$ apart in terms of $\ell_2$ distance.
i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery
We propose a novel, structured pruning algorithm for neural networks -- the iterative, Sparse Structured Pruning algorithm, dubbed as i-SpaSP.
On the Convergence of Shallow Neural Network Training with Randomly Masked Neurons
With the motive of training all the parameters of a neural network, we study why and when one can achieve this by iteratively creating, training, and combining randomly selected subnetworks.
Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentum
Stochastic gradient descent with momentum (SGDM) is the dominant algorithm in many optimization scenarios, including convex optimization instances and non-convex neural network training.
Federated Multiple Label Hashing (FedMLH): Communication Efficient Federated Learning on Extreme Classification Tasks
Federated learning enables many local devices to train a deep learning model jointly without sharing the local data.
How much pre-training is enough to discover a good subnetwork?
Aiming to mathematically analyze the amount of dense network pre-training needed for a pruned network to perform well, we discover a simple theoretical bound in the number of gradient descent pre-training iterations on a two-layer, fully-connected network, beyond which pruning via greedy forward selection [61] yields a subnetwork that achieves good training error.
REX: Revisiting Budgeted Training with an Improved Schedule
Deep learning practitioners often operate on a computational and monetary budget.
Mitigating deep double descent by concatenating inputs
In this work, we explore the connection between the double descent phenomena and the number of samples in the deep neural network setting.
ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
Thus, ResIST reduces the per-iteration communication, memory, and time requirements of ResNet training to only a fraction of the requirements of full-model training.
Momentum-inspired Low-Rank Coordinate Descent for Diagonally Constrained SDPs
We present a novel, practical, and provable approach for solving diagonally constrained semi-definite programming (SDP) problems at scale using accelerated non-convex programming.
Fast quantum state reconstruction via accelerated non-convex programming
Despite being a non-convex method, \texttt{MiFGD} converges \emph{provably} close to the true density matrix at an accelerated linear rate, in the absence of experimental and statistical noise, and under common assumptions.
GIST: Distributed Training for Large-Scale Graph Convolutional Networks
The graph convolutional network (GCN) is a go-to solution for machine learning on graphs, but its training is notoriously difficult to scale both in terms of graph size and the number of model parameters.
Rank-One Measurements of Low-Rank PSD Matrices Have Small Feasible Sets
We study the role of the constraint set in determining the solution to low-rank, positive semidefinite (PSD) matrix sensing problems.
On Continuous Local BDD-Based Search for Hybrid SAT Solving
We explore the potential of continuous local search (CLS) in SAT solving by proposing a novel approach for finding a solution of a hybrid system of Boolean constraints.
On Generalization of Adaptive Methods for Over-parameterized Linear Regression
In this paper, we aim to characterize the performance of adaptive methods in the over-parameterized linear regression setting.
StackMix: A complementary Mix algorithm
On its own, improvements with StackMix hold across different number of labeled samples on CIFAR-100, maintaining approximately a 2\% gap in test accuracy -- down to using only 5\% of the whole dataset -- and is effective in the semi-supervised setting with a 2\% improvement with the standard benchmark $\Pi$-model.
Bayesian Coresets: Revisiting the Nonconvex Optimization Perspective
Bayesian coresets have emerged as a promising approach for implementing scalable Bayesian inference.
FourierSAT: A Fourier Expansion-Based Algebraic Framework for Solving Hybrid Boolean Constraints
By such a reduction to continuous optimization, we propose an algebraic framework for solving systems consisting of different types of constraints.
Optimal Mini-Batch Size Selection for Fast Gradient Descent
This paper presents a methodology for selecting the mini-batch size that minimizes Stochastic Gradient Descent (SGD) learning time for single and multiple learner problems.
Negative sampling in semi-supervised learning
We introduce Negative Sampling in Semi-Supervised Learning (NS3L), a simple, fast, easy to tune algorithm for semi-supervised learning (SSL).
Learning Sparse Distributions using Iterative Hard Thresholding
Iterative hard thresholding (IHT) is a projected gradient descent algorithm, known to achieve state of the art performance for a wide range of structured estimation problems, such as sparse inference.
Demon: Improved Neural Network Training with Momentum Decay
Momentum is a widely used technique for gradient-based optimizers in deep learning.
Distributed Learning of Deep Neural Networks using Independent Subnet Training
These properties of IST can cope with issues due to distributed data, slow interconnects, or limited device memory, making IST a suitable approach for cases of mandatory distribution.
Decaying momentum helps neural network training
Momentum is a simple and popular technique in deep learning for gradient-based optimizers.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Compressing Gradient Optimizers via Count-Sketches
The problem is becoming more severe as deep learning models continue to grow larger in order to learn from complex, large-scale datasets.
Minimum weight norm models do not always generalize well for over-parameterized problems
We empirically show that the minimum weight norm is not necessarily the proper gauge of good generalization in simplified scenaria, and different models found by adaptive methods could outperform plain gradient methods.
Implicit regularization and solution uniqueness in over-parameterized matrix sensing
Surprisingly, recent work argues that the choice of $r \leq n$ is not pivotal: even setting $U \in \mathbb{R}^{n \times n}$ is sufficient for factored gradient descent to find the rank-$r$ solution, which suggests that operating over the factors leads to an implicit regularization.
Provably convergent acceleration in factored gradient descent with applications in matrix sensing
We present theoretical results on the convergence of \emph{non-convex} accelerated gradient descent in matrix factorization models with $\ell_2$-norm loss.
Simple and practical algorithms for $\ell_p$-norm low-rank approximation
We propose practical algorithms for entrywise $\ell_p$-norm low-rank approximation, for $p = 1$ or $p = \infty$.
Approximate Newton-based statistical inference using only stochastic gradients
We present a novel statistical inference framework for convex empirical risk minimization, using approximate stochastic Newton steps.
IHT dies hard: Provable accelerated Iterative Hard Thresholding
We study --both in theory and practice-- the use of momentum motions in classic iterative hard thresholding (IHT) methods.
Statistical inference using SGD
We present a novel method for frequentist statistical inference in $M$-estimation problems, based on stochastic gradient descent (SGD) with a fixed step size: we demonstrate that the average of such SGD sequences can be used for statistical inference, after proper scaling.
Non-square matrix sensing without spurious local minima via the Burer-Monteiro approach
We consider the non-square matrix sensing problem, under restricted isometry property (RIP) assumptions.
Finding Low-Rank Solutions via Non-Convex Matrix Factorization, Efficiently and Provably
We study such parameterization for optimization of generic convex objectives $f$, and focus on first-order, gradient descent algorithmic solutions.
Provable Burer-Monteiro factorization for a class of norm-constrained matrix problems
We study the projected gradient descent method on low-rank matrix problems with a strongly convex objective.
A simple and provable algorithm for sparse diagonal CCA
Given two sets of variables, derived from a common set of samples, sparse Canonical Correlation Analysis (CCA) seeks linear combinations of a small number of variables in each set, such that the induced canonical variables are maximally correlated.
Algorithms for Learning Sparse Additive Models with Interactions in High Dimensions
A function $f: \mathbb{R}^d \rightarrow \mathbb{R}$ is a Sparse Additive Model (SPAM), if it is of the form $f(\mathbf{x}) = \sum_{l \in \mathcal{S}}\phi_{l}(x_l)$ where $\mathcal{S} \subset [d]$, $|\mathcal{S}| \ll d$.
Learning Sparse Additive Models with Interactions in High Dimensions
For some $\mathcal{S}_1 \subset [d], \mathcal{S}_2 \subset {[d] \choose 2}$, the function $f$ is assumed to be of the form: $$f(\mathbf{x}) = \sum_{p \in \mathcal{S}_1}\phi_{p} (x_p) + \sum_{(l, l^{\prime}) \in \mathcal{S}_2}\phi_{(l, l^{\prime})} (x_{l}, x_{l^{\prime}}).$$ Assuming $\phi_{p},\phi_{(l, l^{\prime})}$, $\mathcal{S}_1$ and, $\mathcal{S}_2$ to be unknown, we provide a randomized algorithm that queries $f$ and exactly recovers $\mathcal{S}_1,\mathcal{S}_2$.
Trading-off variance and complexity in stochastic gradient descent
Stochastic gradient descent is the method of choice for large-scale machine learning problems, by virtue of its light complexity per iteration.
Convex block-sparse linear regression with expanders -- provably
Our experimental findings on synthetic and real applications support our claims for faster recovery in the convex setting -- as opposed to using dense sensing matrices, while showing a competitive recovery performance.
Bipartite Correlation Clustering -- Maximizing Agreements
We present a novel approximation algorithm for $k$-BCC, a variant of BCC with an upper bound $k$ on the number of clusters.
A single-phase, proximal path-following framework
First, it allows handling non-smooth objectives via proximal operators; this avoids lifting the problem dimension in order to accommodate non-smooth components in optimization.
Dropping Convexity for Faster Semi-definite Optimization
To the best of our knowledge, this is the first paper to provide precise convergence rate guarantees for general convex functions under standard convex assumptions.
Sparse PCA via Bipartite Matchings
We consider the following multi-component sparse PCA problem: given a set of data points, we seek to extract a small number of sparse components with disjoint supports that jointly capture the maximum possible variance.
Linear Inverse Problems with Norm and Sparsity Constraints
We describe two nonconventional algorithms for linear regression, called GAME and CLASH.
Structured Sparsity: Discrete and Convex approaches
For each, we present the models in their discrete nature, discuss how to solve the ensuing discrete problems and then describe convex relaxations.
Stay on path: PCA along graph paths
We introduce a variant of (sparse) PCA in which the set of feasible support sets is determined by a graph.
Compressive Mining: Fast and Optimal Data Mining in the Compressed Domain
However, distance estimation when the data are represented using different sets of coefficients is still a largely unexplored area.
Scalable sparse covariance estimation via self-concordance
We consider the class of convex minimization problems, composed of a self-concordant function, such as the $\log\det$ metric, a convex data fidelity term $h(\cdot)$ and, a regularizing -- possibly non-smooth -- function $g(\cdot)$.
Provable Deterministic Leverage Score Sampling
We explain theoretically a curious empirical phenomenon: "Approximating a matrix by deterministically selecting a subset of its columns with the corresponding largest leverage scores results in a good low-rank matrix surrogate".
Approximate Matrix Multiplication with Application to Linear Embeddings
In this paper, we study the problem of approximately computing the product of two real matrices.
Non-uniform Feature Sampling for Decision Tree Ensembles
We study the effectiveness of non-uniform randomized feature selection in decision tree classification.
An Inexact Proximal Path-Following Algorithm for Constrained Convex Minimization
Many scientific and engineering applications feature nonsmooth convex minimization problems over convex sets.
Composite Self-Concordant Minimization
We propose a variable metric framework for minimizing the sum of a self-concordant function and a possibly non-smooth convex function, endowed with an easily computable proximal operator.
Group-Sparse Model Selection: Hardness and Relaxations
Group-based sparsity models are proven instrumental in linear regression problems for recovering signals from much fewer measurements than standard compressive sensing.
Sparse projections onto the simplex
Most learning methods with rank or sparsity constraints use convex relaxations, which lead to optimization with the nuclear norm or the $\ell_1$-norm.
