Search Results for author:
Found 74 papers, 16 papers with code
Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation
This improvement is comparable to a square-root of the condition number in the worst case and address criticism that guarantees for stochastic acceleration could be worse than those for SGD.
A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features
We prove that training neural networks on 1-D data is equivalent to solving a convex Lasso problem with a fixed, explicitly defined dictionary matrix of features.
Adaptive Inference: Theoretical Limits and Unexplored Opportunities
This paper introduces the first theoretical framework for quantifying the efficiency and performance gain opportunity size of adaptive inference algorithms.
Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time
In this paper, we study the optimality gap between two-layer ReLU networks regularized with weight decay and their convex relaxations.
Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models
In this work we study the enhancement of Low Rank Adaptation (LoRA) fine-tuning procedure by introducing a Riemannian preconditioner in its optimization step.
Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimization
Diffusion models are becoming widely used in state-of-the-art image, video and audio generation.
Distributed Markov Chain Monte Carlo Sampling based on the Alternating Direction Method of Multipliers
Many machine learning applications require operating on a spatially distributed dataset.
The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Models
We also show that all the stationary of the nonconvex training objective can be characterized as the global optimum of a subsampled convex program.
Volumetric Reconstruction Resolves Off-Resonance Artifacts in Static and Dynamic PROPELLER MRI
Off-resonance artifacts in magnetic resonance imaging (MRI) are visual distortions that occur when the actual resonant frequencies of spins within the imaging volume differ from the expected frequencies used to encode spatial information.
Polynomial-Time Solutions for ReLU Network Training: A Complexity Classification via Max-Cut and Zonotopes
Using this convex formulation, we prove that the hardness of approximation of ReLU networks not only mirrors the complexity of the Max-Cut problem but also, in certain special cases, exactly corresponds to it.
Matrix Compression via Randomized Low Rank and Low Precision Factorization
We propose an algorithm that exploits this structure to obtain a low rank decomposition of any matrix $\mathbf{A}$ as $\mathbf{A} \approx \mathbf{L}\mathbf{R}$, where $\mathbf{L}$ and $\mathbf{R}$ are the low rank factors.
From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexity
In this paper, we introduce a novel analysis of neural networks based on geometric (Clifford) algebra and convex optimization.
Randomized Polar Codes for Anytime Distributed Machine Learning
We present a novel distributed computing framework that is robust to slow compute nodes, and is capable of both approximate and exact computation of linear operations.
Iterative Sketching for Secure Coded Regression
Linear regression is a fundamental and primitive problem in supervised machine learning, with applications ranging from epidemiology to finance.
Gradient Coding through Iterative Block Leverage Score Sampling
This is then used to derive an approximate coded computing approach for first-order methods; known as gradient coding, to accelerate linear regression in the presence of failures in distributed computational networks, \textit{i. e.} stragglers.
Optimal Sets and Solution Paths of ReLU Networks
We show that the global optima of the convex parameterization are given by a polyhedral set and then extend this characterization to the optimal set of the non-convex training objective.
Coil Sketching for computationally-efficient MR iterative reconstruction
Results: First, we performed ablation experiments to validate the sketching matrix design on both Cartesian and non-Cartesian datasets.
Globally Optimal Training of Neural Networks with Threshold Activation Functions
We first show that regularized deep threshold network training problems can be equivalently formulated as a standard convex optimization problem, which parallels the LASSO method, provided that the last hidden layer width exceeds a certain threshold.
Complex Clipping for Improved Generalization in Machine Learning
For many machine learning applications, a common input representation is a spectrogram.
Overparameterized ReLU Neural Networks Learn the Simplest Models: Neural Isometry and Exact Recovery
For randomly generated data, we show the existence of a phase transition in recovering planted neural network models, which is easy to describe: whenever the ratio between the number of samples and the dimension exceeds a numerical threshold, the recovery succeeds with high probability; otherwise, it fails with high probability.
GLEAM: Greedy Learning for Large-Scale Accelerated MRI Reconstruction
However, they require several iterations of a large neural network to handle high-dimensional imaging tasks such as 3D MRI.
Optimal Neural Network Approximation of Wasserstein Gradient Direction via Convex Optimization
We study the variational problem in the family of two-layer networks with squared-ReLU activations, towards which we derive a semi-definite programming (SDP) relaxation.
Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers
Vision transformers using self-attention or its proposed alternatives have demonstrated promising results in many image related tasks.
Scale-Equivariant Unrolled Neural Networks for Data-Efficient Accelerated MRI Reconstruction
Unrolled neural networks have enabled state-of-the-art reconstruction performance and fast inference times for the accelerated magnetic resonance imaging (MRI) reconstruction task.
Approximate Function Evaluation via Multi-Armed Bandits
We design an instance-adaptive algorithm that learns to sample according to the importance of each coordinate, and with probability at least $1-\delta$ returns an $\epsilon$ accurate estimate of $f(\boldsymbol{\mu})$.
Distributed Sketching for Randomized Optimization: Exact Characterization, Concentration and Lower Bounds
Furthermore, we develop unbiased parameter averaging methods for randomized second order optimization for regularized problems that employ sketching of the Hessian.
Minimax Optimal Quantization of Linear Models: Information-Theoretic Limits and Efficient Algorithms
We derive an information-theoretic lower bound for the minimax risk under this setting and propose a matching upper bound using randomized embedding-based algorithms which is tight up to constant factors.
Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositions
We develop fast algorithms and robust software for convex optimization of two-layer neural networks with ReLU activation functions.
Using a Novel COVID-19 Calculator to Measure Positive U.S. Socio-Economic Impact of a COVID-19 Pre-Screening Solution (AI/ML)
The COVID-19 pandemic has been a scourge upon humanity, claiming the lives of more than 5. 1 million people worldwide; the global economy contracted by 3. 5% in 2020.
Using Deep Learning with Large Aggregated Datasets for COVID-19 Classification from Cough
The Covid-19 pandemic has been one of the most devastating events in recent history, claiming the lives of more than 5 million people worldwide.
Greedy Learning for Large-Scale Neural MRI Reconstruction
Model-based deep learning approaches have recently shown state-of-the-art performance for accelerated MRI reconstruction.
Parallel Deep Neural Networks Have Zero Duality Gap
Recent work has proven that the strong duality holds (which means zero duality gap) for regularized finite-width two-layer ReLU networks and consequently provided an equivalent convex training problem.
The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program
We then show that the limit points of non-convex subgradient flows can be identified via primal-dual correspondence in this convex optimization problem.
Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs
We first show that the training of multiple three-layer ReLU sub-networks with weight decay regularization can be equivalently cast as a convex optimization problem in a higher dimensional space, where sparsity is enforced via a group $\ell_1$-norm regularization.
The Hidden Convex Optimization Landscape of Regularized Two-Layer ReLU Networks: an Exact Characterization of Optimal Solutions
As additional consequences of our convex perspective, (i) we establish that Clarke stationary points found by stochastic gradient descent correspond to the global optimum of a subsampled convex problem (ii) we provide a polynomial-time algorithm for checking if a neural network is a global minimum of the training loss (iii) we provide an explicit construction of a continuous path between any neural network and the global minimum of its sublevel set and (iv) characterize the minimal size of the hidden layer so that the neural network optimization landscape has no spurious valleys.
Newton-LESS: Sparsification without Trade-offs for the Sketched Newton Update
In second-order optimization, a potential bottleneck can be computing the Hessian matrix of the optimized function at every iteration.
Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions
In this work, we analyze the training of Wasserstein GANs with two-layer neural network discriminators through the lens of convex duality, and for a variety of generators expose the conditions under which Wasserstein GANs can be solved exactly with convex optimization approaches, or can be represented as convex-concave games.
Adaptive Newton Sketch: Linear-time Optimization with Quadratic Convergence and Effective Hessian Dimensionality
Our first contribution is to show that, at each iteration, the embedding dimension (or sketch size) can be as small as the effective dimension of the Hessian matrix.
Training Quantized Neural Networks to Global Optimality via Semidefinite Programming
Neural networks (NNs) have been extremely successful across many tasks in machine learning.
Fast Convex Quadratic Optimization Solvers with Adaptive Sketching-based Preconditioners
We propose an adaptive mechanism to control the sketch size according to the progress made in each step of the iterative solver.
Efficient Randomized Subspace Embeddings for Distributed Optimization under a Communication Budget
As a consequence, quantizing these embeddings followed by an inverse transform to the original space yields a source coding method with optimal covering efficiency while utilizing just $R$-bits per dimension.
Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization
Batch Normalization (BN) is a commonly used technique to accelerate and stabilize training of deep neural networks.
Boost AI Power: Data Augmentation Strategies with unlabelled Data and Conformal Prediction, a Case in Alternative Herbal Medicine Discrimination with Electronic Nose
Furthermore, this study provides a systematic analysis of different augmentation strategies.
Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time
In this paper, we develop exact convex optimization formulations for two-layer neural networks with second degree polynomial activations based on semidefinite programming.
Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms
We describe the convex semi-infinite dual of the two-layer vector-output ReLU neural network training problem.
Adaptive and Oblivious Randomized Subspace Methods for High-Dimensional Optimization: Sharp Analysis and Lower Bounds
We propose novel randomized optimization methods for high-dimensional convex problems based on restrictions of variables to random subspaces.
Convex Regularization Behind Neural Reconstruction
Neural networks have shown tremendous potential for reconstructing high-resolution images in inverse problems.
Optimal Iterative Sketching Methods with the Subsampled Randomized Hadamard Transform
These show that the convergence rate for Haar and randomized Hadamard matrices are identical, and asymptotically improve upon Gaussian random projections.
Linear Predictive Coding for Acute Stress Prediction from Computer Mouse Movements
This work demonstrates that the damping frequency and damping ratio from LPC are significantly correlated with those from an MSD model, thus confirming the validity of using LPC to infer muscle stiffness and damping.
Debiasing Distributed Second Order Optimization with Surrogate Sketching and Scaled Regularization
In distributed second order optimization, a standard strategy is to average many local estimates, each of which is based on a small sketch or batch of the data.
Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time
We study training of Convolutional Neural Networks (CNNs) with ReLU activations and introduce exact convex optimization formulations with a polynomial complexity with respect to the number of data samples, the number of neurons, and data dimension.
Lower Bounds and a Near-Optimal Shrinkage Estimator for Least Squares using Random Projections
An upper bound on the expected error of this estimator is derived, which is smaller than the error of the classical Gaussian sketch solution for any given data.
The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutions
As additional consequences of our convex perspective, (i) we establish that Clarke stationary points found by stochastic gradient descent correspond to the global optimum of a subsampled convex problem (ii) we provide a polynomial-time algorithm for checking if a neural network is a global minimum of the training loss (iii) we provide an explicit construction of a continuous path between any neural network and the global minimum of its sublevel set and (iv) characterize the minimal size of the hidden layer so that the neural network optimization landscape has no spurious valleys.
Effective Dimension Adaptive Sketching Methods for Faster Regularized Least-Squares Optimization
Our method starts with an initial embedding dimension equal to 1 and, over iterations, increases the embedding dimension up to the effective one at most.
Global Multiclass Classification and Dataset Construction via Heterogeneous Local Experts
In the domains of dataset construction and crowdsourcing, a notable challenge is to aggregate labels from a heterogeneous set of labelers, each of whom is potentially an expert in some subset of tasks (and less reliable in others).
Separating the Effects of Batch Normalization on CNN Training Speed and Stability Using Classical Adaptive Filter Theory
Batch Normalization (BatchNorm) is commonly used in Convolutional Neural Networks (CNNs) to improve training speed and stability.
Convex Geometry and Duality of Over-parameterized Neural Networks
Our analysis also shows that optimal network parameters can be also characterized as interpretable closed-form formulas in some practically relevant special cases.
Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks
We develop exact representations of training two-layer neural networks with rectified linear units (ReLUs) in terms of a single convex program with number of variables polynomial in the number of training samples and the number of hidden neurons.
Revealing the Structure of Deep Neural Networks via Convex Duality
We show that a set of optimal hidden layer weights for a norm regularized DNN training problem can be explicitly found as the extreme points of a convex set.
Optimal Randomized First-Order Methods for Least-Squares Problems
Then, we propose a new algorithm by optimizing the computational complexity over the choice of the sketching dimension.
Distributed Averaging Methods for Randomized Second Order Optimization
We consider distributed optimization problems where forming the Hessian is computationally challenging and communication is a significant bottleneck.
Distributed Sketching Methods for Privacy Preserving Regression
In this work, we study distributed sketching methods for large scale regression problems.
Global Convergence of Frank Wolfe on One Hidden Layer Networks
The classical Frank Wolfe algorithm then converges with rate $O(1/T)$ where $T$ is both the number of neurons and the number of calls to the oracle.
Optimal Iterative Sketching with the Subsampled Randomized Hadamard Transform
These show that the convergence rate for Haar and randomized Hadamard matrices are identical, and asymptotically improve upon Gaussian random projections.
Regularized Momentum Iterative Hessian Sketch for Large Scale Linear System of Equations
In this article, Momentum Iterative Hessian Sketch (M-IHS) techniques, a group of solvers for large scale linear Least Squares (LS) problems, are proposed and analyzed in detail.
Optimization and Control Computational Complexity 15B52, 65F08, 65F10, 65F22, 65F50, 68W20, 90C06
Distributed Black-Box Optimization via Error Correcting Codes
We introduce a novel distributed derivative-free optimization framework that is resilient to stragglers.
High-Dimensional Optimization in Adaptive Random Subspaces
We propose a new randomized optimization method for high-dimensional problems which can be seen as a generalization of coordinate descent to random subspaces.
Straggler Resilient Serverless Computing Based on Polar Codes
We propose a serverless computing mechanism for distributed computation based on polar codes.
Convex Relaxations of Convolutional Neural Nets
We propose convex relaxations for convolutional neural nets with one hidden layer where the output weights are fixed.
Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence
We also describe extensions of our methods to programs involving convex constraints that are equipped with self-concordant barriers.
Randomized sketches for kernels: Fast and optimal non-parametric regression
Kernel ridge regression (KRR) is a standard method for performing non-parametric regression over reproducing kernel Hilbert spaces.
Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squares
We study randomized sketching methods for approximately solving least-squares problem with a general convex constraint.
Randomized Sketches of Convex Programs with Sharp Guarantees
We analyze RP-based approximations of convex programs, in which the original optimization problem is approximated by the solution of a lower-dimensional problem.
Recovery of Sparse Probability Measures via Convex Programming
We propose a direct relaxation of the minimum cardinality problem and show that it can be efficiently solved using convex programming.
