Search Results for author:
Found 67 papers, 19 papers with code
Compressive sensing with un-trained neural networks: Gradient descent finds a smooth approximation
For signal recovery from a few measurements, however, un-trained convolutional networks have an intriguing self-regularizing property: Even though the network can perfectly fit any image, the network recovers a natural image from few measurements when trained with gradient descent until convergence.
CryptoMamba: Leveraging State Space Models for Accurate Bitcoin Price Prediction
Predicting Bitcoin price remains a challenging problem due to the high volatility and complex non-linear dynamics of cryptocurrency markets.
Stability properties of gradient flow dynamics for the symmetric low-rank matrix factorization problem
We demonstrate that the Schur complement to a principal eigenspace of the target matrix is governed by an autonomous system that is decoupled from the rest of the dynamics.
MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models
Multimodal Large Language Models (MLLMs) have tremendous potential to improve the accuracy, availability, and cost-effectiveness of healthcare by providing automated solutions or serving as aids to medical professionals.
Ranked #1 on
Medical Visual Question Answering
on MediConfusion
Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learning
Despite the wide practical adoption of CL and MTL and extensive literature on both areas, there remains a gap in the theoretical understanding of these methods when used with overparameterized models such as deep neural networks.
Serpent: Scalable and Efficient Image Restoration via Multi-scale Structured State Space Models
We propose a novel hierarchical architecture inspired by traditional signal processing principles, that converts the input image into a collection of sequences and processes them in a multi-scale fashion.
Gradient Descent Provably Solves Nonlinear Tomographic Reconstruction
We show that this approach reduces metal artifacts compared to a commercial reconstruction of a human skull with metal dental crowns.
Learning A Disentangling Representation For PU Learning
In this paper we propose to learn a neural network-based data representation using a loss function that can be used to project the unlabeled data into two (positive and negative) clusters that can be easily identified using simple clustering techniques, effectively emulating the phenomenon observed in low-dimensional settings.
Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models
Our framework, Flash-Diffusion, acts as a wrapper that can be combined with any latent diffusion-based baseline solver, imbuing it with sample-adaptivity and acceleration.
mL-BFGS: A Momentum-based L-BFGS for Distributed Large-Scale Neural Network Optimization
Quasi-Newton methods still face significant challenges in training large-scale neural networks due to additional compute costs in the Hessian related computations and instability issues in stochastic training.
Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks
An increasingly popular machine learning paradigm is to pretrain a neural network (NN) on many tasks offline, then adapt it to downstream tasks, often by re-training only the last linear layer of the network.
Don't Memorize; Mimic The Past: Federated Class Incremental Learning Without Episodic Memory
Deep learning models are prone to forgetting information learned in the past when trained on new data.
On the Role of Attention in Prompt-tuning
Despite its success in LLMs, there is limited theoretical understanding of the power of prompt-tuning and the role of the attention mechanism in prompting.
DiracDiffusion: Denoising and Incremental Reconstruction with Assured Data-Consistency
In this work, we propose a novel framework for inverse problem solving, namely we assume that the observation comes from a stochastic degradation process that gradually degrades and noises the original clean image.
Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensing
We show that in this setting, factorized gradient descent enjoys two implicit properties: (1) coupling of the trajectory of gradient descent where the factors are coupled in various ways throughout the gradient update trajectory and (2) an algorithmic regularization property where the iterates show a propensity towards low-rank models despite the overparameterized nature of the factorized model.
CUDA: Convolution-based Unlearnable Datasets
Here, ERM on the clean training data achieves a clean test accuracy of 80. 66$\%$.
SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing
The mixture of Expert (MoE) parallelism is a recent advancement that scales up the model size with constant computational cost.
The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning
Self-supervised learning (SSL) has emerged as a desirable paradigm in computer vision due to the inability of supervised models to learn representations that can generalize in domains with limited labels.
Neural Networks can Learn Representations with Gradient Descent
Furthermore, in a transfer learning setup where the data distributions in the source and target domain share the same representation $U$ but have different polynomial heads we show that a popular heuristic for transfer learning has a target sample complexity independent of $d$.
Toward a Geometrical Understanding of Self-supervised Contrastive Learning
When used for transfer learning, the projector is discarded since empirical results show that its representation generalizes more poorly than the encoder's.
Learning from many trajectories
Specifically, we establish that the worst-case error rate of this problem is $\Theta(n / m T)$ whenever $m \gtrsim n$.
HUMUS-Net: Hybrid unrolled multi-scale network architecture for accelerated MRI reconstruction
These models split input images into non-overlapping patches, embed the patches into lower-dimensional tokens and utilize a self-attention mechanism that does not suffer from the aforementioned weaknesses of convolutional architectures.
Ranked #1 on
MRI Reconstruction
on fastMRI Knee 8x
(using extra training data)
FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks
To bridge the gap and facilitate the development of FL for computer vision tasks, in this work, we propose a federated learning library and benchmarking framework, named FedCV, to evaluate FL on the three most representative computer vision tasks: image classification, image segmentation, and object detection.
SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
In this paper we propose self-supervised federated learning (SSFL), a unified self-supervised and personalized federated learning framework, and a series of algorithms under this framework which work towards addressing these challenges.
Fundamental Limits of Transfer Learning in Binary Classifications
Transfer learning is gaining traction as a promising technique to alleviate this barrier by utilizing the data of a related but different \emph{source} task to compensate for the lack of data in a \emph{target} task where there are few labeled training data.
SLIM-QN: A Stochastic, Light, Momentumized Quasi-Newton Optimizer for Deep Neural Networks
SLIM-QN addresses two key barriers in existing second-order methods for large-scale DNNs: 1) the high computational cost of obtaining the Hessian matrix and its inverse in every iteration (e. g. KFAC); 2) convergence instability due to stochastic training (e. g. L-BFGS).
Outlier-Robust Sparse Estimation via Non-Convex Optimization
We explore the connection between outlier-robust high-dimensional statistics and non-convex optimization in the presence of sparsity constraints, with a focus on the fundamental tasks of robust sparse mean estimation and robust sparse PCA.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
Data augmentation for deep learning based accelerated MRI reconstruction with limited data
Deep neural networks have emerged as very successful tools for image restoration and reconstruction tasks.
Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction
Recently there has been significant theoretical progress on understanding the convergence and generalization of gradient-based methods on nonconvex losses with overparameterized models.
Generalization Guarantees for Neural Architecture Search with Train-Validation Split
In this approach, it is common to use bilevel optimization where one optimizes the model weights over the training data (inner problem) and various hyperparameters such as the configuration of the architecture over the validation data (outer problem).
FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks
Increasing concerns and regulations about data privacy and sparsity necessitate the study of privacy-preserving, decentralized learning methods for natural language processing (NLP) tasks.
Understanding Overparameterization in Generative Adversarial Networks
We also empirically study the role of model overparameterization in GANs using several large-scale experiments on CIFAR-10 and Celeb-A datasets.
PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers
PipeTransformer automatically adjusts the pipelining and data parallelism by identifying and freezing some layers during the training, and instead allocates resources for training of the remaining active layers.
Understanding Over-parameterization in Generative Adversarial Networks
In this work, we present a comprehensive analysis of the importance of model over-parameterization in GANs both theoretically and empirically.
Data augmentation for deep learning based accelerated MRI reconstruction
Inspired by the success of Data Augmentation (DA) for classification problems, in this paper, we propose a pipeline for data augmentation for image reconstruction tasks arising in medical imaging and explore its effectiveness at reducing the required training data in a variety of settings.
Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View
Our theoretical analysis allows us to precisely characterize how the test error varies over different training algorithms, data distributions, problem dimensions as well as number of classes, inter/intra class correlations and class priors.
Precise Statistical Analysis of Classification Accuracies for Adversarial Training
Despite the wide empirical success of modern machine learning algorithms and models in a multitude of applications, they are known to be highly susceptible to seemingly small indiscernible perturbations to the input data known as \emph{adversarial attacks}.
Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks
In this approach a model trained for a source task, where plenty of labeled training data is available, is used as a starting point for training a model on a related target task with only few labeled training data.
Learning the model-free linear quadratic regulator via random search
Model-free reinforcement learning attempts to find an optimal control action for an unknown dynamical system by directly searching over the parameter space of controllers.
Approximation Schemes for ReLU Regression
We consider the fundamental problem of ReLU regression, where the goal is to output the best fitting ReLU with respect to square loss given access to draws from some unknown distribution.
Compressive sensing with un-trained neural networks: Gradient descent finds the smoothest approximation
For signal recovery from a few measurements, however, un-trained convolutional networks have an intriguing self-regularizing property: Even though the network can perfectly fit any image, the network recovers a natural image from few measurements when trained with gradient descent until convergence.
High-Dimensional Robust Mean Estimation via Gradient Descent
We study the problem of high-dimensional robust mean estimation in the presence of a constant fraction of adversarial outliers.
Precise Tradeoffs in Adversarial Training for Linear Regression
Furthermore, we precisely characterize the standard/robust accuracy and the corresponding tradeoff achieved by a contemporary mini-max adversarial training approach in a high-dimensional regime where the number of data points and the parameters of the model grow in proportion to each other.
Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problem
Model-free reinforcement learning attempts to find an optimal control action for an unknown dynamical system by directly searching over the parameter space of controllers.
Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators
A surprising experiment that highlights this architectural bias towards natural images is that one can remove noise and corruptions from a natural image without using any training data, by simply fitting (via gradient descent) a randomly initialized, over-parameterized convolutional generator to the corrupted image.
GENERALIZATION GUARANTEES FOR NEURAL NETS VIA HARNESSING THE LOW-RANKNESS OF JACOBIAN
We show that over the information space learning is fast and one can quickly train a model with zero training loss that can also generalize well.
Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian
We show that over the information space learning is fast and one can quickly train a model with zero training loss that can also generalize well.
Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
In particular, we prove that: (i) In the first few iterations where the updates are still in the vicinity of the initialization gradient descent only fits to the correct labels essentially ignoring the noisy labels.
Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
However, in practice much more moderate levels of overparameterization seems to be sufficient and in many cases overparameterized models seem to perfectly interpolate the training data as soon as the number of parameters exceed the size of the training data by a constant factor.
Fitting ReLUs via SGD and Quantized SGD
Perhaps unexpectedly, we show that QSGD maintains the fast convergence of SGD to a globally optimal model while significantly reducing the communication cost.
Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
In this paper we demonstrate that when the loss has certain properties over a minimally small neighborhood of the initial point, first order methods such as (stochastic) gradient descent have a few intriguing properties: (1) the iterates converge at a geometric rate to a global optima even when the loss is nonconvex, (2) among all global optima of the loss the iterates converge to one with a near minimal distance to the initial point, (3) the iterates take a near direct route from the initial point to this global optima.
Compressed Sensing with Deep Image Prior and Learned Regularization
We propose a novel method for compressed sensing recovery using untrained deep generative models.
Lagrange Coded Computing: Optimal Design for Resiliency, Security and Privacy
We consider a scenario involving computations over a massive dataset stored distributedly across multiple workers, which is at the core of distributed learning algorithms.
Polynomially Coded Regression: Optimal Straggler Mitigation via Data Encoding
In particular, PCR requires a recovery threshold that scales inversely proportionally with the amount of computation/storage available at each worker.
End-to-end Learning of a Convolutional Neural Network via Deep Tensor Decomposition
In this paper we study the problem of learning the weights of a deep convolutional neural network.
Fundamental Resource Trade-offs for Encoded Distributed Optimization
We also analyze the convergence behavior of iterative encoded optimization algorithms, allowing us to characterize fundamental trade-offs between convergence rate, size of data set, accuracy, computational load (or data redundancy), and straggler toleration in this framework.
Gradient Methods for Submodular Maximization
Despite the apparent lack of convexity in such functions, we prove that stochastic projected gradient methods can provide strong approximation guarantees for maximizing continuous submodular functions with convex constraints.
Theoretical insights into the optimization landscape of over-parameterized shallow neural networks
In this paper we study the problem of learning a shallow artificial neural network that best fits a training data set.
Learning ReLUs via Gradient Descent
In this paper we study the problem of learning Rectified Linear Units (ReLUs) which are functions of the form $max(0,<w, x>)$ with $w$ denoting the weight vector.
Structured signal recovery from quadratic measurements: Breaking sample complexity barriers via nonconvex optimization
We focus on the under-determined setting where the number of measurements is significantly smaller than the dimension of the signal ($m<<n$).
Fast and Reliable Parameter Estimation from Nonlinear Observations
In this paper we study the problem of recovering a structured but unknown parameter ${\bf{\theta}}^*$ from $n$ nonlinear observations of the form $y_i=f(\langle {\bf{x}}_i,{\bf{\theta}}^*\rangle)$ for $i=1, 2,\ldots, n$.
Experimental robustness of Fourier Ptychography phase retrieval algorithms
Both noise (e. g. Poisson noise) and model mis-match errors are shown to scale with intensity.
Sharp Time--Data Tradeoffs for Linear Inverse Problems
We sharply characterize the convergence rate associated with a wide variety of random measurement ensembles in terms of the number of measurements and structural complexity of the signal with respect to the chosen penalty function.
Isometric sketching of any set via the Restricted Isometry Property
In this paper we show that for the purposes of dimensionality reduction certain class of structured random matrices behave similarly to random Gaussian matrices.
Approximate Subspace-Sparse Recovery with Corrupted Data via Constrained $\ell_1$-Minimization
High-dimensional data often lie in low-dimensional subspaces corresponding to different classes they belong to.
Robust subspace clustering
Subspace clustering refers to the task of finding a multi-subspace representation that best fits a collection of points taken from a high-dimensional space.
