Search Results for author:
Found 68 papers, 14 papers with code
Semi-supervised Learning using Sparse Eigenfunction Bases
We present a new framework for semi-supervised learning with sparse eigenfunction bases of kernel matrices.
Data Skeletonization via Reeb Graphs
While such data is often high-dimensional, it is of interest to approximate it with a low-dimensional or even one-dimensional space, since many important aspects of data are often intrinsically low-dimensional.
Blind Signal Separation in the Presence of Gaussian Noise
In this paper we propose a new algorithm for solving the blind signal separation problem in the presence of additive Gaussian noise, when we are given samples from X=AS+\eta, where \eta is drawn from an unknown, not necessarily spherical n-dimensional Gaussian distribution.
Inverse Density as an Inverse Problem: The Fredholm Equation Approach
In this paper we address the problem of estimating the ratio $\frac{q}{p}$ where $p$ is a density function and $q$ is another density, or, more generally an arbitrary function.
The More, the Merrier: the Blessing of Dimensionality for Learning Large Gaussian Mixtures
The problem of learning this map can be efficiently solved using some recent results on tensor decompositions and Independent Component Analysis (ICA), thus giving an algorithm for recovering the mixture.
Fast Algorithms for Gaussian Noise Invariant Independent Component Analysis
In our paper we develop the first practical algorithm for Independent Component Analysis that is provably invariant under Gaussian noise.
The Hidden Convexity of Spectral Clustering
Geometrically, the proposed algorithms can be interpreted as hidden basis recovery by means of function optimization.
Eigenvectors of Orthogonally Decomposable Functions
It includes influential Machine Learning methods such as cumulant-based FastICA and the tensor power iteration for orthogonally decomposable tensors as special cases.
Learning with Fredholm Kernels
In this paper we propose a framework for supervised and semi-supervised learning based on reformulating the learning problem as a regularized Fredholm integral equation.
Crowd-ML: A Privacy-Preserving Learning Framework for a Crowd of Smart Devices
Smart devices with built-in sensors, computational capabilities, and network connectivity have become increasingly pervasive.
A Pseudo-Euclidean Iteration for Optimal Recovery in Noisy ICA
We propose a new algorithm, PEGI (for pseudo-Euclidean Gradient Iteration), for provable model recovery for ICA with Gaussian noise.
Probabilistic Zero-shot Classification with Semantic Rankings
In this paper we propose a non-metric ranking-based representation of semantic similarity that allows natural aggregation of semantic information from multiple heterogeneous sources.
Beyond Hartigan Consistency: Merge Distortion Metric for Hierarchical Clustering
In this paper we identify two limit properties, separation and minimality, which address both over-segmentation and improper nesting and together imply (but are not implied by) Hartigan consistency.
Learning Privately from Multiparty Data
How can we build an accurate and differentially private global classifier by combining locally-trained classifiers from different parties, without access to any party's private data?
Graphons, mergeons, and so on!
In this work we develop a theory of hierarchical clustering for graphs.
Clustering with Bregman Divergences: an Asymptotic Analysis
Clustering, in particular $k$-means clustering, is a central topic in data analysis.
Diving into the shallows: a computational perspective on large-scale shallow learning
An analysis based on the spectral properties of the kernel demonstrates that only a vanishingly small portion of the function space is reachable after a polynomial number of gradient descent iterations.
Unperturbed: spectral analysis beyond Davis-Kahan
Classical matrix perturbation results, such as Weyl's theorem for eigenvalues and the Davis-Kahan theorem for eigenvectors, are general purpose.
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
We show that there is a critical batch size $m^*$ such that: (a) SGD iteration with mini-batch size $m\leq m^*$ is nearly equivalent to $m$ iterations of mini-batch size $1$ (\emph{linear scaling regime}).
Approximation beats concentration? An approximation view on inference with smooth radial kernels
We analyze eigenvalue decay of kernels operators and matrices, properties of eigenfunctions/eigenvectors and "Fourier" coefficients of functions in the kernel space restricted to a discrete set of data points.
To understand deep learning we need to understand kernel learning
Certain key phenomena of deep learning are manifested similarly in kernel methods in the modern "overfitted" regime.
Fast Interactive Image Retrieval using large-scale unlabeled data
Active learning reduces the number of user interactions by querying the labels of the most informative points and GSSL allows to use abundant unlabeled data along with the limited labeled data provided by the user.
Parametrized Accelerated Methods Free of Condition Number
Analyses of accelerated (momentum-based) gradient descent usually assume bounded condition number to obtain exponential convergence rates.
Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
Finally, this paper suggests a way to explain the phenomenon of adversarial examples, which are seemingly ubiquitous in modern machine learning, and also discusses some connections to kernel machines and random forests in the interpolated regime.
Kernel machines that adapt to GPUs for effective large batch training
In this paper we develop the first analytical framework that extends linear scaling to match the parallel computing capacity of a resource.
Does data interpolation contradict statistical optimality?
We show that learning methods interpolating the training data can achieve optimal rates for the problems of nonparametric regression and prediction with square loss.
Memorization in Overparameterized Autoencoders
The ability of deep neural networks to generalize well in the overparameterized regime has become a subject of significant research interest.
Accelerating SGD with momentum for over-parameterized learning
This is in contrast to the classical results in the deterministic scenario, where the same step size ensures accelerated convergence of the Nesterov's method over optimal gradient descent.
Kernel Machines Beat Deep Neural Networks on Mask-based Single-channel Speech Enhancement
We apply a fast kernel method for mask-based single-channel speech enhancement.
On exponential convergence of SGD in non-convex over-parametrized learning
Large over-parametrized models learned via stochastic gradient descent (SGD) methods have become a key element in modern machine learning.
Reconciling modern machine learning practice and the bias-variance trade-off
This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.
Two models of double descent for weak features
The "double descent" risk curve was proposed to qualitatively describe the out-of-sample prediction accuracy of variably-parameterized machine learning models.
Downsampling leads to Image Memorization in Convolutional Autoencoders
In this paper, we link memorization of images in deep convolutional autoencoders to downsampling through strided convolution.
Overparameterized Neural Networks Can Implement Associative Memory
Identifying computational mechanisms for memorization and retrieval is a long-standing problem at the intersection of machine learning and neuroscience.
Overparameterized Neural Networks Implement Associative Memory
Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience.
Loss landscapes and optimization in over-parameterized non-linear systems and neural networks
The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks.
Classification vs regression in overparameterized regimes: Does the loss function matter?
We compare classification and regression tasks in an overparameterized linear model with Gaussian features.
Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks
We explore several major neural architectures and a range of standard benchmark datasets for NLP, automatic speech recognition (ASR) and computer vision tasks to show that these architectures, with the same hyper-parameter settings as reported in the literature, perform comparably or better when trained with the square loss, even after equalizing computational resources.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Multiple Descent: Design Your Own Generalization Curve
This paper explores the generalization loss of linear regression in variably parameterized families of models, both under-parameterized and over-parameterized.
Linear Convergence of Generalized Mirror Descent with Time-Dependent Mirrors
GMD subsumes popular first order optimization methods including gradient descent, mirror descent, and preconditioned gradient descent methods such as Adagrad.
Linear Convergence and Implicit Regularization of Generalized Mirror Descent with Time-Dependent Mirrors
The following questions are fundamental to understanding the properties of over-parameterization in modern machine learning: (1) Under what conditions and at what rate does training converge to a global minimum?
On the linearity of large non-linear models: when and why the tangent kernel is constant
We show that the transition to linearity of the model and, equivalently, constancy of the (neural) tangent kernel (NTK) result from the scaling properties of the norm of the Hessian matrix of the network as a function of the network width.
Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures
In this paper, we study this "benign overfitting" phenomenon of the maximum margin classifier for linear classification problems.
Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation
In the past decade the mathematical theory of machine learning has lagged far behind the triumphs of deep neural networks on practical challenges.
Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks
Remarkably, taking the width of a neural network to infinity allows for improved computational performance.
Local Quadratic Convergence of Stochastic Gradient Descent with Adaptive Step Size
Establishing a fast rate of convergence for optimization methods is crucial to their applicability in practice.
Benign Overfitting in Two-layer Convolutional Neural Networks
In this paper, we study the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN).
Limitations of Neural Collapse for Understanding Generalization in Deep Learning
We refine the Neural Collapse conjecture into two separate conjectures: collapse on the train set (an optimization property) and collapse on the test distribution (a generalization property).
Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models
Wide neural networks with linear output layer have been shown to be near-linear, and to have near-constant neural tangent kernel (NTK), in a region containing the optimization path of gradient descent.
Wide and Deep Neural Networks Achieve Optimality for Classification
In this work, we identify and construct an explicit set of neural network classifiers that achieve optimality.
Quadratic models for understanding neural network dynamics
While neural networks can be approximated by linear models as their width increases, certain properties of wide neural networks cannot be captured by linear models.
Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their "width" approaches infinity.
On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensions
``Benign overfitting'', the ability of certain algorithms to interpolate noisy training data and yet perform well out-of-sample, has been a topic of considerable recent interest.
A note on Linear Bottleneck networks and their Transition to Multilinearity
In this work we show that linear networks with a bottleneck layer learn bilinear functions of the weights, in a ball of radius $O(1)$ around initialization.
Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting
In this work we argue that while benign overfitting has been instructive and fruitful to study, many real interpolating methods like neural networks do not fit benignly: modest noise in the training set causes nonzero (but non-infinite) excess risk at test time, implying these models are neither benign nor catastrophic but rather fall in an intermediate regime.
A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictors
Remarkably, while the Marchenko-Pastur analysis is far more precise near the interpolation peak, where the number of parameters is just enough to fit the training data, it coincides exactly with the distribution independent bound as the level of overparametrization increases.
Restricted Strong Convexity of Deep Learning Models with Smooth Activations
Second, we introduce a new analysis of optimization based on Restricted Strong Convexity (RSC) which holds as long as the squared norm of the average gradient of predictors is $\Omega(\frac{\text{poly}(L)}{\sqrt{m}})$ for the square loss.
Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features
In recent years neural networks have achieved impressive results on many technological and scientific tasks.
Toward Large Kernel Models
Recent studies indicate that kernel machines can often perform similarly or better than deep neural networks (DNNs) on small datasets.
Cut your Losses with Squentropy
We provide an extensive set of experiments on multi-class classification problems showing that the squentropy loss outperforms both the pure cross entropy and rescaled square losses in terms of the classification accuracy.
On Emergence of Clean-Priority Learning in Early Stopped Neural Networks
This behaviour is believed to be a result of neural networks learning the pattern of clean data first and fitting the noise later in the training, a phenomenon that we refer to as clean-priority learning.
Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
In this paper, we first present an explanation regarding the common occurrence of spikes in the training loss when neural networks are trained with stochastic gradient descent (SGD).
Mechanism of feature learning in convolutional neural networks
We then demonstrate the generality of our result by using the patch-based AGOP to enable deep feature learning in convolutional kernel machines.
More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatory
In our era of enormous neural networks, empirical progress has been driven by the philosophy that more is better.
On the Nystrom Approximation for Preconditioning in Kernel Machines
Scalable algorithms for learning kernel models need to be iterative in nature, but convergence can be slow due to poor conditioning.
Linear Recursive Feature Machines provably recover low-rank matrices
A possible explanation is that common training algorithms for neural networks implicitly perform dimensionality reduction - a process called feature learning.
Unmemorization in Large Language Models via Self-Distillation and Deliberate Imagination
Our results demonstrate the usefulness of this approach across different models and sizes, and also with parameter-efficient fine-tuning, offering a novel pathway to addressing the challenges with private and sensitive data in LLM applications.
Average gradient outer product as a mechanism for deep neural collapse
In this work, we provide substantial evidence that DNC formation occurs primarily through deep feature learning with the average gradient outer product (AGOP).
