Search Results for author:
Found 60 papers, 10 papers with code
Cut your Losses with Squentropy
We provide an extensive set of experiments on multi-class classification problems showing that the squentropy loss outperforms both the pure cross entropy and rescaled square losses in terms of the classification accuracy.
Toward Large Kernel Models
Recent studies indicate that kernel machines can often perform similarly or better than deep neural networks (DNNs) on small datasets.
Feature learning in neural networks and kernel machines that recursively learn features
In this work, we isolate a key mechanism driving feature learning in fully connected neural networks by connecting neural feature learning to a statistical estimator known as average gradient outer product.
Restricted Strong Convexity of Deep Learning Models with Smooth Activations
Second, we introduce a new analysis of optimization based on Restricted Strong Convexity (RSC) which holds as long as the squared norm of the average gradient of predictors is $\Omega(\frac{\text{poly}(L)}{\sqrt{m}})$ for the square loss.
A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictors
Remarkably, while the Marchenko-Pastur analysis is far more precise near the interpolation peak, where the number of parameters is just enough to fit the training data, in settings of most practical interest it differs from the distribution independent bound by only a modest multiplicative constant.
Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting
In this work we argue that while benign overfitting has been instructive and fruitful to study, many real interpolating methods like neural networks do not fit benignly: modest noise in the training set causes nonzero (but non-infinite) excess risk at test time, implying these models are neither benign nor catastrophic but rather fall in an intermediate regime.
A note on Linear Bottleneck networks and their Transition to Multilinearity
In this work we show that linear networks with a bottleneck layer learn bilinear functions of the weights, in a ball of radius $O(1)$ around initialization.
Kernel Ridgeless Regression is Inconsistent in Low Dimensions
We show that kernel interpolation for a large class of shift-invariant kernels is inconsistent in fixed dimension, even with bandwidth adaptive to the training set.
Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their "width" approaches infinity.
Quadratic models for understanding neural network dynamics
In this work, we propose using a quadratic model as a tool for understanding properties of wide neural networks in both optimization and generalization.
Wide and Deep Neural Networks Achieve Optimality for Classification
In this work, we identify and construct an explicit set of neural network classifiers that achieve optimality.
Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models
Wide neural networks with linear output layer have been shown to be near-linear, and to have near-constant neural tangent kernel (NTK), in a region containing the optimization path of gradient descent.
Limitations of Neural Collapse for Understanding Generalization in Deep Learning
We refine the Neural Collapse conjecture into two separate conjectures: collapse on the train set (an optimization property) and collapse on the test distribution (a generalization property).
Benign Overfitting in Two-layer Convolutional Neural Networks
In this paper, we study the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN).
Local Quadratic Convergence of Stochastic Gradient Descent with Adaptive Step Size
Establishing a fast rate of convergence for optimization methods is crucial to their applicability in practice.
Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks
Remarkably, taking the width of a neural network to infinity allows for improved computational performance.
Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation
In the past decade the mathematical theory of machine learning has lagged far behind the triumphs of deep neural networks on practical challenges.
Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures
In this paper, we study this "benign overfitting" phenomenon of the maximum margin classifier for linear classification problems.
On the linearity of large non-linear models: when and why the tangent kernel is constant
We show that the transition to linearity of the model and, equivalently, constancy of the (neural) tangent kernel (NTK) result from the scaling properties of the norm of the Hessian matrix of the network as a function of the network width.
Linear Convergence and Implicit Regularization of Generalized Mirror Descent with Time-Dependent Mirrors
The following questions are fundamental to understanding the properties of over-parameterization in modern machine learning: (1) Under what conditions and at what rate does training converge to a global minimum?
Linear Convergence of Generalized Mirror Descent with Time-Dependent Mirrors
GMD subsumes popular first order optimization methods including gradient descent, mirror descent, and preconditioned gradient descent methods such as Adagrad.
Multiple Descent: Design Your Own Generalization Curve
This paper explores the generalization loss of linear regression in variably parameterized families of models, both under-parameterized and over-parameterized.
Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks
We explore several major neural architectures and a range of standard benchmark datasets for NLP, automatic speech recognition (ASR) and computer vision tasks to show that these architectures, with the same hyper-parameter settings as reported in the literature, perform comparably or better when trained with the square loss, even after equalizing computational resources.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Classification vs regression in overparameterized regimes: Does the loss function matter?
We compare classification and regression tasks in an overparameterized linear model with Gaussian features.
Loss landscapes and optimization in over-parameterized non-linear systems and neural networks
The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks.
Overparameterized Neural Networks Implement Associative Memory
Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience.
Overparameterized Neural Networks Can Implement Associative Memory
Identifying computational mechanisms for memorization and retrieval is a long-standing problem at the intersection of machine learning and neuroscience.
Downsampling leads to Image Memorization in Convolutional Autoencoders
In this paper, we link memorization of images in deep convolutional autoencoders to downsampling through strided convolution.
Two models of double descent for weak features
The "double descent" risk curve was proposed to qualitatively describe the out-of-sample prediction accuracy of variably-parameterized machine learning models.
Reconciling modern machine learning practice and the bias-variance trade-off
This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.
Kernel Machines Beat Deep Neural Networks on Mask-based Single-channel Speech Enhancement
We apply a fast kernel method for mask-based single-channel speech enhancement.
On exponential convergence of SGD in non-convex over-parametrized learning
Large over-parametrized models learned via stochastic gradient descent (SGD) methods have become a key element in modern machine learning.
Accelerating SGD with momentum for over-parameterized learning
This is in contrast to the classical results in the deterministic scenario, where the same step size ensures accelerated convergence of the Nesterov's method over optimal gradient descent.
Memorization in Overparameterized Autoencoders
The ability of deep neural networks to generalize well in the overparameterized regime has become a subject of significant research interest.
Does data interpolation contradict statistical optimality?
We show that learning methods interpolating the training data can achieve optimal rates for the problems of nonparametric regression and prediction with square loss.
Kernel machines that adapt to GPUs for effective large batch training
In this paper we develop the first analytical framework that extends linear scaling to match the parallel computing capacity of a resource.
Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
Finally, this paper suggests a way to explain the phenomenon of adversarial examples, which are seemingly ubiquitous in modern machine learning, and also discusses some connections to kernel machines and random forests in the interpolated regime.
Parametrized Accelerated Methods Free of Condition Number
Analyses of accelerated (momentum-based) gradient descent usually assume bounded condition number to obtain exponential convergence rates.
Fast Interactive Image Retrieval using large-scale unlabeled data
Active learning reduces the number of user interactions by querying the labels of the most informative points and GSSL allows to use abundant unlabeled data along with the limited labeled data provided by the user.
To understand deep learning we need to understand kernel learning
Certain key phenomena of deep learning are manifested similarly in kernel methods in the modern "overfitted" regime.
Approximation beats concentration? An approximation view on inference with smooth radial kernels
We analyze eigenvalue decay of kernels operators and matrices, properties of eigenfunctions/eigenvectors and "Fourier" coefficients of functions in the kernel space restricted to a discrete set of data points.
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
We show that there is a critical batch size $m^*$ such that: (a) SGD iteration with mini-batch size $m\leq m^*$ is nearly equivalent to $m$ iterations of mini-batch size $1$ (\emph{linear scaling regime}).
Unperturbed: spectral analysis beyond Davis-Kahan
Classical matrix perturbation results, such as Weyl's theorem for eigenvalues and the Davis-Kahan theorem for eigenvectors, are general purpose.
Diving into the shallows: a computational perspective on large-scale shallow learning
An analysis based on the spectral properties of the kernel demonstrates that only a vanishingly small portion of the function space is reachable after a polynomial number of gradient descent iterations.
Clustering with Bregman Divergences: an Asymptotic Analysis
Clustering, in particular $k$-means clustering, is a central topic in data analysis.
Graphons, mergeons, and so on!
In this work we develop a theory of hierarchical clustering for graphs.
Learning Privately from Multiparty Data
How can we build an accurate and differentially private global classifier by combining locally-trained classifiers from different parties, without access to any party's private data?
Beyond Hartigan Consistency: Merge Distortion Metric for Hierarchical Clustering
In this paper we identify two limit properties, separation and minimality, which address both over-segmentation and improper nesting and together imply (but are not implied by) Hartigan consistency.
Probabilistic Zero-shot Classification with Semantic Rankings
In this paper we propose a non-metric ranking-based representation of semantic similarity that allows natural aggregation of semantic information from multiple heterogeneous sources.
A Pseudo-Euclidean Iteration for Optimal Recovery in Noisy ICA
We propose a new algorithm, PEGI (for pseudo-Euclidean Gradient Iteration), for provable model recovery for ICA with Gaussian noise.
Crowd-ML: A Privacy-Preserving Learning Framework for a Crowd of Smart Devices
Smart devices with built-in sensors, computational capabilities, and network connectivity have become increasingly pervasive.
Learning with Fredholm Kernels
In this paper we propose a framework for supervised and semi-supervised learning based on reformulating the learning problem as a regularized Fredholm integral equation.
Eigenvectors of Orthogonally Decomposable Functions
It includes influential Machine Learning methods such as cumulant-based FastICA and the tensor power iteration for orthogonally decomposable tensors as special cases.
The Hidden Convexity of Spectral Clustering
Geometrically, the proposed algorithms can be interpreted as hidden basis recovery by means of function optimization.
Fast Algorithms for Gaussian Noise Invariant Independent Component Analysis
In our paper we develop the first practical algorithm for Independent Component Analysis that is provably invariant under Gaussian noise.
The More, the Merrier: the Blessing of Dimensionality for Learning Large Gaussian Mixtures
The problem of learning this map can be efficiently solved using some recent results on tensor decompositions and Independent Component Analysis (ICA), thus giving an algorithm for recovering the mixture.
Inverse Density as an Inverse Problem: The Fredholm Equation Approach
In this paper we address the problem of estimating the ratio $\frac{q}{p}$ where $p$ is a density function and $q$ is another density, or, more generally an arbitrary function.
Blind Signal Separation in the Presence of Gaussian Noise
In this paper we propose a new algorithm for solving the blind signal separation problem in the presence of additive Gaussian noise, when we are given samples from X=AS+\eta, where \eta is drawn from an unknown, not necessarily spherical n-dimensional Gaussian distribution.
Data Skeletonization via Reeb Graphs
While such data is often high-dimensional, it is of interest to approximate it with a low-dimensional or even one-dimensional space, since many important aspects of data are often intrinsically low-dimensional.
Semi-supervised Learning using Sparse Eigenfunction Bases
We present a new framework for semi-supervised learning with sparse eigenfunction bases of kernel matrices.
