Search Results for author:
Found 13 papers, 1 papers with code
Learning Representation from Neural Fisher Kernel with Low-rank Approximation
We show that the low-rank approximation of NFKs derived from unsupervised generative models and supervised learning models gives rise to high-quality compact representations of data, achieving competitive results on a variety of machine learning tasks.
Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks
Deep linear networks trained with gradient descent yield low rank solutions, as is typically studied in matrix factorization.
Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck.
Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
To facilitate this proof, we develop a graphical notation for Tensor Programs.
On Infinite-Width Hypernetworks
{\em Hypernetworks} are architectures that produce the weights of a task-specific {\em primary network}.
On Random Kernels of Residual Architectures
We derive finite width and depth corrections for the Neural Tangent Kernel (NTK) of ResNets and DenseNets.
On the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width
The Hessian of neural networks can be decomposed into a sum of two matrices: (i) the positive semidefinite generalized Gauss-Newton matrix G, and (ii) the matrix H containing negative eigenvalues.
The Effect of Residual Architecture on the Per-Layer Gradient of Deep Networks
A critical part of the training process of neural networks takes place in the very first gradient steps post initialization.
Regularizing by the Variance of the Activations' Sample-Variances
Normalization techniques play an important role in supporting efficient and often more effective training of deep neural networks.
The Loss Surface of Residual Networks: Ensembles and the Role of Batch Normalization
Deep Residual Networks present a premium in performance in comparison to conventional networks of the same depth and are trainable at extreme depths.
The Multiverse Loss for Robust Transfer Learning
Deep learning techniques are renowned for supporting effective transfer learning.
Spherical Embedding of Inlier Silhouette Dissimilarities
In this paper, we introduce a spherical embedding technique to position a given set of silhouettes of an object as observed from a set of cameras arbitrarily positioned around the object.
