Search Results for author: Etai Littwin

Found 13 papers, 1 papers with code

Learning Representation from Neural Fisher Kernel with Low-rank Approximation

no code implementations ICLR 2022 Ruixiang Zhang, Shuangfei Zhai, Etai Littwin, Josh Susskind

We show that the low-rank approximation of NFKs derived from unsupervised generative models and supervised learning models gives rise to high-quality compact representations of data, achieving competitive results on a variety of machine learning tasks.

Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks

no code implementations2 Jul 2021 Shih-Yu Sun, Vimal Thilak, Etai Littwin, Omid Saremi, Joshua M. Susskind

Deep linear networks trained with gradient descent yield low rank solutions, as is typically studied in matrix factorization.

Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics

no code implementations8 May 2021 Greg Yang, Etai Littwin

To facilitate this proof, we develop a graphical notation for Tensor Programs.

Collegial Ensembles

no code implementations NeurIPS 2020 Etai Littwin, Ben Myara, Sima Sabah, Joshua Susskind, Shuangfei Zhai, Oren Golan

Modern neural network performance typically improves as model size increases.

On Infinite-Width Hypernetworks

1 code implementation NeurIPS 2020 Etai Littwin, Tomer Galanti, Lior Wolf, Greg Yang

{\em Hypernetworks} are architectures that produce the weights of a task-specific {\em primary network}.


On Random Kernels of Residual Architectures

no code implementations28 Jan 2020 Etai Littwin, Tomer Galanti, Lior Wolf

We derive finite width and depth corrections for the Neural Tangent Kernel (NTK) of ResNets and DenseNets.

On the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width

no code implementations ICML Workshop Deep_Phenomen 2019 Etai Littwin, Lior Wolf

The Hessian of neural networks can be decomposed into a sum of two matrices: (i) the positive semidefinite generalized Gauss-Newton matrix G, and (ii) the matrix H containing negative eigenvalues.

The Effect of Residual Architecture on the Per-Layer Gradient of Deep Networks

no code implementations25 Sep 2019 Etai Littwin, Lior Wolf

A critical part of the training process of neural networks takes place in the very first gradient steps post initialization.

Regularizing by the Variance of the Activations' Sample-Variances

no code implementations NeurIPS 2018 Etai Littwin, Lior Wolf

Normalization techniques play an important role in supporting efficient and often more effective training of deep neural networks.

The Loss Surface of Residual Networks: Ensembles and the Role of Batch Normalization

no code implementations8 Nov 2016 Etai Littwin, Lior Wolf

Deep Residual Networks present a premium in performance in comparison to conventional networks of the same depth and are trainable at extreme depths.

The Multiverse Loss for Robust Transfer Learning

no code implementations CVPR 2016 Etai Littwin, Lior Wolf

Deep learning techniques are renowned for supporting effective transfer learning.

Transfer Learning

Spherical Embedding of Inlier Silhouette Dissimilarities

no code implementations CVPR 2015 Etai Littwin, Hadar Averbuch-Elor, Daniel Cohen-Or

In this paper, we introduce a spherical embedding technique to position a given set of silhouettes of an object as observed from a set of cameras arbitrarily positioned around the object.

Cannot find the paper you are looking for? You can Submit a new open access paper.