Search Results for author: Shun-ichi Amari

Found 9 papers, 1 papers with code

When Does Preconditioning Help or Hurt Generalization?

no code implementations ICLR 2021 Shun-ichi Amari, Jimmy Ba, Roger Grosse, Xuechen Li, Atsushi Nitanda, Taiji Suzuki, Denny Wu, Ji Xu

While second order optimizers such as natural gradient descent (NGD) often speed up optimization, their effect on generalization has been called into question.

Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective

no code implementations20 Jan 2020 Shun-ichi Amari

It is known that any target function is realized in a sufficiently small neighborhood of any randomly connected deep network, provided the width (the number of neurons in a layer) is sufficiently large.

Pathological spectra of the Fisher information metric and its variants in deep neural networks

no code implementations14 Oct 2019 Ryo Karakida, Shotaro Akaho, Shun-ichi Amari

The Fisher information matrix (FIM) plays an essential role in statistics and machine learning as a Riemannian metric tensor or a component of the Hessian matrix of loss functions.

The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks

no code implementations NeurIPS 2019 Ryo Karakida, Shotaro Akaho, Shun-ichi Amari

Thus, we can conclude that batch normalization in the last layer significantly contributes to decreasing the sharpness induced by the FIM.

Interpolating between Optimal Transport and MMD using Sinkhorn Divergences

1 code implementation18 Oct 2018 Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouvé, Gabriel Peyré

Comparing probability distributions is a fundamental problem in data sciences.

Statistics Theory Statistics Theory 62

Statistical Neurodynamics of Deep Networks: Geometry of Signal Spaces

no code implementations22 Aug 2018 Shun-ichi Amari, Ryo Karakida, Masafumi Oizumi

The manifold of input signals is embedded in a higher dimensional manifold of the next layer as a curved submanifold, provided the number of neurons is larger than that of inputs.

Fisher Information and Natural Gradient Learning of Random Deep Networks

no code implementations22 Aug 2018 Shun-ichi Amari, Ryo Karakida, Masafumi Oizumi

The natural gradient method uses the steepest descent direction in a Riemannian manifold, so it is effective in learning, avoiding plateaus.

Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach

no code implementations4 Jun 2018 Ryo Karakida, Shotaro Akaho, Shun-ichi Amari

The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs).

Bayesian Robust Tensor Factorization for Incomplete Multiway Data

no code implementations9 Oct 2014 Qibin Zhao, Guoxu Zhou, Liqing Zhang, Andrzej Cichocki, Shun-ichi Amari

We propose a generative model for robust tensor factorization in the presence of both missing data and outliers.

Model Selection Variational Inference

Cannot find the paper you are looking for? You can Submit a new open access paper.