Advanced optimization algorithms such as Newton method and AdaGrad benefit
from second order derivative or second order statistics to achieve better
descent directions and faster convergence rates. At their heart, such
algorithms need to compute the inverse or inverse square root of a matrix whose
size is quadratic of the dimensionality of the search space...
dimensional search spaces, the matrix inversion or inversion of square root
becomes overwhelming which in turn demands for approximate methods. In this
work, we propose a new matrix approximation method which divides a matrix into
blocks and represents each block by one or two numbers. The method allows
efficient computation of matrix inverse and inverse square root. We apply our
method to AdaGrad in training deep neural networks. Experiments show
encouraging results compared to the diagonal approximation.