SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix

no code implementations • 5 Mar 2024 • Gayathri C, Mrinmay Sen, A. K. Qin, Raghu Kishore N, Yen-Wei Chen, Balasubramanian Raman

This paper introduces a new stochastic optimization method based on the regularized Fisher information matrix (FIM), named SOFIM, which can efficiently utilize the FIM to approximate the Hessian matrix for finding Newton's gradient update in large-scale stochastic optimization of machine learning models.

Image Classification Stochastic Optimization