Variance Reduction in Deep Learning: More Momentum is All You Need

no code implementations • 23 Nov 2021 • Lionel Tondji, Sergii Kashubin, Moustapha Cisse

Variance reduction (VR) techniques have contributed significantly to accelerating learning with massive datasets in the smooth and strongly convex setting (Schmidt et al., 2017; Johnson & Zhang, 2013; Roux et al., 2012).

Data Augmentation Distributed Optimization