Scaling Distributed Training with Adaptive Summation

no code implementations • 4 Jun 2020 • Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Olli Saarikivi, Tianju Xu, Vadim Eksarevskiy, Jaliya Ekanayake, Emad Barsoum

This paper introduces a novel method to combine gradients called Adasum (for adaptive sum) that converges faster than prior work.

16k