Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)

no code implementations • 24 Sep 2023 • Guo-qing Jiang, Jinlong Liu, Zixiang Ding, Lin Guo, Wei Lin

As models for nature language processing (NLP), computer vision (CV) and recommendation systems (RS) require surging computation, a large number of GPUs/TPUs are paralleled as a large batch (LB) to improve training throughput.

Recommendation Systems