GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

16 Nov 2018Yanping Huang • Yonglong Cheng • Dehao Chen • HyoukJoong Lee • Jiquan Ngiam • Quoc V. Le • Zhifeng Chen

GPipe is a scalable pipeline parallelism library that enables learning of giant deep neural networks. It partitions network layers across accelerators and pipelines execution to achieve high hardware utilization. It leverages recomputation to minimize activation memory usage.

Full paper


Task Dataset Model Metric name Metric value Global rank Compare
Image Classification CIFAR-10 GPIPE + transfer learning Percentage correct 99 # 1
Image Classification CIFAR-100 GPIPE Percentage correct 91.3 # 1
Image Classification ImageNet GPIPE Top 1 Accuracy 84.3% # 1
Image Classification ImageNet GPIPE Top 5 Accuracy 97% # 1