GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

16 Nov 2018Yanping HuangYonglong ChengDehao ChenHyoukJoong LeeJiquan NgiamQuoc V. LeZhifeng Chen

GPipe is a scalable pipeline parallelism library that enables learning of giant deep neural networks. It partitions network layers across accelerators and pipelines execution to achieve high hardware utilization... (read more)

PDF Abstract

Evaluation results from the paper


Task Dataset Model Metric name Metric value Global rank Compare
Image Classification CIFAR-10 GPIPE + transfer learning Percentage correct 99 # 1
Image Classification CIFAR-10 GPIPE + transfer learning Percentage error 1 # 1
Image Classification CIFAR-100 GPIPE Percentage correct 91.3 # 1
Image Classification ImageNet GPIPE Top 1 Accuracy 84.3% # 1
Image Classification ImageNet GPIPE Top 5 Accuracy 97% # 1