no code implementations • NeurIPS 2021 • Jakub M. Tarnawski, Deepak Narayanan, Amar Phanishayee
The rapid increase in sizes of state-of-the-art DNN models, and consequently the increase in the compute and memory requirements of model training, has led to the development of many execution schemes such as data parallelism, pipeline model parallelism, tensor (intra-layer) model parallelism, and various memory-saving optimizations.
