ZeRO-Offload: Democratizing Billion-Scale Model Training

18 Jan 2021 • Jie Ren • Samyam Rajbhandari • Reza Yazdani Aminabadi • Olatunji Ruwase • Shuangyan Yang • Minjia Zhang • Dong Li • Yuxiong He

Large-scale model training has been a playing ground for a limited few requiring complex model refactoring and access to prohibitively expensive GPU clusters. ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone... (read more)

PDF Abstract

Results from the Paper

Edit

Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

METHOD	TYPE
🤖 No Methods Found	Help the community by adding them if they're not listed; e.g. Deep Residual Learning for Image Recognition uses ResNet

ZeRO-Offload: Democratizing Billion-Scale Model Training

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods used in the Paper

Add Remove

Edit Social Preview

ZeRO-Offload: Democratizing Billion-Scale Model Training Edit social preview

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods used in the Paper Edit Add Remove

ZeRO-Offload: Democratizing Billion-Scale Model Training

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods used in the Paper

Add Remove