A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs

1 code implementation • 22 May 2023 • Siddharth Singh, Prajwal Singhania, Aditya K. Ranjan, Zack Sating, Abhinav Bhatele

This 4D approach is a hybrid of 3D tensor and data parallelism, and is implemented in the AxoNN framework.