Search Results for author:
Found 5 papers, 2 papers with code
Revisiting BFloat16 Training
State-of-the-art generic low-precision training algorithms use a mix of 16-bit and 32-bit precision, creating the folklore that 16-bit hardware compute units alone are not enough to maximize model accuracy.
Understanding the Downstream Instability of Word Embeddings
To theoretically explain this tradeoff, we introduce a new measure of embedding instability---the eigenspace instability measure---which we prove bounds the disagreement in downstream predictions introduced by the change in word embeddings.
PipeMare: Asynchronous Pipeline Parallel DNN Training
Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware utilization.
Low-Memory Neural Network Training: A Technical Report
In this paper we study a fundamental question: How much memory is actually needed to train a neural network?
High-Accuracy Low-Precision Training
Low-precision computation is often used to lower the time and energy cost of machine learning, and recently hardware accelerators have been developed to support it.
