Search Results for author:
Found 7 papers, 3 papers with code
PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction
The key insight of our proposal is that we employ a weight prediction strategy in the forward pass to ensure that each mini-batch uses consistent and staleness-free weights to compute the forward pass.
AdaPlus: Integrating Nesterov Momentum and Precise Stepsize Adjustment on AdamW Basis
This paper proposes an efficient optimizer called AdaPlus which integrates Nesterov momentum and precise stepsize adjustment on AdamW basis.
XGrad: Boosting Gradient-Based Optimizers With Weight Prediction
the future weights to update the DNN parameters, making the gradient-based optimizer achieve better convergence and generalization compared to the original optimizer without weight prediction.
Weight Prediction Boosts the Convergence of AdamW
In this paper, we introduce weight prediction into the AdamW optimizer to boost its convergence when training the deep neural network (DNN) models.
XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
It allows the overlapping of the pipelines of multiple micro-batches, including those belonging to different mini-batches.
Non-ergodic Convergence Analysis of Heavy-Ball Algorithms
For objective functions satisfying a relaxed strongly convex condition, the linear convergence is established under weaker assumptions on the step size and inertial parameter than made in the existing literature.
An Efficient ADMM-Based Algorithm to Nonconvex Penalized Support Vector Machines
Support vector machines (SVMs) with sparsity-inducing nonconvex penalties have received considerable attentions for the characteristics of automatic classification and variable selection.
