Amos: An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale
We present Amos, a stochastic gradient-based optimizer designed for training deep neural networks. It can be viewed as an Adam optimizer with theoretically supported, adaptive learning-rate decay and weight decay. A key insight behind Amos is that it leverages model-specific information to determine the initial learning-rate and decaying schedules. When used for pre-training BERT variants and T5, Amos consistently converges faster than the state-of-the-art settings of AdamW, achieving better validation loss within <=70% training steps and time, while requiring <=51% memory for slot variables. Our code is open-sourced at: https://github.com/google-research/jestimator
PDF Abstract
Tasks
ImageNet
MultiNLI
Results from the Paper
Submit
results from this paper
to get state-of-the-art GitHub badges and help the
community compare results to other papers.
Methods
Adafactor •
Adam •
AdamW •
Attention Dropout •
BERT •
BPE •
Dense Connections •
Dropout •
GELU •
GLU •
Inverse Square Root Schedule •
Layer Normalization •
Linear Layer •
Linear Warmup With Linear Decay •
Multi-Head Attention •
Residual Connection •
Scaled Dot-Product Attention •
SentencePiece •
Softmax •
T5 •
Weight Decay •
WordPiece
