Rethinking Learning Rate Tuning in the Era of Large Language Models

1 code implementation • 16 Sep 2023 • Hongpeng Jin, Wenqi Wei, Xuyu Wang, Wenbin Zhang, Yanzhao Wu

Second, we present LRBench++ to benchmark learning rate policies and facilitate learning rate tuning for both traditional DNNs and LLMs.