Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking

1 code implementation • 24 Mar 2022 • Iñigo Urteaga, Moulay-Zaïdane Draïdia, Tomer Lancewicki, Shahram Khadivi

We propose a multi-armed bandit framework for the sequential selection of TLM pre-training hyperparameters, aimed at optimizing language model performance, in a resource efficient manner.

Bayesian Optimization Decision Making +3