Search Results for author:
Found 2 papers, 0 papers with code
Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler
This is not only because there is a complicated correlation between learning rate, batch size, number of training tokens, model size, and other hyperparameters but also because it is prohibitively expensive to perform a hyperparameter search for large language models with Billions or Trillions of parameters.
Diversity Measurement and Subset Selection for Instruction Tuning Datasets
Our experiments demonstrate that the proposed diversity measure in the normalized weight gradient space is correlated with downstream instruction-following performance.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
