Search Results for author:
Found 6 papers, 2 papers with code
Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning
Vanilla SFT (i. e., no PPO, DPO, or RL methods) on data generated from Instruct-SkillMix leads to strong gains on instruction following benchmarks such as AlpacaEval 2. 0, MT-Bench, and WildBench.
Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models
The paper develops a methodology for (a) designing and administering such an evaluation, and (b) automatic grading (plus spot-checking by humans) of the results using GPT-4 as well as the open LLaMA-2 70B model.
Disentangling the Mechanisms Behind Implicit Regularization in SGD
A number of competing hypotheses have been proposed to explain why small-batch Stochastic Gradient Descent (SGD)leads to improved generalization over the full-batch regime, with recent work crediting the implicit regularization of various quantities throughout training.
On the Maximum Hessian Eigenvalue and Generalization
The mechanisms by which certain training interventions, such as increasing learning rates and applying batch normalization, improve the generalization of deep networks remains a mystery.
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability.
Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?
For a standard convolutional neural network, optimizing over the input pixels to maximize the score of some target class will generally produce a grainy-looking version of the original image.
