Search Results for author:
Found 8 papers, 5 papers with code
Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models
Recent work computes non-vacuous compression-based generalization bounds for LLMs, but these bounds are vacuous for large models at the billion-parameter scale.
Non-Vacuous Generalization Bounds for Large Language Models
Modern language models can contain billions of parameters, raising the question of whether they can generalize beyond the training data or simply parrot their training corpora.
PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization
While there has been progress in developing non-vacuous generalization bounds for deep neural networks, these bounds tend to be uninformative about why deep learning works.
Bayesian Model Selection, the Marginal Likelihood, and Generalization
We provide a partial remedy through a conditional marginal likelihood, which we show is more aligned with generalization, and practically valuable for large-scale hyperparameter learning, such as in deep kernel learning.
Adaptive First- and Second-Order Algorithms for Large-Scale Machine Learning
In this paper, we consider both first- and second-order techniques to address continuous optimization problems arising in machine learning.
Dangers of Bayesian Model Averaging under Covariate Shift
Approximate Bayesian inference for neural networks is considered a robust alternative to standard training, often providing good performance on out-of-distribution data.
Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
In this paper, we show that there are mode-connecting simplicial complexes that form multi-dimensional manifolds of low loss, connecting many independently trained models.
Stochastic Damped L-BFGS with Controlled Norm of the Hessian Approximation
We propose a new stochastic variance-reduced damped L-BFGS algorithm, where we leverage estimates of bounds on the largest and smallest eigenvalues of the Hessian approximation to balance its quality and conditioning.
