Search Results for author:
Found 14 papers, 10 papers with code
What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions
Large language models (LLMs) are trained on a vast amount of human-written data, but data providers often remain uncredited.
Training Data Attribution via Approximate Unrolled Differentiation
While being computationally efficient compared to unrolling-based approaches, Source is suitable in cases where implicit-differentiation-based approaches struggle, such as in non-converged models and multi-stage training pipelines.
Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective
Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers.
Using Large Language Models for Hyperparameter Optimization
This paper studies using foundational large language models (LLMs) to make decisions during hyperparameter optimization (HPO).
Studying Large Language Model Generalization with Influence Functions
When trying to gain better visibility into a machine learning model in order to understand and mitigate the associated risks, a potentially valuable source of evidence is: which training examples most contribute to a given behavior?
Benchmarking Neural Network Training Algorithms
In order to address these challenges, we introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware, the AlgoPerf: Training Algorithms benchmark.
Efficient Parametric Approximations of Neural Network Function Space Distance
It is often useful to compactly summarize important properties of model parameters and training data so that they can be used later without storing and/or iterating over the entire dataset.
Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve
Variational autoencoders (VAEs) are powerful tools for learning latent representations of data used in a wide range of applications.
If Influence Functions are the Answer, Then What is the Question?
Influence functions efficiently estimate the effect of removing a single training data point on a model's learned parameters.
Amortized Proximal Optimization
Using APO to adapt a structured preconditioning matrix generally results in optimization performance competitive with second-order methods.
Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training objective.
Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
Hyperparameter optimization of neural networks can be elegantly formulated as a bilevel optimization problem.
Eigenvalue Corrected Noisy Natural Gradient
A recently proposed method, noisy natural gradient, is a surprisingly simple method to fit expressive posteriors by adding weight noise to regular natural gradient updates.
Learnable Pooling Methods for Video Classification
We demonstrate our solutions in the "The 2nd YouTube-8M Video Understanding Challenge", by using frame-level video and audio descriptors.
