Search Results for author:
Found 10 papers, 7 papers with code
Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data
As a step towards answering this question, we study inductive out-of-context reasoning (OOCR), a type of generalization in which LLMs infer latent information from evidence distributed across training documents and apply it to downstream tasks without in-context learning.
LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language
Machine learning practitioners often face significant challenges in formally integrating their prior knowledge and beliefs into predictive models, limiting the potential for nuanced and context-aware analyses.
Order Matters in the Presence of Dataset Imbalance for Multilingual Learning
In this paper, we empirically study the optimization dynamics of multi-task learning, particularly focusing on those that govern a collection of tasks with significant data imbalance.
Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such as the curvature can help de-sensitize common hyperparameters.
Gradient Estimation with Stochastic Softmax Tricks
The Gumbel-Max trick is the basis of many relaxed gradient estimators.
On Empirical Comparisons of Optimizers for Deep Learning
In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent.
Faster Neural Network Training with Data Echoing
In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training.
Guided Evolutionary Strategies: Escaping the curse of dimensionality in random search
This arises when an approximate gradient is easier to compute than the full gradient (e. g. in meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate (e. g. in certain reinforcement learning applications or training networks with discrete variables).
Guided evolutionary strategies: Augmenting random search with surrogate gradients
We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along with random search.
Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
Gradient-based optimization is the foundation of deep learning and reinforcement learning.
