Search Results for author:
Found 17 papers, 10 papers with code
TaskSet: A Dataset of Optimization Tasks
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers.
Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
We present Brax, an open source library for rigid body simulation with a focus on performance and parallelism on accelerators, written in JAX.
Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
That useful models can arise out of the messy and slow optimization process of evolution suggests that forward-predictive modeling can arise as a side-effect of optimization under the right circumstances.
Model-based Reinforcement Learning
reinforcement-learning
+1
Gradients are Not All You Need
Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades.
Practical tradeoffs between memory, compute, and performance in learned optimizers
We further leverage our analysis to construct a learned optimizer that is both faster and more memory efficient than previous work.
VeLO: Training Versatile Learned Optimizers by Scaling Up
While deep learning models have replaced hand-designed features across many domains, these models are still trained with hand-designed optimizers.
Relaxation dynamics of the toric code in contact with a thermal reservoir: Finite-size scaling in a low temperature regime
In contrast to the size-independent bound predicted for the toric code in the thermodynamic limit, we identify a low-temperature regime on finite lattices below a size-dependent crossover temperature with nontrivial finite-size and temperature scaling of the relaxation time.
Statistical Mechanics Quantum Physics
Engineering autonomous error correction in stabilizer codes at finite temperature
We present an error correcting protocol that enhances the lifetime of stabilizer code based qubits which are susceptible to the creation of pairs of localized defects (due to string-like error operators) at finite temperature, such as the toric code.
Quantum Physics Statistical Mechanics
Understanding and correcting pathologies in the training of learned optimizers
Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks.
Topology and Geometry of Half-Rectified Network Optimization
Our theoretical work quantifies and formalizes two important \emph{folklore} facts: (i) the landscape of deep linear networks has a radically different topology from that of deep half-rectified ones, and (ii) that the energy landscape in the non-linear case is fundamentally controlled by the interplay between the smoothness of the data distribution and model over-parametrization.
Using a thousand optimization tasks to learn hyperparameter search strategies
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers.
Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
In this work we focus on general-purpose learned optimizers capable of training a wide variety of problems with no user-specified hyperparameters.
Overcoming barriers to the training of effective learned optimizers
In this work we focus on general-purpose learned optimizers capable of training a wide variety of problems with no user-specified hyperparameters.
Training Learned Optimizers with Randomly Initialized Learned Optimizers
We show that a population of randomly initialized learned optimizers can be used to train themselves from scratch in an online fashion, without resorting to a hand designed optimizer in any part of the process.
Transformer-Based Learned Optimization
We propose a new approach to learned optimization where we represent the computation of an optimizer's update step using a neural network.
Improving Large Language Model Fine-tuning for Solving Math Problems
With these methods, we present a thorough empirical study on a series of PaLM 2 models and find: (1) The quality and style of the step-by-step solutions used for fine-tuning can make a significant impact on the model performance; (2) While solution re-ranking and majority voting are both effective for improving the model performance when used separately, they can also be used together for an even greater performance boost; (3) Multi-task fine-tuning that sequentially separates the solution generation and evaluation tasks can offer improved performance compared with the solution fine-tuning baseline.
Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?
We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment.
