Search Results for author:
Found 26 papers, 10 papers with code
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
A central problem in machine learning involves modeling complex data-sets using highly flexible families of probability distributions in which learning, sampling, inference, and evaluation are still analytically or computationally tractable.
Deep Learning Models of the Retinal Response to Natural Scenes
Here we demonstrate that deep convolutional neural networks (CNNs) capture retinal responses to natural scenes nearly to within the variability of a cell's response, and are markedly more accurate than linear-nonlinear (LN) models and Generalized Linear Models (GLMs).
Learned Optimizers that Scale and Generalize
Two of the primary barriers to its adoption are an inability to scale to larger problems and a limited ability to generalize to new tasks.
Recurrent Segmentation for Variable Computational Budgets
Importantly, the RNN may be deployed across a range of computational budgets by merely running the model for a variable number of iterations.
Meta-Learning Update Rules for Unsupervised Representation Learning
Specifically, we target semi-supervised classification performance, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations useful for this task.
Guided evolutionary strategies: Augmenting random search with surrogate gradients
We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along with random search.
Understanding and correcting pathologies in the training of learned optimizers
Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks.
Learning Unsupervised Learning Rules
Here, our desired task (meta-objective) is the performance of the representation on semi-supervised classification, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations that perform well under this meta-objective.
Guided Evolutionary Strategies: Escaping the curse of dimensionality in random search
This arises when an approximate gradient is easier to compute than the full gradient (e. g. in meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate (e. g. in certain reinforcement learning applications or training networks with discrete variables).
Learned optimizers that outperform on wall-clock and validation loss
We demonstrate these results on problems where our learned optimizer trains convolutional networks in a fifth of the wall-clock time compared to tuned first-order methods, and with an improvement
Line attractor dynamics in recurrent networks for sentiment classification
Recurrent neural networks (RNNs) are a powerful tool for modeling sequential data.
Using learned optimizers to make models robust to input noise
State-of-the art vision models can achieve superhuman performance on image classification tasks when testing and training data come from the same distribution.
Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics
In this work, we use tools from dynamical systems analysis to reverse engineer recurrent networks trained to perform sentiment classification, a foundational natural language processing task.
Universality and individuality in neural dynamics across large populations of recurrent networks
To address these foundational questions, we study populations of thousands of networks, with commonly used RNN architectures, trained to solve neuroscientifically motivated tasks and characterize their nonlinear dynamics.
Revealing computational mechanisms of retinal prediction via model reduction
Thus overall, this work not only yields insights into the computational mechanisms underlying the striking predictive capabilities of the retina, but also places the framework of deep networks as neuroscientific models on firmer theoretical foundations, by providing a new roadmap to go beyond comparing neural representations to extracting and understand computational mechanisms.
From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction
Thus overall, this work not only yields insights into the computational mechanisms underlying the striking predictive capabilities of the retina, but also places the framework of deep networks as neuroscientific models on firmer theoretical foundations, by providing a new roadmap to go beyond comparing neural representations to extracting and understand computational mechanisms.
Using a thousand optimization tasks to learn hyperparameter search strategies
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers.
How recurrent networks implement contextual processing in sentiment analysis
Here, we propose general methods for reverse engineering recurrent neural networks (RNNs) to identify and elucidate contextual processing.
Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
In this work we focus on general-purpose learned optimizers capable of training a wide variety of problems with no user-specified hyperparameters.
The geometry of integration in text classification RNNs
Using tools from dynamical systems analysis, we study recurrent networks trained on a battery of both natural and synthetic text classification tasks.
Reverse engineering learned optimizers reveals known and novel mechanisms
Learned optimizers are algorithms that can themselves be trained to solve optimization problems.
TaskSet: A Dataset of Optimization Tasks
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers.
Overcoming barriers to the training of effective learned optimizers
In this work we focus on general-purpose learned optimizers capable of training a wide variety of problems with no user-specified hyperparameters.
Training Learned Optimizers with Randomly Initialized Learned Optimizers
We show that a population of randomly initialized learned optimizers can be used to train themselves from scratch in an online fashion, without resorting to a hand designed optimizer in any part of the process.
Understanding How Encoder-Decoder Architectures Attend
Moreover, how these mechanisms vary depending on the particular architecture used for the encoder and decoder (recurrent, feed-forward, etc.)
Practical tradeoffs between memory, compute, and performance in learned optimizers
We further leverage our analysis to construct a learned optimizer that is both faster and more memory efficient than previous work.
