Search Results for author:
Found 10 papers, 3 papers with code
Stepping on the Edge: Curvature Aware Learning Rate Tuners
These models break the stabilization of the sharpness, which we explain using a simplified model of the joint dynamics of the learning rate and the curvature.
Discovering faster matrix multiplication algorithms with reinforcement learning
Particularly relevant is the case of 4 × 4 matrices in a finite field, where AlphaTensor’s algorithm improves on Strassen’s two-level algorithm for the first time, to our knowledge, since its discovery 50 years ago2.
Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
Using an extended and formalized version of the Q/C map analysis of Poole et al. (2016), along with Neural Tangent Kernel theory, we identify the main pathologies present in deep networks that prevent them from training fast and generalizing to unseen data, and show how these can be avoided by carefully controlling the "shape" of the network's initialization-time kernel function.
Gradient Forward-Propagation for Large-Scale Temporal Video Modelling
How can neural networks be trained on large-volume temporal data efficiently?
Sideways: Depth-Parallel Training of Video Models
We propose Sideways, an approximate backpropagation scheme for training video models.
Verification of Non-Linear Specifications for Neural Networks
We show that a number of important properties of interest can be modeled within this class, including conservation of energy in a learned dynamics model of a physical system; semantic consistency of a classifier's output labels under adversarial perturbations and bounding errors in a system that predicts the summation of handwritten digits.
Distilling Policy Distillation
The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning.
Strength in Numbers: Trading-off Robustness and Computation via Adversarially-Trained Ensembles
We show that increasing the number of parameters in adversarially-trained models increases their robustness, and in particular that ensembling smaller models while adversarially training the entire ensemble as a single model is a more efficient way of spending said budget than simply using a larger single model.
Local minima in training of neural networks
Given that deep networks are highly nonlinear systems optimized by local gradient methods, why do they not seem to be affected by bad local minima?
Grouped Orthogonal Matching Pursuit for Variable Selection and Prediction
We consider the problem of variable group selection for least squares regression, namely, that of selecting groups of variables for best regression performance, leveraging and adhering to a natural grouping structure within the explanatory variables.
