Search Results for author:
Found 9 papers, 4 papers with code
Benchmarking Neural Network Training Algorithms
In order to address these challenges, we introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware, the AlgoPerf: Training Algorithms benchmark.
Adaptive Gradient Methods at the Edge of Stability
Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning.
A Unifying View on Implicit Bias in Training Linear Neural Networks
For $L$-layer linear tensor networks that are orthogonally decomposable, we show that gradient flow on separable classification finds a stationary point of the $\ell_{2/L}$ max-margin problem in a "transformed" input space defined by the network.
Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale
The key insight of CGH is that, since the overall gradient for a single step of SGD is the sum of the per-example gradients, it is strongest in directions that reduce the loss on multiple examples if such directions exist.
Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks
Our method outperforms BN and other alternatives in a variety of settings for all batch sizes.
Ranked #730 on
Image Classification
on ImageNet
(using extra training data)
The Effect of Network Depth on the Optimization Landscape
It is well-known that deeper neural networks are harder to train than shallower ones.
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process.
Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks
We demonstrate the effectiveness of our algorithm by successfully training large ImageNet models (Inception-V3, Resnet-50, Resnet-101 and Inception-Resnet-V2) with mini-batch sizes of up to 32000 with no loss in validation error relative to current baselines, and no increase in the total number of steps.
An Information-Theoretic Approach to Detecting Changes in Multi-Dimensional Data Streams
In this paper, we take a general, information-theoretic approach to the change detection problem, which works for multidimensional as well as categorical data.
