Search Results for author:
Found 6 papers, 4 papers with code
Large Learning Rates Improve Generalization: But How Large Are We Talking About?
Inspired by recent research that recommends starting neural networks training with large learning rates (LRs) to achieve the best generalization, we explore this hypothesis in detail.
Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes
In this work, we investigate the properties of training scale-invariant neural networks directly on the sphere using a fixed ELR.
On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
Training neural networks with batch normalization and weight decay has become a common practice in recent years.
On Power Laws in Deep Ensembles
Ensembles of deep neural networks are known to achieve state-of-the-art performance in uncertainty estimation and lead to accuracy improvement.
MARS: Masked Automatic Ranks Selection in Tensor Decompositions
Tensor decomposition methods have proven effective in various applications, including compression and acceleration of neural networks.
Efficient Language Modeling with Automatic Relevance Determination in Recurrent Neural Networks
Reduction of the number of parameters is one of the most important goals in Deep Learning.
