Search Results for author:
Found 9 papers, 0 papers with code
Anchor function: a type of benchmark functions for studying language models
However, language model research faces significant challenges, especially for academic research groups with constrained resources.
Optimistic Estimate Uncovers the Potential of Nonlinear Models
We propose an optimistic estimate to evaluate the best possible fitting performance of nonlinear models.
Stochastic Modified Equations and Dynamics of Dropout Algorithm
In order to investigate the underlying mechanism by which dropout facilitates the identification of flatter minima, we study the noise structure of the derived stochastic modified equation for dropout.
Loss Spike in Training Neural Networks
In this work, we study the mechanism underlying loss spikes observed during neural network training.
Linear Stability Hypothesis and Rank Stratification for Nonlinear Models
By these results, model rank of a target function predicts a minimal training data size for its successful recovery.
Implicit regularization of dropout
Secondly, we experimentally find that the training with dropout leads to the neural network with a flatter minimum compared with standard gradient descent training, and the implicit regularization is the key to finding flat solutions.
Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
We prove a general Embedding Principle of loss landscape of deep neural networks (NNs) that unravels a hierarchical structure of the loss landscape of NNs, i. e., loss landscape of an NN contains all critical points of all the narrower NNs.
Dropout in Training Neural Networks: Flatness of Solution and Noise Structure
It is important to understand how the popular regularization method dropout helps the neural network training find a good generalization solution.
Embedding Principle of Loss Landscape of Deep Neural Networks
Understanding the structure of loss landscape of deep neural networks (DNNs)is obviously important.
