Search Results for author:
Found 8 papers, 4 papers with code
Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization
The early phase of training a deep neural network has a dramatic effect on the local curvature of the loss function.
Differences between human and machine perception in medical diagnosis
We compare the two with respect to their robustness to Gaussian low-pass filtering, performing a subgroup analysis on microcalcifications and soft tissue lesions.
Can Wikipedia Categories Improve Masked Language Model Pretraining?
Pretrained language models have obtained impressive results for a large set of natural language understanding tasks.
We Should at Least Be Able to Design Molecules That Dock Well
To close this gap, we propose a benchmark based on docking, a popular computational method for assessing molecule binding to a protein.
The Break-Even Point on Optimization Trajectories of Deep Neural Networks
We argue for the existence of the "break-even" point on this trajectory, beyond which the curvature of the loss surface and noise in the gradient are implicitly regularized by SGD.
Large Scale Structure of Neural Network Loss Landscapes
There are many surprising and perhaps counter-intuitive properties of optimization of deep neural networks.
Parameter-Efficient Transfer Learning for NLP
On GLUE, we attain within 0. 4% of the performance of full fine-tuning, adding only 3. 6% parameters per task.
Ranked #4 on
Image Classification
on OmniBenchmark
(using extra training data)
Stiffness: A New Perspective on Generalization in Neural Networks
In particular, we study how stiffness depends on 1) class membership, 2) distance between data points in the input space, 3) training iteration, and 4) learning rate.
