Search Results for author:
Found 11 papers, 3 papers with code
Learning and Memorization
In the machine learning research community, it is generally believed that there is a tension between memorization and generalization.
Circuit-Based Intrinsic Methods to Detect Overfitting
By intrinsic methods, we mean methods that rely only on the model and the training data, as opposed to traditional methods (we call them extrinsic methods) that rely on performance on a test set or on bounds from model complexity.
Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
We propose an approach to answering this question based on a hypothesis about the dynamics of gradient descent that we call Coherent Gradients: Gradients from similar examples are similar and so the overall gradient is stronger in certain directions where these reinforce each other.
Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale
The key insight of CGH is that, since the overall gradient for a single step of SGD is the sum of the per-example gradients, it is strongest in directions that reduce the loss on multiple examples if such directions exist.
Making Coherence Out of Nothing At All: Measuring the Evolution of Gradient Alignment
Using $m$-coherence, we study the evolution of alignment of per-example gradients in ResNet and Inception models on ImageNet and several variants with label noise, particularly from the perspective of the recently proposed Coherent Gradients (CG) theory that provides a simple, unified explanation for memorization and generalization [Chatterjee, ICLR 20].
Logic Synthesis Meets Machine Learning: Trading Exactness for Generalization
If the function is incompletely-specified, the implementation has to be true only on the care set.
Making Coherence Out of Nothing At All: Measuring Evolution of Gradient Alignment
Using $m$-coherence, we study the evolution of alignment of per-example gradients in ResNet and EfficientNet models on ImageNet and several variants with label noise, particularly from the perspective of the recently proposed Coherent Gradients (CG) theory that provides a simple, unified explanation for memorization and generalization [Chatterjee, ICLR 20].
Apollo: Transferable Architecture Exploration
We further show that by transferring knowledge between target architectures with different design constraints, Apollo is able to find optimal configurations faster and often with better objective value (up to 25% improvements).
Enabling Binary Neural Network Training on the Edge
The ever-growing computational demands of increasingly complex machine learning models frequently necessitate the use of powerful cloud-based infrastructure for their training.
On the Generalization Mystery in Deep Learning
The generalization mystery in deep learning is the following: Why do over-parameterized neural networks trained with gradient descent (GD) generalize well on real datasets even though they are capable of fitting random datasets of comparable size?
A Closer Look at Hardware-Friendly Weight Quantization
We study the two methods on MobileNetV1 and MobileNetV2 using multiple empirical metrics to identify the sources of performance differences between the two classes, namely, sensitivity to outliers and convergence instability of the quantizer scaling factor.
