Search Results for author:
Found 31 papers, 11 papers with code
Can Neural Network Memorization Be Localized?
Recent efforts at explaining the interplay of memorization and generalization in deep overparametrized networks have posited that neural networks $\textit{memorize}$ "hard" examples in the final few layers of the model.
The Role of Pre-training Data in Transfer Learning
We investigate the impact of pre-training data distribution on the few-shot and full fine-tuning performance using 3 pre-training methods (supervised, contrastive language-image and image-image), 7 pre-training datasets, and 9 downstream datasets.
Leveraging Unlabeled Data to Track Memorization
We empirically show the effectiveness of our metric in tracking memorization on various architectures and datasets and provide theoretical insights into the design of the susceptibility metric.
Layer-Stack Temperature Scaling
Inspired by this, we show that extending temperature scaling across all layers improves both calibration and accuracy.
Teaching Algorithmic Reasoning via In-context Learning
Large language models (LLMs) have shown increasing in-context learning capabilities through scaling up model and data size.
REPAIR: REnormalizing Permuted Activations for Interpolation Repair
In this paper we look into the conjecture of Entezari et al. (2021) which states that if the permutation invariance of neural networks is taken into account, then there is likely no loss barrier to the linear interpolation between SGD solutions.
Understanding the effect of sparsity on neural networks robustness
This paper examines the impact of static sparsity on the robustness of a trained network to weight perturbations, data corruption, and adversarial examples.
Leveraging Unlabeled Data to Predict Out-of-Distribution Performance
Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions that may cause performance drops.
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks
In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them.
Exploring the Limits of Large Scale Pre-training
Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks.
Gradual Domain Adaptation in the Wild: When Intermediate Distributions are Absent
It is shown that under the following two assumptions: (a) access to samples from intermediate distributions, and (b) samples being annotated with the amount of change from the source distribution; self-training can be successfully applied on gradually shifted samples to adapt the model toward the target distribution.
Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent
It has been shown that under the following two assumptions: (a) access to samples from intermediate distributions, and (b) samples being annotated with the amount of change from the source distribution, self-training can be successfully applied on gradually shifted samples to adapt the model toward the target distribution.
The Bootstrap Framework: Generalization Through the Lens of Online Optimization
We propose a new framework for reasoning about generalization in deep learning.
The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
We propose a new framework for reasoning about generalization in deep learning.
What is being transferred in transfer learning?
One desired capability for machines is the ability to transfer their knowledge of one domain to another where data is (usually) scarce.
The intriguing role of module criticality in the generalization of deep networks
We study the phenomenon that some modules of deep neural networks (DNNs) are more critical than others.
Generalization bounds for deep convolutional neural networks
We prove bounds on the generalization error of convolutional networks.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
On the effect of the activation function on the distribution of hidden nodes in a deep network
We analyze the joint probability distribution on the lengths of the vectors of hidden variables in different layers of a fully connected deep network, when the weights and biases are chosen randomly according to Gaussian distributions, and the input is in $\{ -1, 1\}^N$.
The Singular Values of Convolutional Layers
We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation.
Knowledge Completion for Generics using Guided Tensor Factorization
Given a knowledge base or KB containing (noisy) facts about common nouns or generics, such as "all trees produce oxygen" or "some animals live in forests", we consider the problem of inferring additional such facts at a precision similar to that of the starting KB.
Training Input-Output Recurrent Neural Networks through Spectral Methods
We consider the problem of training input-output recurrent neural networks (RNN) for sequence labeling tasks.
Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
We propose a novel algorithm based on tensor decomposition for guaranteed training of two-layer neural networks.
Learning Mixed Membership Community Models in Social Tagging Networks through Tensor Methods
Community detection in graphs has been extensively studied both in theory and in applications.
Score Function Features for Discriminative Learning
In this paper, we consider a novel class of matrix and tensor-valued features, which can be pre-trained using unlabeled samples.
Provable Tensor Methods for Learning Mixtures of Generalized Linear Models
In contrast, we present a tensor decomposition method which is guaranteed to correctly recover the parameters.
Score Function Features for Discriminative Learning: Matrix and Tensor Framework
In this paper, we consider a novel class of matrix and tensor-valued features, which can be pre-trained using unlabeled samples.
Provable Methods for Training Neural Networks with Sparse Connectivity
We provide novel guaranteed approaches for training feedforward neural networks with sparse connectivity.
Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Matrix Decomposition
We first analyze the simple setting, where the optimization problem consists of a loss function and a single regularizer (e. g. sparse optimization), and then extend to the multi-block setting with multiple regularizers and multiple variables (e. g. matrix decomposition into sparse and low rank components).
Statistical Structure Learning, Towards a Robust Smart Grid
We propose a decentralized false data injection detection scheme based on Markov graph of the bus phase angles.
Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Noisy Matrix Decomposition
For sparse optimization, we establish that the modified ADMM method has an optimal convergence rate of $\mathcal{O}(s\log d/T)$, where $s$ is the sparsity level, $d$ is the data dimension and $T$ is the number of steps.
