Search Results for author:
Found 17 papers, 8 papers with code
Simplifying Neural Network Training Under Class Imbalance
Real-world datasets are often highly class-imbalanced, which can adversely impact the performance of deep learning models.
Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
Most interpretability research in NLP focuses on understanding the behavior and features of a fully trained model.
Variance-Covariance Regularization Improves Representation Learning
In summary, VCReg offers a universally applicable regularization framework that significantly advances transfer learning and highlights the connection between gradient starvation, neural collapse, and feature transferability.
Reverse Engineering Self-Supervised Learning
Self-supervised learning (SSL) is a powerful tool in machine learning, but understanding the learned representations and their underlying mechanisms remains a challenge.
To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review
Information theory, and notably the information bottleneck principle, has been pivotal in shaping deep neural networks.
An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization
In this paper, we provide an information-theoretic perspective on Variance-Invariance-Covariance Regularization (VICReg) for self-supervised learning.
How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization
Despite the clear performance benefits of data augmentations, little is known about why they are so effective.
What Do We Maximize in Self-Supervised Learning?
In this paper, we examine self-supervised learning methods, particularly VICReg, to provide an information-theoretical understanding of their construction.
Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors
Deep learning is increasingly moving towards a transfer learning paradigm whereby large foundation models are fine-tuned on downstream tasks, starting from an initialization learned on the source task.
Information Flow in Deep Neural Networks
Then, we propose using the Information Bottleneck (IB) theory to explain deep learning systems.
Tabular Data: Deep Learning is Not All You Need
A key element in solving real-life data science problems is selecting the types of models to use.
Spatial-Temporal Convolutional Network for Spread Prediction of COVID-19
In this work we present a spatial-temporal convolutional neural network for predicting future COVID-19 related symptoms severity among a population, per region, given its past reported symptoms.
The Dual Information Bottleneck
The Information Bottleneck (IB) framework is a general characterization of optimal representations obtained using a principled approach for balancing accuracy and complexity.
Information in Infinite Ensembles of Infinitely-Wide Neural Networks
In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks.
REPRESENTATION COMPRESSION AND GENERALIZATION IN DEEP NEURAL NETWORKS
Specifically, we show that the training of the network is characterized by a rapid increase in the mutual information (MI) between the layers and the target label, followed by a longer decrease in the MI between the layers and the input variable.
Attentioned Convolutional LSTM InpaintingNetwork for Anomaly Detection in Videos
Our model is shown to be effective in detecting anomalies in videos.
Opening the Black Box of Deep Neural Networks via Information
Previous work proposed to analyze DNNs in the \textit{Information Plane}; i. e., the plane of the Mutual Information values that each layer preserves on the input and output variables.
