Search Results for author:
Found 24 papers, 10 papers with code
Noise driven phase transitions in eco-evolutionary systems
In complex ecosystems such as microbial communities, there is constant ecological and evolutionary feedback between the residing species and the environment occurring on concurrent timescales.
Generalized Information Bottleneck for Gaussian Variables
Here we consider a generalized IB problem, in which the mutual information in the original IB method is replaced by correlation measures based on Renyi and Jeffreys divergences.
Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality
Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss.
An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers
Recent work demonstrates that deep neural networks trained using Empirical Risk Minimization (ERM) can generalize under distribution shift, outperforming specialized training algorithms for domain generalization.
Learning Background Invariance Improves Generalization and Robustness in Self-Supervised Learning on ImageNet and Beyond
Through a systematic, comprehensive investigation, we show that background augmentations lead to improved generalization with substantial improvements ($\sim$1-2% on ImageNet) in performance across a spectrum of state-of-the-art self-supervised methods (MoCo-v2, BYOL, SwAV) on a variety of tasks, even enabling performance on par with the supervised baseline.
Perturbation Theory for the Information Bottleneck
Here we derive a perturbation theory for the IB method and report the first complete characterization of the learning onset, the limit of maximum relevant information per bit extracted from data.
Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations
Recent progress in self-supervised learning has demonstrated promising results in multiple visual tasks.
Ranked #83 on
Image Classification
on ObjectNet
(using extra training data)
Implicit Regularization of SGD via Thermophoresis
The strength of this effect is proportional to squared learning rate and inverse batch size, and is more effective during the early phase of training when the model's predictions are poor.
Are all negatives created equal in contrastive instance discrimination?
Using methodology from MoCo v2 (Chen et al., 2020), we divided negatives by their difficulty for a given query and studied which difficulty ranges were most important for learning useful representations.
Learning Optimal Representations with the Decodable Information Bottleneck
We address the question of characterizing and finding optimal representations for supervised learning.
Theory of gating in recurrent neural networks
The gate modulating the dimensionality can induce a novel, discontinuous chaotic transition, where inputs push a stable system to strong chaotic activity, in contrast to the typically stabilizing effect of inputs.
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features.
The Early Phase of Neural Network Training
We perform extensive measurements of the network state during these early iterations of training and leverage the framework of Frankle et al. (2019) to quantitatively probe the weight distribution and its reliance on various aspects of the dataset.
Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs
Here, we take the perspective of studying randomly initialized LSTMs and GRUs as dynamical systems, and ask how the salient dynamical properties are shaped by the gates.
How noise affects the Hessian spectrum in overparameterized neural networks
Stochastic gradient descent (SGD) forms the core optimization method for deep neural networks.
Mean-field Analysis of Batch Normalization
Batch Normalization (BatchNorm) is an extremely useful component of modern neural network architectures, enabling optimization using higher learning rates and achieving faster convergence.
Energy consumption and cooperation for optimal sensing
The reliable detection of environmental molecules in the presence of noise is an important cellular function, yet the underlying computational mechanisms are not well understood.
Biological Physics
A high-bias, low-variance introduction to Machine Learning for physicists
The purpose of this review is to provide an introduction to the core concepts and tools of machine learning in a manner easily understood and intuitive to physicists.
The information bottleneck and geometric clustering
The information bottleneck (IB) approach to clustering takes a joint distribution $P\!\left(X, Y\right)$ and maps the data $X$ to cluster labels $T$ which retain maximal information about $Y$ (Tishby et al., 1999).
Supervised Learning with Tensor Networks
Tensor networks are approximations of high-order tensors which are efficient to work with and have been very successful for physics and mathematics applications.
Comment on "Why does deep and cheap learning work so well?" [arXiv:1608.08225]
", Lin and Tegmark claim to show that the mapping between deep belief networks and the variational renormalization group derived in [arXiv:1410. 3831] is invalid, and present a "counterexample" that claims to show that this mapping does not hold.
Supervised Learning with Quantum-Inspired Tensor Networks
Tensor networks are efficient representations of high-dimensional tensors which have been very successful for physics and mathematics applications.
The deterministic information bottleneck
Here, we introduce an alternative formulation that replaces mutual information with entropy, which we call the deterministic information bottleneck (DIB), that we argue better captures this notion of compression.
An exact mapping between the Variational Renormalization Group and Deep Learning
Here, we show that deep learning is intimately related to one of the most important and successful techniques in theoretical physics, the renormalization group (RG).
