Information Plane
10 papers with code • 0 benchmarks • 0 datasets
To obtain the Information Plane (IP) of deep neural networks, which shows the trajectories of the hidden layers during training in a 2D plane using as coordinate axes the mutual information between the input and the hidden layer, and the mutual information between the output and the hidden layer.
Benchmarks
These leaderboards are used to track progress in Information Plane
Latest papers with no code
Information Plane Analysis Visualization in Deep Learning via Transfer Entropy
Information Plane analysis is a visualization technique used to understand the trade-off between compression and information preservation in the context of the Information Bottleneck method by plotting the amount of information in the input data against the compressed representation.
Improving the Robustness of Quantized Deep Neural Networks to White-Box Attacks using Stochastic Quantization and Information-Theoretic Ensemble Training
We present a method to improve the robustness of quantized DNNs to white-box adversarial attacks.
SoFaiR: Single Shot Fair Representation Learning
To avoid discriminatory uses of their data, organizations can learn to map them into a representation that filters out information related to sensitive attributes.
Mutual information estimation for graph convolutional neural networks
Mutual information can be used as a measure of the quality of internal representations in deep learning models, and the information plane may provide insights into whether the model exploits the available information in the data.
A Comparative Genomic Analysis of Coronavirus Families Using Chaos Game Representation and Fisher-Shannon Complexity
From its first emergence in Wuhan, China in December, 2019 the COVID-19 pandemic has caused unprecedented health crisis throughout the world.
Fundamental Limits and Tradeoffs in Invariant Representation Learning
A wide range of machine learning applications such as privacy-preserving learning, algorithmic fairness, and domain adaptation/generalization among others, involve learning invariant representations of the data that aim to achieve two competing goals: (a) maximize information or accuracy with respect to a target response, and (b) maximize invariance or independence with respect to a set of protected features (e. g., for fairness, privacy, etc).
On Information Plane Analyses of Neural Network Classifiers -- A Review
Specifically, we argue that even in feed-forward neural networks the data processing inequality need not hold for estimates of mutual information.
On Predictive Information in RNNs
Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future.
Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels
In this paper, we propose an IP analysis using the new matrix--based R\'enyi's entropy coupled with tensor kernels over convolutional layers, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data.
On Predictive Information Sub-optimality of RNNs
Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future.
