Information Plane
11 papers with code • 0 benchmarks • 0 datasets
To obtain the Information Plane (IP) of deep neural networks, which shows the trajectories of the hidden layers during training in a 2D plane using as coordinate axes the mutual information between the input and the hidden layer, and the mutual information between the output and the hidden layer.
Benchmarks
These leaderboards are used to track progress in Information Plane
Most implemented papers
Opening the Black Box of Deep Neural Networks via Information
Previous work proposed to analyze DNNs in the \textit{Information Plane}; i. e., the plane of the Mutual Information values that each layer preserves on the input and output variables.
On the Information Bottleneck Theory of Deep Learning
The practical successes of deep neural networks have not been matched by theoretical progress that satisfyingly explains their behavior.
Scalable Mutual Information Estimation using Dependence Graphs
To the best of our knowledge EDGE is the first non-parametric MI estimator that can achieve parametric MSE rates with linear time complexity.
On the Information Plane of Autoencoders
Recently, the Information Plane (IP) was proposed to analyze them, which is based on the information-theoretic concept of mutual information (MI).
The Dual Information Bottleneck
The Information Bottleneck (IB) framework is a general characterization of optimal representations obtained using a principled approach for balancing accuracy and complexity.
Malicious Network Traffic Detection via Deep Learning: An Information Theoretic View
Applying our results can serve to guide analysis methods for machine learning engineers and suggests that neural networks that can exploit the convolution theorem are equally accurate as standard convolutional neural networks, and can be more computationally efficient.
A Provably Convergent Information Bottleneck Solution via ADMM
Conventionally, it resorts to characterizing the information plane, that is, plotting $I(Y;Z)$ versus $I(X;Z)$ for all solutions obtained from different initial points.
Information flows of diverse autoencoders
Thus, we conclude that the compression phase is not necessary for generalization in representation learning.
HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
Even after pruning the filters from convolutional layers of LeNet-5 drastically (i. e. from 20, 50 to 2, 3, respectively), only a small accuracy drop of 0. 52\% is observed.
End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training
End-to-end (E2E) training, optimizing the entire model through error backpropagation, fundamentally supports the advancements of deep learning.