Search Results for author:
Found 15 papers, 3 papers with code
Minimum Description Length and Generalization Guarantees for Representation Learning
Rather than the mutual information between the encoder's input and the representation, which is often believed to reflect the algorithm's generalization capability in the related literature but in fact, falls short of doing so, our new bounds involve the "multi-letter" relative entropy between the distribution of the representations (or labels) of the training and test sets and a fixed prior.
Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD
Neural network compression has been an increasingly important subject, not only due to its practical relevance, but also due to its theoretical implications, as there is an explicit connection between compressibility and generalization error.
Federated Learning You May Communicate Less Often!
Moreover, specialized to the case $R=1$ (sometimes referred to as "one-shot" FL or distributed learning) our bounds suggest that the generalization error of the FL setting decreases faster than that of centralized learning by a factor of $\mathcal{O}(\sqrt{\log(K)/K})$, thereby generalizing recent findings in this direction to arbitrary loss functions and algorithms.
More Communication Does Not Result in Smaller Generalization Error in Federated Learning
We establish an upper bound on the generalization error that accounts explicitly for the effect of $R$ (in addition to the number of participating devices $K$ and dataset size $n$).
Data-dependent Generalization Bounds via Variable-Size Compressibility
In this framework, the generalization error of an algorithm is linked to a variable-size 'compression rate' of its input data.
Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learning
In this paper, we use tools from rate-distortion theory to establish new upper bounds on the generalization error of statistical distributed learning algorithms.
In-Network Learning: Distributed Training and Inference in Networks
It is widely perceived that leveraging the success of modern machine learning techniques to mobile devices and wireless networks has the potential of enabling important new services.
On learning parametric distributions from quantized samples
We consider the problem of learning parametric distributions from their quantized samples in a network.
On In-network learning. A Comparative Study with Federated and Split Learning
In this paper, we consider a problem in which distributively extracted features are used for performing inference in wireless networks.
Scalable Vector Gaussian Information Bottleneck
In the context of statistical learning, the Information Bottleneck method seeks a right balance between accuracy and generalization capability through a suitable tradeoff between compression complexity, measured by minimum description length, and distortion evaluated under logarithmic loss measure.
On the Relevance-Complexity Region of Scalable Information Bottleneck
The Information Bottleneck method is a learning technique that seeks a right balance between accuracy and generalization capability through a suitable tradeoff between compression complexity, measured by minimum description length, and distortion evaluated under logarithmic loss measure.
On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views
This tutorial paper focuses on the variants of the bottleneck problem taking an information theoretic perspective and discusses practical methods to solve it, as well as its connection to coding and learning aspects.
An Information Theoretic Approach to Distributed Representation Learning
The problem of distributed representation learning is one in which multiple sources of information X1,..., XK are processed separately so as to extract useful information about some statistically correlated ground truth Y.
Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding
In this paper, we develop an unsupervised generative clustering framework that combines the Variational Information Bottleneck and the Gaussian Mixture Model.
Distributed Variational Representation Learning
The problem of distributed representation learning is one in which multiple sources of information $X_1,\ldots, X_K$ are processed separately so as to learn as much information as possible about some ground truth $Y$.
