1 code implementation • 17 Dec 2023 • Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, Philippe Rigollet
Transformers play a central role in the inner workings of large language models.
1 code implementation • NeurIPS 2023 • Patrik Róbert Gerber, Tianze Jiang, Yury Polyanskiy, Rui Sun
Given $n$ observations from two balanced classes, consider the task of labeling an additional $m$ inputs that are known to all belong to \emph{one} of the two classes.
1 code implementation • NeurIPS 2023 • Tejas Jayashankar, Gary C. F. Lee, Alejandro Lancho, Amir Weiss, Yury Polyanskiy, Gregory W. Wornell
We propose a new method for separating superimposed sources using diffusion-based generative models.
1 code implementation • NeurIPS 2023 • Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, Philippe Rigollet
Cluster locations are determined by the initial tokens, confirming context-awareness of representations learned by Transformers.
1 code implementation • 11 Mar 2023 • Gary C. F. Lee, Amir Weiss, Alejandro Lancho, Yury Polyanskiy, Gregory W. Wornell
We study the single-channel source separation problem involving orthogonal frequency-division multiplexing (OFDM) signals, which are ubiquitous in many modern-day digital communication systems.
no code implementations • 9 Feb 2023 • Adam Block, Yury Polyanskiy
Suppose we are given access to $n$ independent samples from distribution $\mu$ and we wish to output one of them with the goal of making the output distributed as close as possible to a target distribution $\nu$.
1 code implementation • 11 Sep 2022 • Alejandro Lancho, Amir Weiss, Gary C. F. Lee, Jennifer Tang, Yuheng Bu, Yury Polyanskiy, Gregory W. Wornell
We study the potential of data-driven deep learning methods for separation of two communication signals from an observation of their mixture.
1 code implementation • 22 Aug 2022 • Gary C. F. Lee, Amir Weiss, Alejandro Lancho, Jennifer Tang, Yuheng Bu, Yury Polyanskiy, Gregory W. Wornell
We study the problem of single-channel source separation (SCSS), and focus on cyclostationary signals, which are particularly suitable in a variety of application domains.
no code implementations • 8 Sep 2021 • Yury Polyanskiy, Yihong Wu
We show that for the Poisson model with compactly supported and subexponential priors, the optimal regret scales as $\Theta((\frac{\log n}{\log\log n})^2)$ and $\Theta(\log^3 n)$, respectively, both attained by the original estimator of Robbins.
no code implementations • 8 Jun 2021 • Adam Block, Zeyu Jia, Yury Polyanskiy, Alexander Rakhlin
It has long been thought that high-dimensional data encountered in many practical machine learning tasks have low-dimensional structure, i. e., the manifold hypothesis holds.
no code implementations • 29 Jan 2021 • Meir Feder, Yury Polyanskiy
The well-specified case corresponds to an additional assumption that the data-generating distribution belongs to the hypothesis class as well.
no code implementations • 29 Jan 2021 • Emmanuel Abbe, Elisabetta Cornacchia, Yuzhou Gu, Yury Polyanskiy
The limit of the entropy in the stochastic block model (SBM) has been characterized in the sparse regime for the special case of disassortative communities [COKPZ17] and for the classical case of assortative communities but in the dense regime [DAM16].
Probability Information Theory Information Theory
no code implementations • 19 Aug 2020 • Yury Polyanskiy, Yihong Wu
Notably, any such Gaussian mixture is statistically indistinguishable from a finite one with $O(\log n)$ components (and this is tight for certain mixtures).
no code implementations • 21 May 2020 • Soham Jana, Yury Polyanskiy, Yihong Wu
Nevertheless, we show that in the sublinear regime of $m =\omega(k/\log k)$, it is possible to consistently estimate in total variation the \emph{profile} of the population, defined as the empirical distribution of the sizes of each type, which determines many symmetric properties of the population.
no code implementations • 30 Apr 2020 • Ziv Goldfeld, Yury Polyanskiy
The information bottleneck (IB) theory recently emerged as a bold information-theoretic paradigm for analyzing DL systems.
no code implementations • ICLR 2019 • Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Brian Kingsbury, Igor Melnyk, Nam Nguyen, Yury Polyanskiy
We then develop a rigorous estimator for I(X;T) in noisy DNNs and observe compression in various models.
no code implementations • 25 Jan 2019 • Uri Hadar, Jingbo Liu, Yury Polyanskiy, Ofer Shayevitz
Our results also imply an $\Omega(n)$ lower bound on the information complexity of the Gap-Hamming problem, for which we show a direct information-theoretic proof.
no code implementations • 12 Oct 2018 • Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, Yury Polyanskiy
We then develop a rigorous estimator for $I(X;T)$ in noisy DNNs and observe compression in various models.
no code implementations • 8 May 2018 • Ziv Goldfeld, Guy Bresler, Yury Polyanskiy
We first show that at zero temperature, order of $\sqrt{n}$ bits can be stored in the system indefinitely by coding over stable, striped configurations.
Information Theory Statistical Mechanics Information Theory
no code implementations • 18 Feb 2017 • Yury Polyanskiy, Ananda Theertha Suresh, Yihong Wu
For noisy population recovery, the sharp sample complexity turns out to be more sensitive to dimension and scales as $\exp(\Theta(d^{1/3} \log^{2/3}(1/\delta)))$ except for the trivial cases of $\epsilon=0, 1/2$ or $1$.