Search Results for author:
Found 11 papers, 4 papers with code
Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains
Inspired by the Markovianity of natural languages, we model the data as a Markovian source and utilize this framework to systematically study the interplay between the data-distributional properties, the transformer architecture, the learnt distribution, and the final model performance.
LASER: Linear Compression in Wireless Distributed Optimization
On the latter, we obtain $50$-$64 \%$ improvement in perplexity over our baselines for noisy channels.
Machine Learning-Aided Efficient Decoding of Reed-Muller Subcodes
Next, we derive the soft-decision based version of our algorithm, called soft-subRPA, that not only improves upon the performance of subRPA but also enables a differentiable decoding algorithm.
CRISP: Curriculum based Sequential Neural Decoders for Polar Code Family
We design a principled curriculum, guided by information-theoretic insights, to train CRISP and show that it outperforms the successive-cancellation (SC) decoder and attains near-optimal reliability performance on the Polar(32, 16) and Polar(64, 22) codes.
KO codes: Inventing Nonlinear Encoding and Decoding for Reliable Wireless Communication via Deep-learning
In this paper, we construct KO codes, a computationaly efficient family of deep-learning driven (encoder, decoder) pairs that outperform the state-of-the-art reliability performance on the standardized AWGN channel.
Reed-Muller Subcodes: Machine Learning-Aided Design of Efficient Soft Recursive Decoding
To lower the complexity of our decoding algorithm, referred to as subRPA in this paper, we investigate different ways for pruning the projections.
Information Theory Information Theory
Towards Principled Objectives for Contrastive Disentanglement
%In fact, contrastive disentanglement and unsupervised recovery are often combined in that we seek additional variations that exhibit salient factors/properties.
Optimal transport mapping via input convex neural networks
Building upon recent advances in the field of input convex neural networks, we propose a new framework where the gradient of one convex function represents the optimal transport mapping.
Learning in Gated Neural Networks
Gating is a key feature in modern neural networks including LSTMs, GRUs and sparsely-gated deep neural networks.
Learning One-hidden-layer Neural Networks under General Input Distributions
Significant advances have been made recently on training neural networks, where the main challenge is in solving an optimization problem with abundant critical points.
Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms
Once the experts are known, the recovery of gating parameters still requires an EM algorithm; however, we show that the EM algorithm for this simplified problem, unlike the joint EM algorithm, converges to the true parameters.
