Search Results for author:
Found 24 papers, 9 papers with code
Universal Estimation of Directed Information
Four estimators of the directed information rate between a pair of jointly stationary ergodic finite-alphabet processes are proposed, based on universal probability assignments.
Information Theory Information Theory
Beyond Maximum Likelihood: from Theory to Practice
In a nutshell, a message of this recent work is that, for a wide class of functionals, the performance of these essentially optimal estimators with $n$ samples is comparable to that of the MLE with $n \ln n$ samples.
Demystifying ResNet
We show that with or without nonlinearities, by adding shortcuts that have depth two, the condition number of the Hessian of the loss function at the zero initial point is depth-invariant, which makes training very deep models no more difficult than shallow ones.
Estimating the Fundamental Limits is Easier than Achieving the Fundamental Limits
We show through case studies that it is easier to estimate the fundamental limits of data processing than to construct explicit algorithms to achieve those limits.
Approximate Profile Maximum Likelihood
We propose an efficient algorithm for approximate computation of the profile maximum likelihood (PML), a variant of maximum likelihood maximizing the probability of observing a sufficient statistic rather than the empirical sample.
Entropy Rate Estimation for Markov Chains with Large State Space
For estimating the Shannon entropy of a distribution on $S$ elements with independent samples, [Paninski2004] showed that the sample complexity is sublinear in $S$, and [Valiant--Valiant2011] showed that consistent estimation of Shannon entropy is possible if and only if the sample size $n$ far exceeds $\frac{S}{\log S}$.
Local moment matching: A unified methodology for symmetric functional estimation and distribution estimation under Wasserstein distance
We present \emph{Local Moment Matching (LMM)}, a unified methodology for symmetric functional estimation and distribution estimation under Wasserstein distance.
NECST: Neural Joint Source-Channel Coding
For reliable transmission across a noisy communication channel, classical results from information theory show that it is asymptotically optimal to separate out the source and channel coding processes.
Towards improved lossy image compression: Human image reconstruction with public-domain images
The images, results and additional data are available at https://compression. stanford. edu/human-compression
Neural Joint Source-Channel Coding
For reliable transmission across a noisy communication channel, classical results from information theory show that it is asymptotically optimal to separate out the source and channel coding processes.
LFZip: Lossy compression of multivariate floating-point time series data via improved prediction
Time series data compression is emerging as an important problem with the growth in IoT devices and sensors.
Optimal No-regret Learning in Repeated First-price Auctions
In this paper, we develop the first learning algorithm that achieves a near-optimal $\widetilde{O}(\sqrt{T})$ regret bound, by exploiting two structural properties of first-price auctions, i. e. the specific feedback structure and payoff function.
Learning to Bid Optimally and Efficiently in Adversarial First-price Auctions
In this paper, we take an online learning angle and address the fundamental problem of learning to bid in repeated first-price auctions, where both the bidder's private valuations and other bidders' bids can be arbitrary.
Noisy Neural Network Compression for Analog Storage Devices
Efficient compression and storage of neural network (NN) parameters is critical for resource-constrained, downstream machine learning applications.
Reducing latency and bandwidth for video streaming using keypoint extraction and digital puppetry
COVID-19 has made video communication one of the most important modes of information exchange.
Neural Network Compression for Noisy Storage Devices
Compression and efficient storage of neural network (NN) parameters is critical for applications that run on resource-constrained devices.
An Information-Theoretic Justification for Model Pruning
We study the neural network (NN) compression problem, viewing the tension between the compression ratio and NN performance through the lens of rate-distortion theory.
Txt2Vid: Ultra-Low Bitrate Compression of Talking-Head Videos via Text
Video represents the majority of internet traffic today, driving a continual race between the generation of higher quality content, transmission of larger file sizes, and the development of network infrastructure.
Lossy Compression of Noisy Data for Private and Data-Efficient Learning
In this sense, the utility of the data for learning is essentially maintained, while reducing storage and privacy leakage by quantifiable amounts.
An Interactive Annotation Tool for Perceptual Video Compression
We use this tool to collect data in-the-wild (10 videos, 17 users) and utilize the obtained importance maps in the context of x264 coding to demonstrate that the tool can indeed be used to generate videos which, at the same bitrate, look perceptually better through a subjective study - and are 1. 9 times more likely to be preferred by viewers.
Sparse Random Networks for Communication-Efficient Federated Learning
At the end of the training, the final model is a sparse network with random weights -- or a subnetwork inside the dense random network.
Leveraging the Hints: Adaptive Bidding in Repeated First-Price Auctions
In the past four years, a particularly important development in the digital advertising industry is the shift from second-price auctions to first-price auctions for online display ads.
PIM: Video Coding using Perceptual Importance Maps
The contributions of this work are threefold: (1) we introduce a web-tool which allows scalable collection of fine-grained perceptual importance, by having users interactively paint spatio-temporal maps over encoded videos; (2) we use this tool to collect a dataset with 178 videos with a total of 14443 frames of human annotated spatio-temporal importance maps over the videos; and (3) we use our curated dataset to train a lightweight machine learning model which can predict these spatio-temporal importance regions.
Adaptive Compression in Federated Learning via Side Information
The high communication cost of sending model updates from the clients to the server is a significant bottleneck for scalable federated learning (FL).
