Search Results for author:
Found 15 papers, 8 papers with code
The Expressive Power of Tuning Only the Norm Layers
Feature normalization transforms such as Batch and Layer-Normalization have become indispensable ingredients of state-of-the-art deep neural networks.
Looped Transformers as Programmable Computers
We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop.
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
LIFT does not make any changes to the model architecture or loss function, and it solely relies on the natural language interface, enabling "no-code machine learning with LMs."
Utilizing Language-Image Pretraining for Efficient and Robust Bilingual Word Alignment
Word translation without parallel corpora has become feasible, rivaling the performance of supervised methods.
Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond
In distributed learning, local SGD (also known as federated averaging) and its simple baseline minibatch SGD are widely studied optimization methods.
Finding Everything within Random Binary Networks
A recent work by Ramanujan et al. (2020) provides significant empirical evidence that sufficiently overparameterized, random neural networks contain untrained subnetworks that achieve state-of-the-art accuracy on several predictive tasks.
An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks
Recently, Vershynin (2020) settled a long standing question by Baum (1988), proving that \emph{deep threshold} networks can memorize $n$ points in $d$ dimensions using $\widetilde{\mathcal{O}}(e^{1/\delta^2}+\sqrt{n})$ neurons and $\widetilde{\mathcal{O}}(e^{1/\delta^2}(d+\sqrt{n})+n)$ weights, where $\delta$ is the minimum distance between the points.
Permutation-Based SGD: Is Random Optimal?
However, for general strongly convex functions, random permutations are optimal.
Optimal Lottery Tickets via Subset Sum: Logarithmic Over-Parameterization is Sufficient
We show that any target network of width $d$ and depth $l$ can be approximated by pruning a random network that is a factor $O(log(dl))$ wider and twice as deep.
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training.
Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
We show that any target network of width $d$ and depth $l$ can be approximated by pruning a random network that is a factor $O(\log(dl))$ wider and twice as deep.
Closing the convergence gap of SGD without replacement
A recent line of breakthrough works on SGD without replacement (SGDo) established an $\mathcal{O}\left(\frac{n}{T^2}\right)$ convergence rate when the function minimized is strongly convex and is a sum of $n$ smooth functions, and an $\mathcal{O}\left(\frac{1}{T^2}+\frac{n^3}{T^3}\right)$ rate for sums of quadratics.
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation
In this work, we present DETOX, a Byzantine-resilient distributed training framework that combines algorithmic redundancy with robust aggregation.
Convergence and Margin of Adversarial Training on Separable Data
Our results are derived by showing that adversarial training with gradient updates minimizes a robust version of the empirical risk at a $\mathcal{O}(\ln(t)^2/t)$ rate, despite non-smoothness.
Does Data Augmentation Lead to Positive Margin?
Data augmentation (DA) is commonly used during model training, as it significantly improves test error and model robustness.
