Search Results for author:
Found 45 papers, 21 papers with code
Dissecting Chain-of-Thought: A Study on Compositional In-Context Learning of MLPs
These findings collectively provide insights into the mechanics of CoT, inviting further investigation of its role in complex reasoning tasks.
Prompted LLMs as Chatbot Modules for Long Open-domain Conversation
In this paper, we propose MPC (Modular Prompted Chatbot), a new approach for creating high-quality conversational agents without the need for fine-tuning.
Cuttlefish: Low-Rank Model Training without All the Tuning
Cuttlefish leverages the observation that after a few epochs of full-rank training, the stable rank (i. e., an approximation of the true rank) of each layer stabilizes at a constant value.
The Expressive Power of Tuning Only the Norm Layers
Feature normalization transforms such as Batch and Layer-Normalization have become indispensable ingredients of state-of-the-art deep neural networks.
Looped Transformers as Programmable Computers
We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop.
Transformers as Algorithms: Generalization and Stability in In-context Learning
We first explore the statistical aspects of this abstraction through the lens of multitask learning: We obtain generalization bounds for ICL when the input prompt is (1) a sequence of i. i. d.
A Better Way to Decay: Proximal Gradient Training Algorithms for Neural Nets
For neural networks with ReLU activations, solutions to the weight decay objective are equivalent to those of a different objective in which the regularization term is instead a sum of products of $\ell_2$ (not squared) norms of the input and output weights associated each ReLU.
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
LIFT does not make any changes to the model architecture or loss function, and it solely relies on the natural language interface, enabling "no-code machine learning with LMs."
Utilizing Language-Image Pretraining for Efficient and Robust Bilingual Word Alignment
Word translation without parallel corpora has become feasible, rivaling the performance of supervised methods.
Rare Gems: Finding Lottery Tickets at Initialization
Frankle & Carbin conjecture that we can avoid this by training "lottery tickets", i. e., special sparse subnetworks found at initialization, that can be trained to high accuracy.
GenLabel: Mixup Relabeling using Generative Models
Mixup is a data augmentation method that generates new data points by mixing a pair of input data.
Finding Everything within Random Binary Networks
A recent work by Ramanujan et al. (2020) provides significant empirical evidence that sufficiently overparameterized, random neural networks contain untrained subnetworks that achieve state-of-the-art accuracy on several predictive tasks.
An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks
Recently, Vershynin (2020) settled a long standing question by Baum (1988), proving that \emph{deep threshold} networks can memorize $n$ points in $d$ dimensions using $\widetilde{\mathcal{O}}(e^{1/\delta^2}+\sqrt{n})$ neurons and $\widetilde{\mathcal{O}}(e^{1/\delta^2}(d+\sqrt{n})+n)$ weights, where $\delta$ is the minimum distance between the points.
Pufferfish: Communication-efficient Models At No Extra Cost
In this work, we present Pufferfish, a communication and computation efficient distributed training framework that incorporates the gradient compression into the model training process via training low-rank, pre-factorized deep networks.
On the Utility of Gradient Compression in Distributed Training Systems
A rich body of prior work has highlighted the existence of communication bottlenecks in synchronous data-parallel training.
Permutation-Based SGD: Is Random Optimal?
However, for general strongly convex functions, random permutations are optimal.
Optimal Lottery Tickets via Subset Sum: Logarithmic Over-Parameterization is Sufficient
We show that any target network of width $d$ and depth $l$ can be approximated by pruning a random network that is a factor $O(log(dl))$ wider and twice as deep.
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
The techniques usually require choosing a static compression ratio, often requiring users to balance the trade-off between model accuracy and per-iteration speedup.
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training.
Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
We show that any target network of width $d$ and depth $l$ can be approximated by pruning a random network that is a factor $O(\log(dl))$ wider and twice as deep.
Closing the convergence gap of SGD without replacement
A recent line of breakthrough works on SGD without replacement (SGDo) established an $\mathcal{O}\left(\frac{n}{T^2}\right)$ convergence rate when the function minimized is strongly convex and is a sum of $n$ smooth functions, and an $\mathcal{O}\left(\frac{1}{T^2}+\frac{n^3}{T^3}\right)$ rate for sums of quadratics.
Federated Learning with Matched Averaging
Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud.
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation
In this work, we present DETOX, a Byzantine-resilient distributed training framework that combines algorithmic redundancy with robust aggregation.
Bad Global Minima Exist and SGD Can Reach Them
Several works have aimed to explain why overparameterized neural networks generalize well when trained by Stochastic Gradient Descent (SGD).
Convergence and Margin of Adversarial Training on Separable Data
Our results are derived by showing that adversarial training with gradient updates minimizes a robust version of the empirical risk at a $\mathcal{O}(\ln(t)^2/t)$ rate, despite non-smoothness.
Does Data Augmentation Lead to Positive Margin?
Data augmentation (DA) is commonly used during model training, as it significantly improves test error and model robustness.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
ErasureHead: Distributed Gradient Descent without Delays Using Approximate Gradient Coding
We present ErasureHead, a new approach for distributed gradient descent (GD) that mitigates system delays by employing approximate gradient coding.
A Geometric Perspective on the Transferability of Adversarial Directions
We show that these "transferable adversarial directions" are guaranteed to exist for linear separators of a given set, and will exist with high probability for linear classifiers trained on independent sets drawn from the same distribution.
ATOMO: Communication-efficient Learning via Atomic Sparsification
We present ATOMO, a general framework for atomic sparsification of stochastic gradients.
The Effect of Network Width on the Performance of Large-batch Training
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training.
Gradient Coding via the Stochastic Block Model
Gradient descent and its many variants, including mini-batch stochastic gradient descent, form the algorithmic foundation of modern large-scale machine learning.
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients
Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i. e., nodes that use malicious updates to corrupt the global model stored at a parameter server (PS).
Approximate Gradient Coding via Sparse Random Graphs
Distributed algorithms are often beset by the straggler effect, where the slowest compute nodes in the system dictate the overall running time.
Stability and Generalization of Learning Algorithms that Converge to Global Optima
Finally, we show that although our results imply comparable stability for SGD and GD in the PL setting, there exist simple neural networks with multiple local minima where SGD is stable but GD is not.
Gradient Diversity: a Key Ingredient for Scalable Distributed Learning
It has been experimentally observed that distributed implementations of mini-batch stochastic gradient descent (SGD) algorithms exhibit speedup saturation and decaying generalization ability beyond a particular batch-size.
CYCLADES: Conflict-free Asynchronous Machine Learning
We present CYCLADES, a general framework for parallelizing stochastic optimization algorithms in a shared memory setting.
Bipartite Correlation Clustering -- Maximizing Agreements
We present a novel approximation algorithm for $k$-BCC, a variant of BCC with an upper bound $k$ on the number of clusters.
Speeding Up Distributed Machine Learning Using Codes
We focus on two of the most basic building blocks of distributed learning algorithms: matrix multiplication and data shuffling.
Orthogonal NMF through Subspace Exploration
Our algorithm relies on a novel approximation to the related Nonnegative Principal Component Analysis (NNPCA) problem; given an arbitrary data matrix, NNPCA seeks $k$ nonnegative components that jointly capture most of the variance.
Sparse PCA via Bipartite Matchings
We consider the following multi-component sparse PCA problem: given a set of data points, we seek to extract a small number of sparse components with disjoint supports that jointly capture the maximum possible variance.
Perturbed Iterate Analysis for Asynchronous Stochastic Optimization
We demonstrate experimentally on a 16-core machine that the sparse and parallel version of SVRG is in some cases more than four orders of magnitude faster than the standard SVRG algorithm.
On the Worst-Case Approximability of Sparse PCA
It is well known that Sparse PCA (Sparse Principal Component Analysis) is NP-hard to solve exactly on worst-case instances.
Parallel Correlation Clustering on Big Graphs
We present C4 and ClusterWild!, two algorithms for parallel correlation clustering that run in a polylogarithmic number of rounds and achieve nearly linear speedups, provably.
Provable Deterministic Leverage Score Sampling
We explain theoretically a curious empirical phenomenon: "Approximating a matrix by deterministically selecting a subset of its columns with the corresponding largest leverage scores results in a good low-rank matrix surrogate".
