Search Results for author:
Found 17 papers, 6 papers with code
Efficient Stagewise Pretraining via Progressive Subnetworks
RaPTr achieves better pre-training loss for BERT and UL2 language models while requiring 20-33% fewer FLOPs compared to standard training, and is competitive or better than other efficient training methods.
The Inductive Bias of Flatness Regularization for Deep Matrix Factorization
Recent works on over-parameterized neural networks have shown that the stochasticity in optimizers has the implicit regularization effect of minimizing the sharpness of the loss function (in particular, the trace of its Hessian) over the family zero-loss solutions.
FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients
In classical federated learning, the clients contribute to the overall training by communicating local updates for the underlying model on their private data to a coordinating server.
Efficient Training of Retrieval Models using Negative Cache
These models are typically trained by optimizing the model parameters to score relevant positive" pairs higher than the irrelevantnegative" ones.
Breaking the centralized barrier for cross-device federated learning
Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon.
Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
Negative sampling schemes enable efficient training given a large number of classes, by offering a means to approximate a computationally expensive loss function that takes all labels into account.
O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers
We propose sufficient conditions under which we prove that a sparse attention model can universally approximate any sequence-to-sequence function.
Federated Composite Optimization
We first show that straightforward extensions of primal algorithms such as FedAvg are not well-suited for FCO since they suffer from the "curse of primal averaging," resulting in poor convergence.
Adaptive Federated Optimization
Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data.
Breaking the Glass Ceiling for Embedding-Based Classifiers for Large Output Spaces
In extreme classification settings, embedding-based neural network models are currently not competitive with sparse linear and tree-based methods in terms of accuracy.
Multilabel reductions: what is my loss optimising?
Multilabel classification is a challenging problem arising in applications ranging from information retrieval to image tagging.
Learning to Learn by Zeroth-Order Oracle
In the learning to learn (L2L) framework, we cast the design of optimization algorithms as a machine learning problem and use deep neural networks to learn the update rules.
Concise Multi-head Attention Models
Attention based Transformer architecture has enabled significant advances in the field of natural language processing.
LEARNING TO LEARN WITH BETTER CONVERGENCE
We consider the learning to learn problem, where the goal is to leverage deeplearning models to automatically learn (iterative) optimization algorithms for training machine learning models.
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.
Ranked #11 on
Question Answering
on SQuAD1.1 dev
(F1 metric)
Adaptive Methods for Nonconvex Optimization
In this work, we provide a new analysis of such methods applied to nonconvex stochastic optimization problems, characterizing the effect of increasing minibatch size.
Large-scale randomized-coordinate descent methods with non-separable linear constraints
We develop randomized (block) coordinate descent (CD) methods for linearly constrained convex optimization.
