Search Results for author:
Found 8 papers, 2 papers with code
q2d: Turning Questions into Dialogs to Teach Models How to Search
One of the exciting capabilities of recent language models for dialog is their ability to independently search for relevant information to ground a given dialog response.
Fine-tuning giant neural networks on commodity hardware with automatic pipeline model parallelism
Fine-tuning is an increasingly common technique that leverages transfer learning to dramatically expedite the training of huge, high-quality models.
Learning Under Delayed Feedback: Implicitly Adapting to Gradient Delays
We consider stochastic convex optimization problems, where several machines act asynchronously in parallel while sharing a common memory.
Gap-Aware Mitigation of Gradient Staleness
In this paper we define the Gap as a measure of gradient staleness and propose Gap-Aware (GA), a novel asynchronous-distributed method that penalizes stale gradients linearly to the Gap and performs well even when scaling to large numbers of workers.
Gap Aware Mitigation of Gradient Staleness
In this paper we define the Gap as a measure of gradient staleness and propose Gap-Aware (GA), a novel asynchronous-distributed method that penalizes stale gradients linearly to the Gap and performs well even when scaling to large numbers of workers.
Taming Momentum in a Distributed Asynchronous Environment
We propose DANA: a novel technique for asynchronous distributed SGD with momentum that mitigates gradient staleness by computing the gradient on an estimated future position of the model's parameters.
DANA: Scalable Out-of-the-box Distributed ASGD Without Retuning
We propose DANA, a novel approach that scales out-of-the-box to large clusters using the same hyperparameters and learning schedule optimized for training on a single worker, while maintaining similar final accuracy without additional overhead.
Faster Neural Network Training with Approximate Tensor Operations
We propose a novel technique for faster deep neural network training which systematically applies sample-based approximation to the constituent tensor operations, i. e., matrix multiplications and convolutions.
